KnativeTwitter Source CRD.

Overview

This repository implements a simple Event Source for wiring Twitter events into Knative Eventing.

Details

This uses containersource. Until Istio 1.1 you'll need to annotate the twitter source with the traffic.sidecar.istio.io/includeOutboundIPRanges annotation to ensure the source can emit events into the mesh.

Prerequisites

1. Create a Google Cloud project and install the gcloud CLI and run gcloud auth login. This sample will use a mix of gcloud and kubectl commands. The rest of the sample assumes that you've set the $PROJECT_ID environment variable to your Google Cloud project id, and also set your project ID as default using gcloud config set project $PROJECT_ID.

2. Setup Knative Serving

3. Configure outbound network access note you will need to determine the IP range of your cluster. So determine the IP range of your cluster. For example if your IP ranges are: 10.16.0.0/14,10.19.240.0/20 export them like so.

export INCLUDE_OUTBOUND_IPRANGES="10.16.0.0/14,10.19.240.0/20"

1. Setup Knative Eventing using the release.yaml file. This example does not require GCP.

2. Have Twitter API access keys. Good instructions on how to get them

Create the function that receives the tweets matching your query

We're going to launch a service in Knative that gets invoked for each of the tweets matching your query term that we'll set up below.

kubectl --namespace default apply -f https://raw.githubusercontent.com/vaikas-google/twitter/master/config/service.yaml
kubectl --namespace default apply -f https://raw.githubusercontent.com/vaikas-google/twitter/master/config/trigger.yaml

Create a secret with your Twitter secrets

Modify (or create a file like this) ./secret.yaml and replace TWITTER_* entries with real entries for your account and then create the secret:

apiVersion: v1
kind: Secret
metadata:
  name: twitter-secret
type: Opaque
stringData:
  consumer-key: TWITTER_CONSUMER_KEY
  consumer-secret-key: TWITTER_CONSUMER_SECRET_KEY
  access-token: TWITTER_ACCESS_TOKEN
  access-secret: TWITTER_ACCESS_SECRET

And then create the secret like so:

kubectl create -f ./secret.yaml

Launch the twitter source

The source expects you to specify a query string that tells what to search for in the sea of tweets. For example, if you wanted to look for knative, you'd do:

curl https://raw.githubusercontent.com/vaikas-google/twitter/master/config/search-source.yaml | \
sed "s/QUERY/knative/g" | \
sed "s@INCLUDE_OUTBOUND_IPRANGES@$INCLUDE_OUTBOUND_IPRANGES@g" | \
kubectl apply -f -

if you want to search for something else, replace knative with the query string you want to look for.

curl https://raw.githubusercontent.com/vaikas-google/twitter/master/config/search-source.yaml | \
sed "s/QUERY/yourquerystring/g" | kubectl apply -f -

Look for the results of your function execution

You might have to wait some seconds while the elves are busily fetching your tweets, be patient...

kubectl -l 'serving.knative.dev/service=twitter-dumper' logs -c user-container

and you should see tweets that match your query string. When I look for knative, I might see things like this:

2019/01/11 23:03:10 Received Cloud Event Context as: {CloudEventsVersion:0.1 EventID:1083397354315792386 EventTime:2019-01-10 16:17:12 +0000 UTC EventType:com.twitter EventTypeVersion: SchemaURL: ContentType:application/json Source:com.twitter Extensions:map[]}
2019/01/11 23:03:10 Got tweet from "Serverless Fan" text: "RT @sarbjeetjohal: Lambda comes to your data enter through #knative lambda runtime! We knew it was matter of days for people to spread it l…"
2019/01/11 23:03:10 Received Cloud Event Context as: {CloudEventsVersion:0.1 EventID:1083397331918106625 EventTime:2019-01-10 16:17:07 +0000 UTC EventType:com.twitter EventTypeVersion: SchemaURL: ContentType:application/json Source:com.twitter Extensions:map[]}
2019/01/11 23:03:10 Got tweet from "Sarbjeet Johal" text: "Lambda comes to your data enter through #knative lambda runtime! We knew it was matter of days for people to spread… https://t.co/aWo8BEh7GS"
2019/01/11 23:03:10 Received Cloud Event Context as: {CloudEventsVersion:0.1 EventID:1083393535674601472 EventTime:2019-01-10 16:02:02 +0000 UTC EventType:com.twitter EventTypeVersion: SchemaURL: ContentType:application/json Source:com.twitter Extensions:map[]}
2019/01/11 23:03:10 Got tweet from "David Metcalfe" text: "RT @RedHatPartners: #RedHat collaborates with @Google, @SAP, @IBM and others on @KnativeProject to deliver hybrid #serverless workloads to…"

Creating another trigger that sends events to Slack Channel.

This is just a simple extension, showing couple of things:

1. Once event source is configured to send events to broker, wiring them to additional functions is simple.
2. If you want to watch for different kinds of query terms, you will need to create a new source as things work currently.
3. showing fanout that is achieved by using the Broker as Event Delivery System, which manages channels underneath.

For this you need valid Slack credentials, you can obtain them from here: https://api.slack.com/tutorials/slack-apps-hello-world

Basically grab the Webhook URL from there, that's where our function is going to post the tweets that it is seeing. The URL should look like: https://hooks.slack.com/services/

Create a secret with your Slack secrets

Modify (or create a file like this) ./slack-secret.yaml and replace SLACK_POST_URL with the Webhook URL from above.

apiVersion: v1
kind: Secret
metadata:
  name: slack-secret
type: Opaque
stringData:
  slack-post-url: SLACK_POST_URL

And then create the secret like so:

kubectl create -f ./slack-secret.yaml

kubectl --namespace default apply -f https://raw.githubusercontent.com/vaikas-google/twitter/master/config/slacker.yaml
kubectl --namespace default apply -f https://raw.githubusercontent.com/vaikas-google/twitter/master/config/trigger-slack.yaml

And when a new twitter matching your query will be sent to your Slack channel.