Skip to content

Latest commit

 

History

History
158 lines (115 loc) · 4.15 KB

github-to-discord.md

File metadata and controls

158 lines (115 loc) · 4.15 KB
Raw

Github to Discord

This project creates an end-to-end event pipeline that detects changes in github stars & forks and publishes the result to Discord using connectors:

  • http-source: to read periodically from a github, parse the fields from the json output, and publish the result to a topic.

  • http-sink: to listen to the same topic, detect changes, and publish the result to Discord.

Objective

Show an example on how to create a data streaming pipeline that reads from an HTTP API, transforms the output, and invokes a webhook to generate an alert.

Prerequsites

Step-by-Step

  1. Create http-source configuration file
  2. Create http-sink configuration file
  3. Download smartmodules
  4. Start Connectors
  5. Test Data Pipeline

Create http-source configuration file

Create an HTTP source connector configuration file called github.yaml :

apiVersion: 0.1.0
meta:
  version: 0.3.6
  name: github-stars-in
  type: http-source
  topic: stars-forks
  secrets:
    - name: GITHUB_TOKEN
http:
  endpoint: 'https://api.github.com/repos/infinyon/fluvio'
  method: GET
  headers: 
    - 'Authorization: token ${{ secrets.GITHUB_TOKEN }}'
  interval: 30s
transforms:
  - uses: infinyon/jolt@0.4.1
    with:
      spec:
        - operation: shift
          spec:
            "stargazers_count": "stars"
            "forks_count": "forks"

Github rate-limits API requests to 60 per hour, which you an extend to 5000 by creating an application token. Check out github documentation on how to create an Access Tokens.

Add the access token secret in InfinyOn Cloud :

fluvio cloud secret set GITHUB_TOKEN <access-token>

Create http-sink configuration file

Create an HTTP source connector configuration file called discord.yaml :

apiVersion: 0.1.0
meta:
  version: 0.2.8
  name: discord-stars-out
  type: http-sink
  topic: stars-forks
  secrets:
    - name: DISCORD_TOKEN
http:
  endpoint: "https://discord.com/api/webhooks/${{ secrets.DISCORD_TOKEN }}"
  headers:
    - "Content-Type: application/json"
transforms:
  - uses: infinyon-labs/stars-forks-changes@0.1.4
    lookback:
      last: 1
  - uses: infinyon/jolt@0.4.1
    with:
      spec:
        - operation: shift
          spec:
            "result": "content"

Check out Discord Webhooks on how to create a channel webhook token.

Add the access token secret in InfinyOn Cloud :

fluvio cloud secret set DISCORD_TOKEN <webhook-token>

Download Smartmodules

Download the smartmodules used by the connectors to your cluster:

fluvio hub sm download infinyon/jolt@0.4.1
fluvio hub sm download infinyon-labs/stars-forks-changes@0.1.4

Check fluvio smartmodule list to ensure they've been downloaded.

Start Connectors

Start source & sink connectors:

fluvio cloud connector create -config github.yaml
fluvio cloud connector create -config discord.yaml

Check fluvio cloud connector log to ensure they have been successfully provisioned.

Test Data Pipeline

Check the last values generated by the github connector:

$ fluvio consume -dT 1 stars-forks
{"stars":1770,"forks":138}

Produce a new value

$ fluvio produce stars-forks
> {"stars":1769,"forks":138}
OK

An alert with :star2: 1769 will show-up in your discord channel.

References