hackernews-reader.md

Hackernews Reader

Hackernews Reader uses http-source to build an XML reader that ingests hackernews articles, converts them to json, divides them into records, and publishes each record to a topic.

Objective

Show an example on how to query XML RSS feeds and convert each article into an individual JSON event for downstream processing.

Prerequsites

• Fluvio CLI running locally
• Account on InfinyOn Cloud

Step-by-Step

1. Create http-source configuration file
2. Download smartmodules
3. Start Connector
4. Check Results

Create http-source configuration file

Create an HTTP source connector configuration file called hackernews.yaml :

apiVersion: 0.1.0
meta:
  version: 0.3.6
  name: hackernews 
  type: http-source
  topic: hackernews
http:
  method: GET
  endpoint: 'https://hnrss.org/newest'
  interval: 600s
transforms:
  - uses: infinyon-labs/rss-json@0.1.1
  - uses: infinyon/jolt@0.4.1
    with:
      spec:
      - operation: shift
        spec:
          items: ""
  - uses: infinyon-labs/array-map-json@0.1.1

Download startmodules

Download the smartmodules used by the connectors to your cluster:

fluvio hub sm download infinyon/jolt@0.4.1
fluvio hub sm download infinyon-labs/rss-json@0.1.1
fluvio hub sm download infinyon-labs/array-map-json@0.1.1

Check fluvio smartmodule list to ensure they've been downloaded.

Start Connector

fluvio cloud connector create --config hackernews.yaml

Check Results

Connector logs:

fluvio cloud connector log hackernews

Records produced:

fluvio consume hackernews -T 10

References

• How to Stream and Transform Data from Hacker News RSS Feed (YouTube Video)
• labs-rss-json-sm
• labs-array-map-json-sm

---