Data pipeline for Filecoin analytical data for https://provider.quest/
The following scripts live here:
- scripts to collect data from "Provider.Quest" ObservableHQ notebooks and store the data as JSON files
- a Pyspark script to run Apache Spark Structured Streaming, which ingests the JSON files continuously, and generates new JSON files with aggregations and mapping
- scripts to publish aggregrated data to Textile Buckets for syndication
Dual-licensed under MIT + Apache 2.0
Data is licenced as CC-BY-SA 3.0 unless otherwised noted.