Next Step Microgrant: Provider.Quest - Accurate geolocation data for visualizations

[jimpick]

1. What is your project? (max 100 words)

Miner.Report is a project to collect analytics data from the Filecoin blockchain. It is also a collection of notebooks on ObservableHQ with interactive visualizations that use the data.

https://miner.report/

https://github.com/jimpick/miner-report-spark

2. How are you planning to improve this project? (max 200 words)

Collect data to determine miner liveness and geolocation using advanced techniques for greater accuracy.

• Regular miner-info queries for on-chain multiaddrs [completed], DHT lookups for active miners, DNS -> IP address lookups for /dns4/ and /dns6/ multiaddrs, libp2p pings + collect version info
• IPv4 and IPv6 GeoIP lookups using MaxMind GeoLite2 (free) and GeoIP2 API (non-free) + Baidu for China IPs
• Traceroutes + ASN lookups via RIPE database per IP
• JSON schema to represent multiple locations per miner ... publish/syndicate data to Textile Bucket
• deterministic geo "faker" to generate realistic locations (based on earth population data) for addresses without geo data (eg. client addresses)
• Visualization utilizing Deck.gl Trips Layer + 24 hours of deal data => https://deck.gl/examples/trips-layer/

3. Will the work be Open Source?

Yes, MIT+Apache. Data is CC-BY-SA 3.0.

4. If selected, do you agree to share weekly updates and a grant report upon conclusion?

Yes.

5. Does your proposal comply with our Community Code of Conduct?

Yes.

6. Links and submissions

Have you submitted to the Filecoin Community Showcase?

Yes. PR is not merged yet, I will update when it is merged: filecoin-project/community#206

Branch version: https://github.com/jimpick/community/blob/project/miner-report/projects/application/miner-report-en.md

Have you created a thread introducing your project in the community discussion board's Project Showcase category?

filecoin-project/community#208

If your project began at a hackathon, have you submitted it for the relevant Protocol Labs prizes? Include links here if available:

N/A

Additional questions:

For each team member(s), please list name, email, Github account, and role in the project.

Jim Pick, jim@jimpick.com, @jimpick - sole contributor

If your project was created as part of an event or hackathon: What was the name of the event? (e.g. ETHGlobal NFTHack, Cal Hacks hello:world, Chainlink, CivHacks, GameDevJ, ETHGlobal Scaling Ethereum)

N/A

[realChainLife]

This microgrant has been approved! Look out for an email to submit payment details. Let's use your existing Community Discussion thread for updates.

[jimpick]

A quick summary of the results ... I was able to integrate geo IP lookups into the Provider.Quest project (renamed from Miner.Report). I have lookups with GeoLite2 and Baidu, but I didn't implement GeoIP2 (non-free) or traceroute-based lookups. That might be something to do in the future for a small accuracy boost.

The location data was used to segment providers by regions and generate notebooks for examining activity per region:

• https://observablehq.com/@jimpick/provider-quest-regional-stats
• https://observablehq.com/@jimpick/provider-quest-zoomable-sunburst

I did implement the deterministic geo "faker" to generate realistic locations: https://observablehq.com/@jimpick/65-536-cities-demo

I also using the Deck.gl TripsLayer to implement a deal visualization video which was revealed during the Filecoin Orbit event.

• https://observablehq.com/@jimpick/videos-provider-quest
• https://observablehq.com/@jimpick/deals-trips-provider-quest
• https://twitter.com/momack28/status/1452629517856501763 (@momack2 tweeted it and got over 7000 views!)

filecoin-orbit-2.mp4