Geolocation API

[ferristocrat]

Background

What is the problem/pain point?

Customers want to be able to identify the (approximate) geolocations of all segments/pieces on the network for a given "dataset."

Who is impacted?

All customers/users

What is the impact?

It is not possible to easily see the geolocation of object data (with the exception of files that are shared via Linksharing). Customers will be able to easily see and showcase where their data resides. If customers are using geofencing, they can confirm the locations within their selected region that the data resides.

Requirements

User Story

As a Storj user, I want to be able to retrieve the geolocation data for all the pieces of a specified dataset stored on Storj DCS so I can use them in a web application or data visualization tool.

Acceptance Criteria [WIP]

• Geolocation data can be retrieved for a given dataset
  • A dataset can be one or many objects within a whole bucket, prefix, or single object
• Geolocation data can be retrieved by a frontend application (such as a data visualization tool or web app) without the use of a server
  • If we go the route of dumping the JSON into a DCS bucket, linksharing can be used to enable this
• Geolocation data should be refreshed at least once per week (may not be relevant if generated real time)
• User should be able to have Geolocation data publicly accessible OR private (limited) access
  • Enabling public view of Geolocation data should not expose the underlying objects

Success Metrics

• Geolocation data can be easily retrieved for a given dataset
• Performance isn't degraded

Key Considerations

• The dataset may not be at the "root" of the bucket so should be able to handle whole bucket, prefix, or single object
• The dataset can be comprised of many objects

Open Discussion/Questions

• How is "dataset" defined? Will it be a single file? Multiple files?
  • The answer to this will impact the design (bucket level, object level, object tag level, etc)
  • What is the frequency in which the data needs to be refreshed? Does it need to be refreshed?
• Does this functionality need to live in uplink? S3 gateway? both? Doesn't matter?
• Are we able to share the coordinates generated (any licensing restrictions?)
• Our use of geolocation database requires attribution unless we purchase a commercial license ($456/year)

Possible Design

• Add new endpoint/method to Linksharing that returns GeoJSON (similar to the map endpoint: https://link.storjshare.io/s/accesshere/bucketname%2Ffilename?map=1)
• Dump JSON into Buckets:
  • Use object tags (PutObjectTagging) or other metadata when uploading a dataset to DCS
    • If a dataset if we need to perform this query on a bucket, the use of object tags might not be the right solution (note that we do not currently support PutBucketTagging)
  • Periodically "dump" a GeoJSON file into a specified bucket for a specified object tag - say each week
    • or even simpler per request, but don't generate a new GeoJSON if the existing one is younger than, say, a week
    • Or even simpler -- a one time operation that generates the GeoJSON once and dumps it into a bucket

Milestone(s)

• https://github.com/storj/gateway-mt/milestone/19

[kaloyan-raev]

Note that such an API is already implemented in both the satellite and libuplink. It's just only a private API in libuplink, as the only use case so far was linksharing: https://github.com/storj/uplink/blob/main/private/object/object.go#L34-L35

Making the API public would require a review to ensure we can maintain it backward-compatible long-term. Then every app would be able to use to retrieve the data directly from the satellite.

We could also extract the map generation code from linksharing into a separate API package to make it easier for app developers to generate their own maps.

[kaloyan-raev]

[Need to validate] User should be able to have Geolocation data publicly accessible OR private (limited) access

 * Enabling public view of Geolocation data should not expose the underlying objects

This one has already been implemented with storj/edge#209

[ferristocrat]

Thanks @kaloyan-raev!

[kaloyan-raev]

We could also extract the map generation code from linksharing into a separate API package to make it easier for app developers to generate their own maps.

I've just remembered we've also done that a while ago: https://github.com/storj/dotworld

So maybe we just need to a code example on how to use the GetObjectIPSummary API with the dotworld map generator package.

[ferristocrat]

We should close this as the end goal can be achieved by using one of the following methods:

1. Retrieve Geolocation for Entire Network of Nodes
   • Since large data sets will saturate all nodes, the geolocation of all active storage nodes can be used when a dataset is very large
   • All active nodes: https://stats.storjshare.io/nodes_geo.json (refreshed daily)
2. Retrieve Geolocation for Individual Object and/or Prefix
   • Prerequisites
     • GeoLite2 Free Geolocation Data downloaded
     • Go installed
   • Run the following Gist: https://gist.github.com/kaloyan-raev/c3dd95f2347a8d49f882b5576508f294
     • Note that you’ll need to update the constants and only call either GetObjectGeoJSON OR GetPrefixGeoJSON depending on the dataset
     • You can copy the resulting JSON and paste it into https://geojson.io/ to render it on the globe.
     • The code sample also has a helper function that allows retrieving the geolocation for a prefix or an entire bucket. The data is still retrieved object by object, and aggregated at the end.

[iglesiasbrandon]

Closing this issue per Michaels comment.