Stickshift?

[lxbarth]

@Rub21 @tmcw - longer term, how could we use Stickshift to power OSM edit reports? https://github.com/mapbox/stickshift

• For an example of our report-dt's are used https://github.com/mapbox/data/issues/807#issuecomment-74202296
• Functioning instance is on @Rub21's server right now http://rub21.github.io/report-dt/#d&2015-02-06&2015-02-12

cc @aaronlidman @geohacker @planemad

[tmcw]

Huh, this could be very interesting. Really what we need is something of a hybrid @miccolis @ericfischer monster that could build something like:

• A watchbot task that takes changesets, derives metrics of them, and pushes them into a redshift cluster. A potential layout would be like a table of timestamp-user-userstats

Beyond this, I have some UI tweaks to make to make more advanced charts possible but for the most part this is a matter of getting the data in place.

[yhahn]

Following. cc @rclark

I'd love to see an OSM => S3 changesets mirror (would love to use this for other stacks rather than hitting OSM directly), and whenever an object is pushed to this s3 bucket it could trigger s3 notifications or lambda for redshift processing.

[ianshward]

@yhahn @Rub21 @lxbarth

• How many records are there each day in the changesets?
• How many changeset files are published each day?
• Is there any notification / feed / etc. for when a new changeset file is published currently?
• @lxbarth - would it be useful to be able to do one-off / adhoc queries of the last 24 to 48 hours of data, on a daily basis?

[lxbarth]

@ianshward - at a minimum we're talking about mirroring the replication feeds in these directories:

http://planet.osm.org/replication/

How many changeset files are published each day?

• One per minute, hour, day for data updates
• One per minute for changesets

Is there any notification / feed / etc. for when a new changeset file is published currently?

There's no notification when replication files are updated, you'd poll them.

whenever an object is pushed to this s3 bucket it could trigger s3 notifications or lambda for redshift processing.

This is why the s3 notifications are interesting.

@lxbarth - would it be useful to be able to do one-off / adhoc queries of the last 24 to 48 hours of data, on a daily basis?

No, at least not within the scope of a replication mirror.

@ianshward - could we run this as a public service for anyone looking for an on-AWS mirror? This is interesting because I'd like to be able to share any code we produce supporting the data team as openly as possible. If we build on proprietary services sharing our code becomes less useful.

[rclark]

I have code already written to poll http://planet.osm.org/replication/ for changes in dynamosm. When it finds new minutely files there it creates SNS notifications that a file is ready to be downloaded.

If this is broadly useful I can pull that code out of dynamosm into a standalone service. Systems could listen to this SNS in order to grab files from planet.osm.org when they're ready, or this service could fuel a full-blown mirror.

[rclark]

Our OSM mirror is alive and mirroring changesets onto S3. I got it running to see if it works, but we don't need to keep it running if no one is ready to consume it yet.

If we build on proprietary services sharing our code becomes less useful.

The changesets on S3 are currently publicly accessible for download, but the notifications are simply pushed to SNS topics that require adequate permission to subscribe to.

[ianshward]

@rclark nice!

@lxbarth @rclark @Rub21 if adhoc querying is not interesting, then I wonder whether it'd make sense to run a service which does something like:

• parses the .osc
• transforms it to JSON
• unloads it to S3

I'm not sure if it's interesting or even practical that the unloaded data be all the data, but, maybe the parts that we wrap in such a service are already open source or could be written as such, and there'd be a facility to unload data related to only a specified list of users.

I don't know enough about the data in .osc to know if possible, but could you also create any interesting vector tiles out of it?

[ianshward]

Ahh, I left out, that once you have days, months or years of these JSON files in S3, you could then suck them into Redshift w/ something like metrics-warehouse, and if you wanted, do adhoc querying from StickShift that way. This assumes you're happy with whatever attributes you've included in the JSON files.

[rclark]

could you also create any interesting vector tiles out of it?

@ianshward sounds like you're coming close to describing what dynamosm is intended to do -- transform this stream of .osc into a stream of geojson, basically.

You may or may not have to go that far to extract metrics from osc files, it all depends on what exactly the metrics are that you want to collect.

[lxbarth]

Stale.