Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data updates #10

Open
tyrasd opened this issue Sep 1, 2016 · 18 comments

Comments

@tyrasd
Copy link
Owner

commented Sep 1, 2016

from https://github.com/tyrasd/taghistory#todos: implement regular (e.g. daily/weekly) data updates. Possible solutions:

  • reprocess the weekly full history dumps (maybe only all keys as a limited subset of tags to limit memory requirements?)
  • use Overpass augmented diffs to calculate daily diffs (currently not feasible because of performance issues, see drolbr/Overpass-API#322, possibly fixed by drolbr/Overpass-API#342)
  • use Overpass augmented diffs by aggregating standard minutely diffs (currently not feasible because of drolbr/Overpass-API#346)
  • download daily planet dumps and calculate tag usage deltas
  • maybe someone out there already runs a global, daily (or quicker) updated osm database which could generate daily tag count diffs as a side product of their update process. see also https://www.openstreetmap.org/user/tyr_asd/diary/39402#comment39983
  • compare daily results from taginfo

@tyrasd tyrasd added the enhancement label Sep 1, 2016

@kocio-pl

This comment has been minimized.

Copy link

commented Oct 18, 2016

I would be happy if the updates would happen once a week or even month. It is useful tool to see the trends for example when deciding if something is worth rendering in osm-carto.

@tyrasd

This comment has been minimized.

Copy link
Owner Author

commented Feb 2, 2017

FYI: I've just manually uploaded some updated data (including changes until 2017-01-15).

I'll soon look into setting up more regular updates.

//cc @morray @Klumbumbus @nebulon42

@Klumbumbus

This comment has been minimized.

Copy link

commented Feb 2, 2017

Great! Thanks.

@morray

This comment has been minimized.

Copy link

commented Feb 4, 2017

thumbsup

@kocio-pl

This comment has been minimized.

Copy link

commented Jun 29, 2017

Hi, Martin, any news on regular updates or (at least) manual update after few months?

@tyrasd

This comment has been minimized.

Copy link
Owner Author

commented Jun 29, 2017

Sorry, I'm currently not working on this. The goal is still to have regular updates eventually, but maybe I can do another manual update soon…

@kocio-pl

This comment has been minimized.

Copy link

commented Nov 21, 2017

Monthly updates would be highly appreciated and probably enough - be it automatic or even manual. Currently "shop=yes" query shows ~90k in taghistory, while taginfo shows ~126k, so the change is substantial.

@tyrasd

This comment has been minimized.

Copy link
Owner Author

commented Nov 25, 2017

I just did another manual data update.

Still no real news regarding more regular updates, though. 😒 But let me just repeat my comment/request from https://www.openstreetmap.org/user/tyr_asd/diary/39402#comment39986 here real quick:

if anyone out there already runs a (daily) updated OSM DB which could produce deltas of the counts of tags in their db (for little extra processing cost), please contact me – I'd love to use that data for updating the taghistory service.

@kocio-pl

This comment has been minimized.

Copy link

commented Nov 25, 2017

Thanks for the update and for sharing where the problem lies - I passed the message on Talk list:

https://lists.openstreetmap.org/pipermail/talk/2017-November/079751.html

@Zverik

This comment has been minimized.

Copy link

commented Nov 25, 2017

I assume the updates can be done without an OSM DB, but with augmented diffs from Overpass API.

@mmd-osm

This comment has been minimized.

Copy link

commented Nov 26, 2017

I assume the updates can be done without an OSM DB, but with augmented diffs from Overpass API.

In theory yes, but there's an issue in drolbr/Overpass-API#346 as @tyrasd mentioned before. I added a comment there how to resolve the issue for the time being.

Another option might be to use the cached augmented diffs as per drolbr/Overpass-API#346 (comment)

@kocio-pl

This comment has been minimized.

Copy link

commented Apr 1, 2018

Hi,

Could we have the 2018 update, please?

What about more regular updates and the approach mentioned in a post above? What is the current state of things?

@chazanov

This comment has been minimized.

Copy link

commented Jun 1, 2018

@kocio-pl If you really need it, you may do it on your own - the source code is free. You are always demanding - not a good person!

@kocio-pl

This comment has been minimized.

Copy link

commented Jun 22, 2018

Thanks for updating! It helped me to see another problem with tag changes: gravitystorm/openstreetmap-carto#3158 (comment).

@Marc-marc-marc

This comment has been minimized.

Copy link

commented Aug 7, 2018

https://www.openstreetmap.org/user/tyr_asd/diary/39402#comment42459
let's see what is possible to have a regular update

@kocio-pl

This comment has been minimized.

Copy link

commented Feb 2, 2019

Hi,

I'm back again with the kindly question if it's possible to update it? Maybe there's some help you would need for this and regular updates, as @Marc-marc-marc asked? It's valuable resource to me.

@kocio-pl

This comment has been minimized.

Copy link

commented May 10, 2019

This change is very useful, thanks! I'm just curious why now different keys have different last update time (so interpolated lines start in different moments)?

@tyrasd

This comment has been minimized.

Copy link
Owner Author

commented May 12, 2019

I'm just curious why now different keys have different last update time

I did a quick data update using a new approach earlier this year. But this one was limited to only key=* data. See https://twitter.com/tyr_asd/status/1112387792942841857 (sorry, I forgot to keep you updated here)

Btw: this might also be of interest for you:

You might want to give the #ohsome dashboard a try https://ohsome.org/apps/dashboard/ which I've been working on together with my colleagues at @GIScience – it does all what the taghistory page does, but much more! Only downside are the longer query times for global requests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.