Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data updates #10

Open
tyrasd opened this issue Sep 1, 2016 · 20 comments
Open

data updates #10

tyrasd opened this issue Sep 1, 2016 · 20 comments

Comments

@tyrasd
Copy link
Owner

@tyrasd tyrasd commented Sep 1, 2016

from https://github.com/tyrasd/taghistory#todos: implement regular (e.g. daily/weekly) data updates. Possible solutions:

  • reprocess the weekly full history dumps (maybe only all keys as a limited subset of tags to limit memory requirements?)
  • use Overpass augmented diffs to calculate daily diffs (currently not feasible because of performance issues, see drolbr/Overpass-API#322, possibly fixed by drolbr/Overpass-API#342)
  • use Overpass augmented diffs by aggregating standard minutely diffs (currently not feasible because of drolbr/Overpass-API#346)
  • download daily planet dumps and calculate tag usage deltas
  • maybe someone out there already runs a global, daily (or quicker) updated osm database which could generate daily tag count diffs as a side product of their update process. see also https://www.openstreetmap.org/user/tyr_asd/diary/39402#comment39983
  • compare daily results from taginfo
@kocio-pl
Copy link

@kocio-pl kocio-pl commented Oct 18, 2016

I would be happy if the updates would happen once a week or even month. It is useful tool to see the trends for example when deciding if something is worth rendering in osm-carto.

Loading

@tyrasd
Copy link
Owner Author

@tyrasd tyrasd commented Feb 2, 2017

FYI: I've just manually uploaded some updated data (including changes until 2017-01-15).

I'll soon look into setting up more regular updates.

//cc @morray @Klumbumbus @nebulon42

Loading

@Klumbumbus
Copy link

@Klumbumbus Klumbumbus commented Feb 2, 2017

Great! Thanks.

Loading

@morray
Copy link

@morray morray commented Feb 4, 2017

thumbsup

Loading

@kocio-pl
Copy link

@kocio-pl kocio-pl commented Jun 29, 2017

Hi, Martin, any news on regular updates or (at least) manual update after few months?

Loading

@tyrasd
Copy link
Owner Author

@tyrasd tyrasd commented Jun 29, 2017

Sorry, I'm currently not working on this. The goal is still to have regular updates eventually, but maybe I can do another manual update soon…

Loading

@kocio-pl
Copy link

@kocio-pl kocio-pl commented Nov 21, 2017

Monthly updates would be highly appreciated and probably enough - be it automatic or even manual. Currently "shop=yes" query shows ~90k in taghistory, while taginfo shows ~126k, so the change is substantial.

Loading

@tyrasd
Copy link
Owner Author

@tyrasd tyrasd commented Nov 25, 2017

I just did another manual data update.

Still no real news regarding more regular updates, though. 😒 But let me just repeat my comment/request from https://www.openstreetmap.org/user/tyr_asd/diary/39402#comment39986 here real quick:

if anyone out there already runs a (daily) updated OSM DB which could produce deltas of the counts of tags in their db (for little extra processing cost), please contact me – I'd love to use that data for updating the taghistory service.

Loading

@kocio-pl
Copy link

@kocio-pl kocio-pl commented Nov 25, 2017

Thanks for the update and for sharing where the problem lies - I passed the message on Talk list:

https://lists.openstreetmap.org/pipermail/talk/2017-November/079751.html

Loading

@Zverik
Copy link

@Zverik Zverik commented Nov 25, 2017

I assume the updates can be done without an OSM DB, but with augmented diffs from Overpass API.

Loading

@mmd-osm
Copy link

@mmd-osm mmd-osm commented Nov 26, 2017

I assume the updates can be done without an OSM DB, but with augmented diffs from Overpass API.

In theory yes, but there's an issue in drolbr/Overpass-API#346 as @tyrasd mentioned before. I added a comment there how to resolve the issue for the time being.

Another option might be to use the cached augmented diffs as per drolbr/Overpass-API#346 (comment)

Loading

@kocio-pl
Copy link

@kocio-pl kocio-pl commented Apr 1, 2018

Hi,

Could we have the 2018 update, please?

What about more regular updates and the approach mentioned in a post above? What is the current state of things?

Loading

@chazanov
Copy link

@chazanov chazanov commented Jun 1, 2018

@kocio-pl If you really need it, you may do it on your own - the source code is free. You are always demanding - not a good person!

Loading

@kocio-pl
Copy link

@kocio-pl kocio-pl commented Jun 22, 2018

Thanks for updating! It helped me to see another problem with tag changes: gravitystorm/openstreetmap-carto#3158 (comment).

Loading

@Marc-marc-marc
Copy link

@Marc-marc-marc Marc-marc-marc commented Aug 7, 2018

https://www.openstreetmap.org/user/tyr_asd/diary/39402#comment42459
let's see what is possible to have a regular update

Loading

@kocio-pl
Copy link

@kocio-pl kocio-pl commented Feb 2, 2019

Hi,

I'm back again with the kindly question if it's possible to update it? Maybe there's some help you would need for this and regular updates, as @Marc-marc-marc asked? It's valuable resource to me.

Loading

@kocio-pl
Copy link

@kocio-pl kocio-pl commented May 10, 2019

This change is very useful, thanks! I'm just curious why now different keys have different last update time (so interpolated lines start in different moments)?

Loading

@tyrasd
Copy link
Owner Author

@tyrasd tyrasd commented May 12, 2019

I'm just curious why now different keys have different last update time

I did a quick data update using a new approach earlier this year. But this one was limited to only key=* data. See https://twitter.com/tyr_asd/status/1112387792942841857 (sorry, I forgot to keep you updated here)

Btw: this might also be of interest for you:

You might want to give the #ohsome dashboard a try https://ohsome.org/apps/dashboard/ which I've been working on together with my colleagues at @GIScience – it does all what the taghistory page does, but much more! Only downside are the longer query times for global requests.

Loading

@tilmanb
Copy link
Contributor

@tilmanb tilmanb commented Jan 24, 2020

I know about the ohsome dashboard (which is really cool, by the way), but for small things where you only need "the graph" taginfo is still a valuable tool. Plus it now has permalinks ;-). Long story short: could you please update again? Thanks :-)

Loading

@sun-geo sun-geo mentioned this issue May 28, 2020
@tyrasd
Copy link
Owner Author

@tyrasd tyrasd commented Nov 16, 2020

FYI: taginfo now features daily updated tag count statistics for the "most frequent" tags: https://blog.jochentopf.com/2020-11-08-10-years-of-taginfo.html

Loading

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
9 participants