Mutual update of imagery databases

[bastik]

I recently compared the editor-imagery-index and the JOSM imagery wiki and saw that they have diverged quite a lot in the last 2 years. Both had a significant amount of additions and corrections, but apparently not the same for both. In fact, each database would grow by about 50% if they pulled the updates from the other one!

There is a certain threshold for users to report a new source of imagery, we can be happy if the users contribute new sources at all. It would be too much to ask for an iD user to edit the JOSM wiki in addition to a change request in the editor-imagery-index. Likewise a JOSM user will usually just care about the JOSM repository.

But as far as imagery goes, the choice of editor shouldn't really matter: Any background that is useful in iD will also be helpful in JOSM and vice versa (TMS). So as a common repository for both project seems currently out of reach, we should aim for the next best thing, which is two databases with the same content. As long as the entries get synchronized on a regular basis, this is the same as a shared database from the user perspective. Of course it is more work for the maintainers.

The JOSM team will care for the IIE -> JOSM update. This ticket is basically a reminder that we have lots of good new stuff and you are welcome to take it! :)

JOSM ticket for reference (includes some basic stats): http://josm.openstreetmap.de/ticket/10760

[pnorman]

So as a common repository for both project seems currently out of reach

Just to clarify - there are no technical barriers, and in fact I use this repo with JOSM.

[bastik]

So as a common repository for both project seems currently out of reach

Just to clarify - there are no technical barriers, and in fact I use this repo with JOSM.

Yes. The other way around would also work fine: It shouldn't be hard to write a xml -> json converter, if it doesn't exist already. The issue is not so much the output, but more different philosophies in how to allow user input and "not invented here" syndrome, see #67 for an earlier discussion.

As I have not much hope for a resolution in the near future, I suggest this more modest goal, which would also be a necessary intermediate step towards unification.

[Skippern]

I sort of agree with @bastik though would prefere if JOSM actually adopts this. Currently I do as @pnorman and load this repo into JOSM.

If somebody are willing to take the time and add the outstanding sources from JOSM into this repo, and maintaining it as up-to-date as possible, i.e. re-running every x weeks, than it might be easier to sell this in with JOSM developers.

Also maybe it is time to look at dead-link/layer-health-check, and flag dead or unhealthy entries. One could also have an experimental OpenLayer where each layer could be health-checked in a web browser. There are many things we can do to make this an easier sell to JOSM dev.

[iandees]

Those are all great suggestions for pull requests. If someone were to write a dead-link check there are plenty of places to run it. Also, we have a good start at a frontend for this listing at http://osmlab.github.io/editor-imagery-index/. It would be great to have someone add a slippy-map interface to that.

[bhousel]

Closing as there is nothing actionable here. Please open individual issues or pull requests for any sources that are missing from this repository.

[Klumbumbus]

Here is an update on the current sync state between the JOSM and ELI databases.
The output of the comparison script is here: https://josm.openstreetmap.de/wiki/ImageryCompare (There is also written what is compared and what not).

The JOSM database was synced recently. There are 166 entries, which are in JOSM but not in ELI (the blue entries). JOSM has about 30 mirror entries (e.g. WMS and TMS for the same source). So for some of the 166 cases there is maybe already the TMS in ELI.

I checked all these 166 entries in the last 2 weeks if they still deliver tiles and they do (with a few exceptions, which are discussed at https://josm.openstreetmap.de/ticket/14086). So this would be a good moment to add them to the ELI database. (I personally don't plan to do this.)

[simonpoole]

@Klumbumbus I had a look at the blue entries I know something about, there are some duplicates, some stuff that is actually missing, and there are quite a few that have been removed from the ELI because they are of no practical use anymore. Now I can see that you could make an argument that you now and then might want to refer back to historic imagery, however that would seem to be somewhat at odds with presenting a user with a not overwhelming list of layers that they can reasonably use at the current time.

A simple solution would be to have a "historic" flag for such imagery, the not so simple would be something like a supersedes=xxxx attribute that would provide linkage between sources covering the same areas (or likely a combination of both).

[Klumbumbus]

One more update. The script was enhanced to compare icons and shapes.
There are 9 entries with missing shape, 19 entries where there is only a rectangle, but a more detailed shape is available in JOSM and 10 entries with outdated shapes. So in total 38 entries, which can be updated in ELI.

(I'll fix/add the 26 icons but not the shapes.)

[Klumbumbus]

This is an invitation again to update and enhance the ELI database.

The JOSM comparison script has grown up to more than 900 lines of code. It compares now basically everything comparable, see the output. All the blue and yellow entries there should be fixed in ELI.

The script output is integrated into the JOSM continuous integration system, so new differences of both imagery databases are reported there. The test runs every 6 hours.

To ease copy and paste from JOSM to ELI there is now also an geojson output of the josm database. It can be found here (condensed version) and here (readable version for copy-paste). As the geojson output is not much tested please report if there are any problems.

[pnorman]

The JOSM comparison script has grown up to more than 900 lines of code. It compares now basically everything comparable, see the output. All the blue and yellow entries there should be fixed in ELI.

Given that JOSM has improved its imagery XML editing so it works reasonably well, I wonder if it now makes sense to make ELI pull the JOSM XML and convert it to its existing formats. This probably needs discussion in its own issue.

[simonpoole]

A late note on this, straight from the horses (RMS) mouth: "but GPL version 3 does not permit relicensing to CC-BY-SA" with other words we shouldn't be using JOSM configuration for anything in ELI (the other way around is fine .....).

Not that the choice of CC BY-SA is free from problems.

[pnorman]

Did JOSM change their wiki license?

[simonpoole]

AH I found it, but no wonder I couldn't find it, it is essentially invisible (right at the bottom of the page in light light grey).

[Klumbumbus]

The 2 licenses are also written at the bottom of the front page above the heading contact in dark dark black ;)