OSMCha shows features and tags not edited in the selected changeset

[musuruan]

I'm submitting a bug report

Brief Description

OSMCha shows features and tags not edited in the selected changeset.

What is the current behaviour?

For example, in changeset #114816402, only one feature has been edited but OSMCha shows two:

The other feature was edited in changeset #114816420.

Moreover, for the feature edited in the changeset OSMCha reports two tag changes, but only one was added. The other was added in changeset #114816412.

What is the expected behaviour ?

Only changes in the current changeset should be shown.

When does this occur ?

Very often, especially with changeset made with StreetComplete.

Please tell us about your environment:

Firefox 95.0 on Fedora 35/x64_64.
Firefox 95.0 on Windows 10/x64_64.

[mmd-osm]

That’s very likely an overpass API limitation. StreetComplete is known for uploading multiple versions of an object in a very short amount of time. Overpass cannot distinguish those object versions, in case they were created at the exact same hour:minute:second, giving you incorrect results for a particular changeset.

Related discussion: streetcomplete/StreetComplete#2318 (comment)

[willemarcel]

As @mmd-osm said, it's an issue on Overpass and StreetComplete

[matkoniecz]

It is not a StreetComplete bug, as uploading multiple changesets shortly after each other is perfectly fine use of OSM API.

Though it is anyway covered by #492

[mmd-osm]

Depends a bit on your point of view. Every other editing app would merge multiple changes to the same object into a single object version upload only.

SC has its reasons to work differently. The more messages I see in my log files, the more I’m disagreeing with it.

[matkoniecz]

Hmm, adding one second delay may be actually doable. Open issue on SC issue tracker once you will be sufficiently disagreeable with it.

(no promising anything as I have no idea how this code works and about potential impact, but maybe it is possible to easily achieve more friendly behavior)

[musuruan]

That’s very likely an overpass API limitation. StreetComplete is known for uploading multiple versions of an object in a very short amount of time. Overpass cannot distinguish those object versions, in case they were created at the exact same hour:minute:second, giving you incorrect results for a particular changeset.

Related discussion: streetcomplete/StreetComplete#2318 (comment)

I do not agree. Achavi is also based on overpass but it doesn't show this behaviour.

For example, changeset #114816412:
https://nrenner.github.io/achavi/?changeset=114816412

[mmd-osm]

Let's take a look at the queries both tools are sending to the Overpass instances. First of all, you need to enable relations for Achavi, to have comparable queries: https://nrenner.github.io/achavi/?changeset=114816412&relations=true

Achavi: https://overpass-api.de/api/interpreter?data=[adiff:%222021-12-11T12:33:05Z%22,%222021-12-11T12:54:14Z%22];(node(bbox)(changed);way(bbox)(changed);relation(bbox)(changed););out%20meta%20geom(bbox);&bbox=8.0519203,45.5679148,8.052059,45.5685817

OSMCha: https://overpass.maptime.in/api/interpreter?data=[out:xml][adiff:%222021-12-11T12:32:58.000Z,%22,%222021-12-11T12:54:14Z%22];(node(bbox)(changed);way(bbox)(changed);relation(bbox)(changed););out%20meta%20geom(bbox);&bbox=8.0519203,45.5679148,8.052059,45.5685817

It seems that OSMCha is using an earlier start time: 12:32:58 vs. 12:33:05 in Achavi. Apart from this difference, I don't see other differences? Maybe this is reason for different results.

[nrenner]

This is a case of changesets overlapping by time and bbox [1] (shortened):

c114816420 s2021-12-11T12:33:23Z e2021-12-11T12:54:14Z TStreetComplete:quest_type=AddBuildingLevels,
c114816412 s2021-12-11T12:33:06Z e2021-12-11T12:54:14Z TStreetComplete:quest_type=AddWayLit,        
c114816402 s2021-12-11T12:32:59Z e2021-12-11T12:54:14Z TStreetComplete:quest_type=AddRoadSurface,   

StreetComplete opens multiple changesets in parallel, one for each active quest.

Each changeset has only one modify edit, two editing the same object [2] (shortened):

  <way id="340649056" version="3" changeset="114816420" timestamp="2021-12-11T12:33:23Z"
  <way id="314892545" version="6" changeset="114816412" timestamp="2021-12-11T12:33:07Z"
  <way id="314892545" version="5" changeset="114816402" timestamp="2021-12-11T12:32:59Z"

Using the times and bbox of changeset 114816402 (metadata, OSMCha) for the Overpass adiff query, it returns changes from other changesets that are within the same time range and bbox:

<way id="314892545" version="6" timestamp="2021-12-11T12:33:07Z" changeset="114816412"
<way id="340649056" version="3" timestamp="2021-12-11T12:33:23Z" changeset="114816420"

https://overpass.maptime.in/api/interpreter?data=[out:xml][adiff:"2021-12-11T12:32:58.000Z,","2021-12-11T12:54:14Z"];(node(bbox)(changed);way(bbox)(changed);relation(bbox)(changed););out meta geom(bbox);&bbox=8.0519203,45.5679148,8.052059,45.5685817

As the adiff query only returns the state at the given start and end time, not changes in between, version 5 of way 314892545 for this changeset is missing because of the later change in version 6.

Other examples

by same user later in December (with filter listing all other changesets in edit session) [3]:

• 115164251
• 115015641 (S3 json elements not empty as in other examples (?), changes from changeset 115015640),
  115015639,
  115015637 (node 3202699546 v3 and v4 in same second)
• 114920616
• 114880955

What OSMCha could do

1. Filter out other changesets from the fallback query result.
   I suppose this is already done in the server (see also changeset-map#103), therefore the elements property is empty in the S3 json.
2. Don't run the client-side fallback query when the server-side was successful and an empty result is expected with the current implementation
3. Request missing versions with an additional query, e.g. using timeline/retro (blog):

   [out:json];
   timeline(way, 314892545, 5);
   for (t["created"])
   {
     retro (_.val)
     {
       way(314892545);
       out meta geom;
     }
   }
   

   Overpass Turbo
   Limitation: does not work for same-second versions (like node 3202699546 v3, Overpass Turbo)

---

Footnotes

• [1] curl -s "https://www.openstreetmap.org/api/0.6/changesets?display_name=xxx&time=2021-12-11T12:32:59Z,2021-12-11T12:54:14Z" | osmium cat -F .osm.xml -f .opl
  changesets API call using changeset 114816402 metadata user (anonymized), created_at and closed_at, formatted as osmium-tool OPL (object per line)
• [2] curl -s https://www.openstreetmap.org/api/0.6/changeset/{114816420,114816412,114816402}/download | grep changeset
• [3] get later changesets by user (anonymized), manually group into edit sessions (same/close changeset closed timestamp), check each session for empty json elements using Content-Length HTTP header
  • wget -qO - "https://www.openstreetmap.org/api/0.6/changesets?display_name=xxx&time=2021-12-11T12:32:59Z" | osmium cat -F .osm.xml -f .opl
  • curl -s --head https://s3.amazonaws.com/mapbox/real-changesets/production/{114816402,114816412,114816420}.json | grep "Content-Length"