Attic output contains later introduced tag

[tyrasd]

The following query

[out:xml][timeout:25][date:"2012-10-01T00:00:01"];
way(30713396);
out tags;

which basically requests version 7 of way 30713396, returns the tag tunnel with a very suspicious value (see later). The way only recently got this tag added, see http://osmlab.github.io/osm-deep-history/#/way/30713396. Looking at the returned content, the value of the tag appears to contain quite strange data: Running curl http://overpass-api.de/api/interpreter?data=%5Bout%3Axml%5D%5Btimeout%3A25%5D%5Bdate%3A%222012-10-01T00%3A00%3A01%22%5D%3B%0Away%2830713396%29%3B%0Aout%20tags%3B | hexdump -C produces:

[…]
00000150  67 20 6b 3d 22 74 75 6e  6e 65 6c 22 20 76 3d 22  |g k="tunnel" v="|
00000160  ff 20 22 2f 3e 0a 20 20  3c 2f 77 61 79 3e 0a 0a  |. "/>.  </way>..|
00000170  3c 2f 6f 73 6d 3e 0a                              |</osm>.|

i.e. the string starts with 0xff which doesn't seem to be valid utf-8, is it?

[mmd-osm]

Imho 0xff is being used as an internal identifier for a void tag value (see here). It's a bit strange that the tag contains 0xff 0x20 (instead of 0xff) as value, so that it doesn't get filtered out. If we're lucky, we can also filter out 0xff 0x20 without any changes to the underlying database. Worst case could be that 0xff should have been written to the database instead of 0xff 0x20 and we need a db fix/rebuild.

In any case, this looks like a bug to me. Unfortunately, attic issues are the most difficult ones to analyze, so that's another one for Roland...

[mmd-osm]

More candidates:

[timeout:25][date:"2012-10-01T00:00:01"];
(way(24527829);
 way(27429264);
 way(28419711);
 way(42365448);
 way(43789072);
 way(57687991);
 way(60855715);
 way(60966427);
 way(85705356);
 way(85741442);
 way(85826994);
 way(90144168);
 way(92904180);
 way(93190272);
 way(97329747);
 way(106587533);
 way(111142495);
 way(114275781);
 way(125872832);
 way(133333499);
 way(137844783);
 way(150410708);
 way(162413477);

 rel(223991);

);
out tags;

  <way id="42365448">
    <tag k="layer" v="-1"/>
    <tag k="CEMT" v="IV"/>
    <tag k="name" v="Po di Volano"/>
    <tag k="tunnel" v="� "/>    <!-- <-- corrupt data / tunnel should not be in result -->
    <tag k="waterway" v="canal"/>
  </way>

Good query to find this stuff: http://overpass-turbo.eu/s/cwv

[mmd-osm]

Further analysis:

dumpfile ... --attic-way-tags-local | grep $'[\xff][\x20]' | cut -f 2 | sort | uniq -c > way_candidates.txt

• A total of 141497 ways affected: http://dev.overpass-api.de/issue241/way_candidates.txt
• 6680 rels are affected: http://dev.overpass-api.de/issue241/rel_candidates.txt
• No nodes are affected

Examples:

[date:"2012-11-01T00:00:00Z"];
way(100019095);out tags;
rel(101110);out tags;

[drolbr]

Fixed in 919f06c .

A rebuild of the database could be avoided if we told every using function to ignore tag with those values. But due to bug 276 it makes sense to rebuild to database anyway.