Japan OSM improvement

[planemad]

Based on observations from #116. There are multiple data issues in japan which requires a comprehensive cleanup strategy with close involvement of the Japanese OSM community.

This ticket will track all the high level conversations and strategy on improving OSM data for Japan.

Existing data issues

• Most of Japan road network has been imported from Yahoo Japan in 2011
• The road data has positional errors (5-30m) when compared to Strava GPS and includes incorrect topology that does not match satellite imagery.
  
• Roads in metropolitan areas have been realigned to the correct position, but large parts of the country are still untouched since 2011.
  
• The Bing imagery coverage for Japan is comprehensive but does not match the OSM data or Strava GPS data. There is both an offset and orthorectification errors that varies throughout Japan.
  
• There is high resolution orthorectified imagery for Japan from GSI which perfectly matches Strava GPS data and is the best imagery source available. The coverage of this imagery is not complete.
  

Data improvement strategy

1. Select project area
   1. Priority: Urban areas > Highway corridors > Rural areas
   2. Check the accuracy of orthorectified imagery and Bing imagery by comparing with GPS data. Publish offsets if required.
   3. Evaluate extent of data offset and data freshness for the area
   4. Publish findings and communicate with local mappers to initiate a mapping task
2. Road realignment
   1. Create a road improvement task using the tasking manager
   2. Verify imagery alignment using GPS data
   3. For gridded networks, attempt to offset data to match imagery. If not successful delete imported unnamed roads and retrace area
   4. Delete imported roads with no match to imagery
   5. Realign all other roads and mark as finished
3. Road tracing
   1. Go through realigned tasks and invalidate to commence road tracing
   2. Trace missing roads and tracks from imagery and mark as finished
4. Validation

Next steps

• Get feedback from Japan OSM community on above process
• @planemad to set up trial map improvement task: http://tasks.openstreetmap.in/project/38
• @planemad and @Rub21 to run task with team and closely monitor issues

cc @mapbox/team-data

[planemad]

Realignment trial report 09/16

• Today @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy realigned 15,194 nodes in the Mutsu area
• Initial pass was to realign all roads in the task area, but due to inconsistencies in the reference sources and lack of GSI satellite imagery coverage, we could not verify that our work was accurate.
• Our revised strategy was to realign only highways (motorway-secondary) based on Strava GPS which was more accurate, but this created inconsistencies wrt to the existing data
• In the interest of keeping the improvement process simple for the community to contribute to, the conclusion was to sacrifice small positional accuracies to maintain large scale data consistency.

Revised data improvement strategy

1. Focus on high priority areas based on @Rub21 tweet map
   
2. Use the primary tracing source as GSI Japan Satellite (ort) where available and fallback on Bing in other areas. Most high priority areas have GSI Satellite coverage.

3. Realign road network only in urban areas to `GSI Japan Satellite (ort)`, ignore offsets (<10m) wrt `Bing` 4. Trace missing roads based on tracing source

From tomorrow team-blr will try the above approach and report findings. A log of observations is maintained here

cc @mapbox/team-data

[lxbarth]

Wow, great progress.

In the interest of keeping the improvement process simple for the community to contribute to, the conclusion was to sacrifice small positional accuracies to maintain large scale data consistency.

@planemad - what does this concretely mean? Can you give an example for that?

[planemad]

@lxbarth to be as accurate as possible we were using Strava and OSM GPS to offset the Bing/GSI imagery for the best match, this is never perfect and results in an offset in another area. These continuous localized corrections which cannot be replicated exactly be someone else and adds to the data inconsistency since no source perfectly will match the data everywhere.

To make it simple, we're just going to use GSI Satellite or Bing as is without offset correction. We will skip realigning data with offsets under 10m from satellite imagery such as these and focus on adding missing roads.

[mapconcierge]

@planemad

I understand all steps.
Yes, GSI ortho image layer (ort) has good accuracy.

電子国土基本図（オルソ画像）MaxZoomLv18
http://cyberjapandata.gsi.go.jp/xyz/ort/{z}/{x}/{y}.jpg
source=GSImaps/ort

We have started special project team on Facebook Group with Asian addicted mappers, OSM teachers in Japan. My students will join.
https://www.facebook.com/groups/697237443754399/

[mapconcierge]

@planemad

Mutsu city area doesn't have good GSI ort images and Strava GPS.
We should choice

std 標準地図 MaxZoomLv18
http://cyberjapandata.gsi.go.jp/xyz/std/{z}/{x}/{y}.png
source=GSImaps/std

[pratikyadav]

Realignment trial report 09/17

• Today @planemad @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy realigned 6437 nodes in the Aomori region as suggested by @mapconcierge .
• The focus was only on merging/aligning highways (secondary+) to GSI Satellite imagery for pass 1.
• Tomorrow in pass 2 the team will focus on realigning the minor roads and compare the time taken with pass 1.

[unsungNovelty]

@mapconcierge, what kind of highway will the below road be? a service road, residential road or a living-street road?

https://www.openstreetmap.org/#map=17/40.785891873025996/140.7407283726662

[pratikyadav]

Realignment trial report 09/21

Today @planemad @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy realigned 3201 nodes in the Aomori region.

The focus under pass 1A and 2 was -

• to create dual carriageways on major highways,
• realign and merge tertiary
• Adding missing tertiary+ highways
• Adding missing bridges/tunnels on tertiary+ highways;

Today under pass 3, the team will realign, merge and add missing unclassified, residential roads, living streets.

[planemad]

Realignment Summary

• Team of @pratikyadav @ruthmaben @nikhilprabhakar and @ramyaragupathy conducted a multi phase data correction exercise in Aomori
• We synced daily to break the data correction in multiple steps to prioritize effort to the most critical data first to least important and improve the process from our observations
• General observations:
  • Time taken to realign an object is proportional to the number of ways and nodes
  • By merging consecutive ways and removing unnecessary nodes in straight sections, the effort can be reduced
  • In areas with dense road network, it is faster to delete roads without extra properties (name, ref) and retrace than realign
  • Focusing a task on a limited set of features allows the use of filters to remove distracting objects
  • Time per user per task should be limited to under 60 mins, else there is a feeling of stagnation
  • Retagging roads should not be done without valid reason
  • At all points the larger objective should be to simplify the contribution of any newbie to OSM in Japan
  • Tasks must be selected away from active ones to avoid conflicts

Realignment proposal for Japan
Based on these finding this is the proposal for a large scale road network correction for Japan:

• Select a project area with GSI imagery coverage and create a mapping task
• Notify OSM-JP of the task
• Make multiple passes of each area and make improvements in steps:
  • Pass 1: Realign and merge major highways (motorway-secondary); Create dual carriageways and link roads with correct oneways
  • Pass 2: Realign and merge tertiary; Adding missing tertiary+ highways and link roads; Adding missing bridges/tunnels on tertiary+ highways;
  • Pass 3: Realign, merge and add missing unclassified, residential roads, living streets
  • Pass 4: Realign, merge and add missing service roads, tracks and paths
  • Pass 5: Correct natural features polygons and waterways, add missing bridges, tunnels and viaducts
• The @mapbox/team-data will focus on completion upto pass 2 throughout Japan and support the community for further improvements

cc @mapconcierge @lxbarth @mapbox/team-data

[planemad]

Since we have Aomori completed till pass 2, the @mapbox/team-data is going to follow the similar procedure in the more dense Beppu-Oita region on Kyushu island from today http://tasks.openstreetmap.in/project/44

@mapconcierge would be great to have the OSM-JP community help with the validation of our work and feedback on the process.

[planemad]

While realigning highway junctions, make sure to fix any kinks on branching roads.

Fix any incorrect highway junctions where there is no crossing

.

[unsungNovelty]

@mapbox/team-data - Also merge and align the 4 way junctions which are split like this:

[samely]

We already finished http://tasks.openstreetmap.in/project/31#, which is a pilot task for realignment and adding roads in Chiba.

Those are the number of editions per user

User	Num OSM Objects	Num Changeset	Num nodes V1	Num nodes Vx	Num ways V1	Num ways Vx	Num relation V1	Num relation Vx
samely	77,862	333	45,220	18,416	5,856	8,370	0	0
RichRico	65,217	441	31,035	28,070	2,511	3,601	0	0
karitotp	64,290	215	32,538	21,848	4,531	5,370	0	3
Luis36995	53,561	429	24,981	21,864	2,751	3,965	0	0
calfarome	40,985	309	19,079	16,561	2,624	2,721	0	0
dannykath	38,549	653	20,531	11,481	3,563	2,973	0	1
ediyes	32,781	540	15,149	12,291	2,467	2,869	0	5
andygol	10,087	24	2,007	7,382	240	455	2	1
abel801	31,879	289	9,912	19,519	638	1,810	0	0
Rub21	1,815	31	759	711	131	214	0	0
pratikyadav	986	26	693	76	158	59	0	0
ruthmaben	767	18	236	405	58	68	0	0
jinalfoflia	311	7	240	5	52	14	0	0
nikhilprabhakar	310	5	254	9	34	13	0	0
srividya_c	294	17	167	14	75	38	0	0
ramyaragupathy	266	13	158	29	45	34	0	0
Chetan_Gowda	265	7	154	94	5	12	0	0
Total	420,225	3,357	203,113	158,775	25,739	32,586	2	10

[calfarome]

Team progress in Chiba Prefecture, Japan.

[unsungNovelty]

@mapbox/team-data : I have come across a lot of duplicate roads in the current task that we are doing (Ishikawa). For now we will be ignoring these duplicate roads and will be taking care of it in the later passes.

[jinalfoflia]

@mapconcierge, there is this tunnel like construction, what should it be tagged as?

Here is the link to the road.

[planemad]

Japan realignment progress map

https://gist.github.com/planemad/64da69b20ea9705b5673

• Stroke width indicates importance based on population/road density
• Colors indicate data status
  • Red: Not started
  • Yellow: Active
  • Brown: Pass 1
  • Orange: Pass 2
  • White: Pass 3
  • Green: Complete

[friedbunny]

@jinalfoflia Coincidentally, I've been through that tunnel many times. It's an enclosed "snow shed"/"snow shelter" (スノーシェッド or スノーシェルター) situated at a mountain pass with hills rising on either side.

Here are some pictures of this structure.

It seems like the OSM tagging is covered=yes.

[jothirnadh]

@mapbox/team-data : Whenever there is a pre-defined tunnel or a bridge such as one in the below image, we are not making any changes due to lack of evidence if the crossing is a bridge or tunnel.

[planemad]

Progress

• Task: http://tasks.openstreetmap.in/project/45
• Task area: ~8400km2
• Pass 1 completion: 3 days with 18 members (Mapbox)
  
• Pass 2: in progress
• Notes

cc @mapbox/team-data

[jinalfoflia]

@friedbunny Thank you 👍

[jinalfoflia]

As there are multiple roads for re-aligning, I tend to miss a few of them. To better keep track of the roads that I modified, I use this style that highlights freshness of data.

• Before I started editing, we can see that the roads are not highlighted.

• After aligning them, I uploaded the data. After that I downloaded the same area again, this time the edits made by the me were highlighted and thus it'll be easy to spot the roads that were missed.
  

/cc @mapbox/team-data

[planemad]

@jinalfoflia nice tip and its always a good practice to upload/download data to keep it in sync.

[Rub21]

@mapbox/team-data :
We've finished up the pass 2 in http://tasks.openstreetmap.in/project/45, let's start in new task http://tasks.openstreetmap.in/project/48

[mapconcierge]

＞ @Rub21
OK!! I've confirmed Fukushima Area.
Our students team will join today. :-)

＞ @Rub21
Please add admin permission for Project 48 to me(MAPconcierge), I can translate in Japanese.

[Rub21]

@mapconcierge , That is great :) , could you give me your osm users, to add in Task Manager, hmm but I think first you have to login http://tasks.openstreetmap.in/, after it I can give you permission to access.

[lxbarth]

[mapconcierge]

＞ @rub
MAPconcierge
http://www.openstreetmap.org/user/MAPconcierge

＞ @lxbarth
That's great!! :-)

[planemad]

Have filed an issue in iD to prompt alternate imagery based on map location: openstreetmap/iD#2826

This is an important issue to fix if we want new users to trace and realign from the correct source as we cannot expect them to know the availability of GSI and the tileurl.

cc @mapbox/team-data

[lxbarth]

Completed Tokyo Mega task

HUGE! @pratikyadav - we should diary-post this in Japanese and English. Maybe we can get @mapconcierge's help?

[shvrm]

💥 @pratikyadav!

[pratikyadav]

https://www.openstreetmap.org/user/pratikyadav/diary/36280
OSM diary entry up.

cc @lxbarth @maning @planemad @mapconcierge

[abel801]

@pratikyadav I don't know where can I translate the instructions, in a gist or do I have to have access to the task?

[pratikyadav]

@abel801 you can make a gist, and i'll update them on the task

[abel801]

@pratikyadav the translation to Spanish is Here

[mapconcierge]

I've made a Osaka Mega Task now!! Boom! :-)
http://tasks.teachosm.org/project/105

[maning]

Nagoya DONE! 🚀

Next task is Mega Osaka: http://tasks.teachosm.org/project/105
Project is available in EN, ES, JP and RU languages.

Thanks! @mapconcierge

Again, a some reminders when you edit:

• Before aligning, use the combine-highways script in JOSM to merge highways. Then, upload.
• Use todo list plugin in JOSM, in order no to miss anything.
• Changeset comment and source tag were not assigned automatically in JOSM. It works on iD. Make sure you copy the recommended changeset and source comment from the Instructions.

[lxbarth]

Nagoya DONE! 🚀

Incredible.

[pratikyadav]

We are almost done with Osaka. (21% left)

We are extending the list of mega tasks to one more by including Hiroshima and thus will be covered with most of the east coast coverage of GSI imagery.

link of geojson for Hiroshima task -> https://www.dropbox.com/s/y4l84qw2bjtrlcm/Hiroshima.geojson?dl=0

Next action

• @mapconcierge to make task in teach OSM
• @pratikyadav to add instructions in ru, es and en
• @nikhilprabhakar to post diary entry about the Japan progress

[mapconcierge]

I found this interest area in Nara city in OSAKA mega task.
But I guess, Under construction area. We shouldn't modify.
http://www.openstreetmap.org/edit?editor=id&#map=18/34.68176/135.80939

[mapconcierge]

＞ @pratikyadav
Okay, I will make a Hiroshima Task soon.
Wait a moment... :-)

[wdolphin]

I found someone edited wrongly in Nara city, Osaka mega task. So I fixed.

[hackablemeister]

This area is off to the left side over a wide range in Matsuzaka city.
http://www.openstreetmap.org/edit?editor=id&#map=17/34.54174/136.55632

[hageshi1]

I have no idea , how to fix? :-)

http://www.openstreetmap.org/edit?editor=id&#map=19/34.70021/135.81981

[mapconcierge]

＞ @abel801 Excelente!!

＞ @planemad
Hiroshima Task!! http://tasks.teachosm.org/project/111

[pratikyadav]

@mapconcierge Great 🎉
@hageshi1 Seems like a bridge connecting to a nearby parking area. Will be nice to add a layer=1 tag to this road.

[mapconcierge]

Thanks!! @pratikyadav
I've confirmed!! :-)

[mapconcierge]

• GSI ortho imagery for Japan osmlab/editor-layer-index#111
• added GSI imagery for Japan osmlab/editor-layer-index#117

[planemad]

@mapconcierge many thanks for all the help from the Japanese community in the task. We will continue to interact on our diary posts and the mailing lists.

No immediate actions here. Amazing job everyone!
🚀 🚀 🚀

cc @mapbox/team-data

[talllguy]

@mapconcierge I stumbled across this highway system overlapping water in #103. Since it is related to Japan, I thought you might want to check it out. Should that street network be on top of that water feature?

map link

[jeffrey-friedl]

Is this project still ongoing? In many cases it's possible to use road-edge data from the Japanese government to guide extremely accurate updates that are not subject to image misalignment or elevation-change skew. I've been doing this around Kyoto a lot, but it'd be nice if more folks could do it.

[miurahr]

@mapconcierge I stumbled across this highway system overlapping water in #103. Since it is related to Japan, I thought you might want to check it out. Should that street network be on top of that water feature?

There were actually a many roads in this area. When the earthquake / tsunami in 2011 caught here, many of this area, Kamatani and Nagato region, fully submerged.

We should mark these roads as non-usable or lost.
Is this an enough answer for #103 ?

[pratikyadav]

Is this project still ongoing?

@jeffrey-friedl No, this is not an on-going project. But I would love to learn more about road-edge data from the Japanese government. Can you provide some relevant link?

[jeffrey-friedl]

You may need to create a (free) account to access this link, but the data is downloadable at:

http://fgd.gsi.go.jp/download/kibansel.php

It's the "道路縁" field. The data is difficult to work with, but in the end you can turn in into a GPX/KML file that ends up drawing the edges of roads. Using that as a guide, a human can infer road paths.

Here's an example that you can load into an openstreetmap editor or Google Earth or other visualizer: http://regex.info/tmp/road-edge-sample.gpx

The data is usually very high quality, but occasionally it is out of date due to recent construction. Also, in some areas the data seems more like an old planned road, rather than actual construction. But mostly it's spot on correct.

I came across your project because it updated a road I had used this technique with. The project update broke map correctness because the GSI image from which your update was based was old... the road had since been move. (I'm still trying to get that update, changlist #35265050, reverted.)