[SF Buildings] Tracing buildings in San Francisco Peninsula

[srividyacb]

Objective: Add missing building footprints in SF Peninsula

Prior Tasks: #135 #136

**Focus Area**: [1290 sq.km](https://gist.github.com/732acea19b0336e418d0) **Estimates**: [~1,79,000 buildings](https://github.com//issues/141#issuecomment-160596321) **Duration**: **[18 days](https://github.com//issues/141#issuecomment-160604840)** **Progress**: [173447 buildings](https://github.com//issues/141#issuecomment-164547016)

Open Datasets availability : Overview

Team: @mapbox/india-office and @mapbox/peru-office

Mapping Sources

Source	Description	Quality	License
Mapbox Satellite	Imagery	High	✅
Bing Imagery	Imagery	High	✅
Mountain View and Palo Alto	Building Footprints	High	❔

Subtasks

Tasks	Description	Start	End(ETA)	Validation
Mountain View	Adding buildings	11/17	11/24	Completed
Menlo Park	Adding buildings	11/25	12/03	Completed
Missing Buildings	Adding buildings	12/04	12/28	Completed
Missing Buildings in SF Peninsula	Adding buildings	01/04	01/06	Completed
Filling gaps in SF Bay	Adding buildings	01/08	01/18	Completed
Palo Alto/Cupertino import cleanup	Realign buildings	Active

Workflow:

1. Pick a task from the Tasking Manager.
2. Load the Bing or Mapbox Satellite imagery to JOSM.
3. Install JOSM Buildings Plugin for easy tracing.
4. Inspect the imagery to find the missing buildings.
5. Add only missing buildings.

Shortcuts

• B to start with buildings tool
• X to extrude building shapes as per the imagery.
• Shift+Ctrl to rotate buildings.
• Q to orthogonalize buildings
• Shift+J to merge polygons

Filters

Add a inverse filter building=yes to highlight only buildings in the downloaded area.

Changeset Comment: Added buildings https://github.com/mapbox/mapping/issues/141
Source: Bing/Mapbox

[srividyacb]

We compared the building coverage using publicy available data in Palo Alto. Based on this analysis and visual check with Bing imagery, only a few buildings and building details are missing in OpenStreetMap.

Using Palo Alto public data building as baseline, we ran the analysis for building coverage estimates.

How did we do this?

• We converted the Palo Alto building polygons to points/centroids to improve the speed in processing.
• We ran a point-in-polyogn analysis in QGIS to count the number of buildings within each block of the TM.
• Most of Palo Alto lies within 14 TM blocks. The table of raw counts below:

Block	Building Count
1	818
2	1283
3	1367
4	1313
5	809
6	1158
7	1204
8	1263
9	1136
10	1095
11	1118
12	1038
13	922
14	784

Findings

• Building count range is from 784 to 1367 for each block. The average is 1,093 per block.
• For 164 blocks we initially identified as less building coverage area, an estimate of ~1,79,000 buildings should be added to SF peninsula.

Trial Run

• Set up the task in OSM-US tasking Manager.
• @srividyacb @ramyaragupathy to run a trial trace to calculate the time needed to add buildings per block.
• Calculate how much time it take for a 10 members team to finish the building improvement.

[ramyaragupathy]

to run a trial trace to calculate the time needed to add buildings per block.

After loading the 164 blocks onto Tasking Manager, it was split into 667 blocks.

We ran a trial trace picking blocks where there were no buildings. Here are the number of buildings added in this trial run.

Looking at the numbers above, an average of 120 buildings can be added per person per hour.

Calculate how much time it take for a 10 members team to finish the building improvement.

As per our open dataset building analysis, we need to add ~1,79,000 buildings in SF peninsula. Considering 10 member data team working in it, here are the time estimates:

Per person : 120 buildings per hour

Buildings to be added	Total buildings per person	Number of people working	Number of hours	Number of buildings per day	Estimated number of days to finish task
~1,79,000	120	10	8	~9600	18

[srividyacb]

We have been investigating on the availability of open building datasets for SF peninsula. Here is an overview of the usage terms.

City	Dataset Portal	Dataset Availability	License
Palo Alto	KML	OSM coverage is good
Redwood City	http://webgis.redwoodcity.org/community/	Yes and already imported
San Mateo	https://data.smcgov.org/browse	No
Sunnyvale	CommunityDevelopmentDivisions	No
San Bruno	No portal
Millbrae	No portal
Daly City	No portal
Menlo Park	https://data.menlopark.org/browse?q=building	No

cc @ramyaragupathy @maning @planemad

[planemad]

Progress

Date	Number of buildings
Dec - 4	4053
Dec - 5	3608
Dec - 6	1
Dec - 7	1811
Dec - 8	3202
Dec - 9	1334
Dec - 10	2084
Dec - 11	3551
Dec - 12	870
Dec - 14	5144
Dec - 15	12183
Dec - 16	10418
Dec - 17	9768
Dec - 18	9533
Dec - 19	2050
Dec - 20	551
Dec - 21	4125
Dec - 22	8992
Dec - 23	5393
Dec - 24	8805
Dec - 25	2835
Dec - 26	1242
Dec - 27	901
Dec - 28	2433
Jan - 04	8023
Jan - 05	12355
Jan - 06	7385
Jan - 07	5027
Jan - 14	9378
Jan - 15	21730
Jan - 18	4659

[ramyaragupathy]

Knocked off those blocks already covered in #135 and #136. This marks 35% of the task complete!

[pratikyadav]

Use this building layer to ensure that all buildings are covered in the blocks without downloading the data.
tms[99]:https://api.mapbox.com/v4/pratikyadav.332dd523/{zoom}/{x}/{y}.png?access_token=pk.eyJ1IjoicHJhdGlreWFkYXYiLCJhIjoiMTA2YWUxNjRkNmFmZGQ4YzAxZWFiNDk0NDM1YjE1YjAifQ.4P6N5dNmA_WQXd3BsJvu5w

This will make sure that we are not missing buildings at the corners of the task boundary.

[jothirnadh]

The houses in the above image are transportable or temporary ones, so we are not tracing such kind of buildings/houses.

[ramyaragupathy]

Now that the tracing and validation is complete for #12, we are focusing on project #14.

[karitotp]

In the task I was working I found this Apple Campus that does not match Bing satellite.
What to do in this case?

[talllguy]

@karitotp Cool! I believe you've found the Spaceship campus!

It is under construction, so you may want to trace a landuse=construction around the whole area.

More Info

[abhisheksaikia]

We are focusing now on project #15 in SF building tracing as the Beverly Hills Task is now on hold.

[RichRico]

I found some imported buildings that not match with Bing or Mapbox satellite imagery. Here are the picture and the changeset.

https://www.openstreetmap.org/way/377379461/history

[Rub21]

@planemad @maning @srividyacb : How is @mapbox/india-office working in places where the buildings are offset? . Which decision are you taking?, we found some places where the buildings are offset and they are created in 2011, 2012 some of them are recently editions. We are consider realign those buildings, however we need to be agree all the data, What do you think?

[Andygol]

p/chat with @Rub21

in the same area I faced with buildings without offset and buildings that were shifted - so I decide realign all shifted buildings
sometime shape of imported building is mismatched to the real building on the imagery
/cc @mapbox/india-office

[ediyes]

Here https://www.openstreetmap.org/way/240425868/history#map=19/37.47142/-122.25689 I found many buildings are offset that were created two years ago.

What should i do?

cc. @Rub21 @Andygol

[Andygol]

@ediyes
I realigned buildings in such case

[abhisheksaikia]

In relation to the comment by @ediyes above, I have noticed significant amount of edits in the Palo Alto area by one certain user: oldtopos. The link to a changeset containing mass edits in buildings in the area: https://www.openstreetmap.org/changeset/17095207. In most cases there is an offset of the buildings with the Bing imagery as pointed by @ediyes , but in some cases there is also the problem of buildings having different footprints compared to Bing imagery and also buildings not being present in the Bing imagery.

The footprints of buildings with addresses which are not present in Bing Imagery

Buildings with significant offset with the Bing imagery and also buildings with footprints that differ quite radically from the Bing imagery

Next actions:

• Should these buildings be aligned and corrected according to the satellite imagery or should they be left untouched?
• Possibly contact the user: oldtopos asking him to look into his edits?

cc: @mapbox/team-data

[abhisheksaikia]

An invalid coastline boundary is also observed inside residential areas in Palo Alto area. The changeset comment for the incorrect coastline is: https://www.openstreetmap.org/changeset/33675094.

Incorrect coastline inside residential area in Palo Alto

cc: @mapbox/team-data

[planemad]

@abhisheksaikia

• Comment on the user's changeset to find the source date of the building footprint data. It will help establish wether the imagery or data is outdated
• Comment on the user's changeset with incorrect coastline to ask if any help is needed with the cleanup

Lets give a day for the users to respond, failing which lets have a diary post highlighting the issues and that we will proceed with the cleanup if there is no conflicting opinions. In the meantime, lets skip these areas for now.

cc @mapbox/team-data

[abhisheksaikia]

@planemad The following actions have been taken on the issues:

Comment on the user's changeset to find the source date of the building footprint data. It will help establish wether the imagery or data is outdated.

Commented on the user's changeset: http://www.openstreetmap.org/changeset/17095207. Until there is a reply let us hold off on realigning buildings in the Palo Alto area.

Comment the the user's changeset with incorrect coastline to ask if any help is needed with the cleanup

@maning is looking into the issue and the possibility of reverting the changeset.

cc @mapbox/team-data

[maning]

incorrect coastline is: https://www.openstreetmap.org/changeset/33675094.

I investigated this data today. It looks like the landuse=residential, place=neighborhood was orthogonalized by mistake!

Based on the changeset info, the last editor was Sarr_Cat, but that is not the full story. The OSM changeset view does not tell you who was the last editor of the existing nodes. I used the augmented diff visualizer to check this area.

The visualizer clearly shows changes in a given view. In this case, it was confirmed that the polys was edited by Sarr_Cat.

But, if you select a specific node, you see another user.

Investigating further, you see that this user moved the node by a few hundred meters. If you can read lons/lats, you will notice this. See 3rd and 4th decimal places.

Now we know who the actual editor and changeset who messed this up: https://www.openstreetmap.org/changeset/36659139

Unfortunately a full revert is not possible because the editor did a lot edits: 420 - created, 1380 - modified and 157 - deleted. Some are good ones. I did a partial revert by selecting only the nodes affected. All good now.

Faith in humanity restored.

Takeaways

• Do not blindly square any feature within your task. There are ways to avoid this.
  • Run JOSM filters to avoid touching other data.
  • If you have to "square" many features, select only those you modified. In JOSM, click Search and type type:way modified or building modified .
  • Always review your edits before upload, run the JOSM validator, review what features you modified before hitting the Upload changes button.

• We need better tools to get a better picture of what was changed (created, modified and deleted). Augmented diff visualizer gives us this view. Fortunately, we have such a tool under testing. See: http://ec2-52-90-240-32.compute-1.amazonaws.com/

cc/ @abhisheksaikia @planemad @Rub21 @dannykath

[planemad]

Nice sleuthing @maning 🔍

Wonder if we could catch this with osmlint? @Rub21

[abhisheksaikia]

In case of invalidated blocks in the SF task-3, it has been observed that a lot of invalidation was due to alignment issues. As of now, we are not re aligning the buildings. So, if there are missing buildings in a invalidated block and other issues those can be corrected please go ahead doing it. But no re-alignment should be carried out until further update on this.

cc @mapbox/team-data

[talllguy]

I have seen other questionable edits by that Sarr_Cat user. Noted.

[abhisheksaikia]

SF building tracing Summary

Objective: Add missing building footprints in SF Peninsula

Prior Tasks: #135, #136

Focus Area: 1290 sq.km

Estimates: ~1,79,000 buildings

Duration: 63 days[17/11/2015-18/01/2016]

Buildings Added: 173447 buildings

Team: @mapbox/india-office and @mapbox/peru-office

Subtasks

Tasks	Description	Start	End(ETA)	Validation
Mountain View	Adding buildings	11/17	11/24	Completed
Menlo Park	Adding buildings	11/25	12/03	Completed
Missing Buildings	Adding buildings	12/04	12/28	Completed
Missing Buildings in SF Peninsula	Adding buildings	01/04	01/06	Completed
Filling gaps in SF Bay	Adding buildings	01/08	01/18	Completed

Missing buildings added

Mapper	SF Task-1	SF Task-2	SF Task-3	Total buildings per person
aarthy	0	384	0	384
abel801	10070	0	0	10070
andygol	2661	1421	15	4097
calfarome	8041	4259	3814	16114
Chetan_Gowda	1365	2035	559	3959
dannykath	1661	2701	2698	7060
ediyes	18340	0	5047	23387
jinalfoflia	1601	723	0	2324
Jothirnadh	4450	622	199	5271
karitotp	3152	2770	2634	8556
Luis36995	13610	2936	4336	20882
lxbarth	383	0	0	383
manings	3944	0	1	3945
mikelmaron	2	0	0	2
nikhilprabhakar	7913	1268	1041	10222
oini	2317	0	72	2389
PlaneMad	1292	0	0	1292
pratikyadav	2382	515	155	3052
ramyaragupathy	2573	1283	160	4016
RichRico	5713	5856	6537	18106
Rub21	565	0	0	565
ruthmaben	1502	457	898	2857
saikabhi	4676	1493	407	6576
samely	4734	3435	7129	15298
srividya_c	1943	632	65	2640
Total	104890	32790	35767	173447

Observations:

• Imagery is outdated in some cases, where new buildings have cropped up. In some of these cases, the new buildings were deleted and the building footprint was traced according to the prevailing satellite imagery. One must careful while coming across such cases. One such instance was the new Apple spaceship campus in OSM which was absent from Bing imagery.
• Large scale imports were present in some areas in SF. While coming across such cases, it is advisable to leave those areas untouched and start aligning them in a separate cleanup task.

What more needs to be done:

• A separate cleanup task needs to be prepared for the SF area to align all the buildings which were part of some previous imports.
• Publish comms on the task in the form of a blog and diary post.

Next actions:

• @ramyaragupathy to publish a blog and a diary post on the task completion.
• Add the project to completed projects in mapping wiki and update the final statistics of the project.
• @ramyaragupathy and me to put up a cleanup task by the end of the week.

/cc @mapbox/team-data @planemad @lxbarth

[planemad]

@mapbox/team-data based on reports of misaligned imported buildings #141 (comment) we have a cleanup task to focus on these areas with misaligned footprints.

Task http://tasks.openstreetmap.us/project/20

Important

• The cleanup areas cover the Apple campus at Cupertino and other locations which may have new developments not in the imagery [SF Buildings] Tracing buildings in San Francisco Peninsula #141 (comment)
• Only cleanup footprints that have not been edited in the last two years. Relevant JOSM filters are included in the task instructions.

[geochetan]

When proceeding with the cleanup task: http://tasks.openstreetmap.us/project/20 , this filter must be applied: timestamp:2005-12-31/2014-12-31 building .This filter must be added to hide the recently added buildings (from 2015 onwards) as these buildings might be added by local mappers from the ground even though they are absent in satellite imagery e.g. Apple Spaceship Campus and we dont delete any recent building inadvertently.

[lxbarth]

@mapbox/team-data based on reports of misaligned imported buildings #141 (comment) we have a cleanup task to focus on these areas with misaligned footprints.

@planemad - can we break this work into a separate ticket and close here? The misaligned footprints aren't due to any mapping efforts led up on this ticket here.

[abhisheksaikia]

@planemad - can we break this work into a separate ticket and close here? The misaligned footprints aren't due to any mapping efforts led up on this ticket here.

@lxbarth Separate ticket for the cleanup task is created: #152.

This present ticket will be closed after the comms are up and validation for SF task-3 is complete.

cc @mapbox/team-data

[ramyaragupathy]

Comms are up and the validation is 💯 done.