Creating OSM Features in iD from ArcGIS Services

[slibby]

The Goal

Knowledgeable users can easily contribute reviewed and authoritative data to OpenStreetMap with the iD editor using data extracted from an ArcGIS Map or Feature Services. This issue serves to introduce the concept and ask for feedback from the primary iD contributors and maintainers before submitting a PR (as requested in Contributing.md).

Background

(edit: the use of authoritative in this statement should be reconsidered)
Many Esri customers using ArcGIS are authoritative data creators and maintainers. In many areas, densely detailed user-submitted data is available in OpenStreetMap, but in many others, there is a limited density of data even though easily-accessible authoritative data exists. Currently, to make use of these datasets OpenStreetMap contributors would need to export the ArcGIS Map Service to a Shapefile or PostGIS database and then use desktop tools such as JOSM to edit and upload to OpenStreetMap, or go through a lengthy import process to import a selected set of data which is not already available in OSM or which is more precise or more accurate than current OSM features.

This proposed feature aims to make it easier for volunteers to add features queried from the GeoServices REST API (ArcGIS Server Map Services and Feature Services) to OpenStreetMap using iD. By connecting directly to the ArcGIS Service form an iD session, contributors are getting the most up to date data and also can more easily contribute this data without the need for complex workflows. The tool will also allow users to match up existing attributes from the source data (for example, house numbers on building footprints) to relevant OSM tags to make adding authoritative attribution straightforward along with geometries.

Solution

This feature will add the capability for contributors to select “Add GeoService Layer” and then provide the URL to an ArcGIS REST Service or Feature Layer (ex. http://maps2.dcgis.dc.gov/dcgis/rest/services/DCGIS_DATA/Transportation_WebMercator/MapServer/5). Before loading, the contributor will be able to specify whether specific attributes should be mapped from the source layer to specific OSM tags, or whether any other static tags should be added to all features. The iD editor will load the vector features of the current map extent as temporary objects in the browser window (not immediately submitted as edits to OSM).

The contributor can then select objects generated from the ArcGIS Layer, inspect and alter any tag values or geometries and submit them as edits to OSM (additions or replacements for features). They can then save their changeset, contributing these new objects to OpenStreetMap via iD's existing functions.

Implementation

An initial working version of this feature has already been developed using a branch from the iD GitHub repository and is nearing readiness for submitting a PR to the master iD branch. We have attempted to address potential concerns about ease of import and user experience in the initial version.

cc @jgravois @mapmeld @ajturner

[jgravois]

well said @slibby! in general the flow would be:

1. user identifies a public live GeoService/Esri service with features that would be appropriate to import into OSM.
2. user provides url in iD and confirms that its license allows the data to be imported.
3. features are fetched. optionally a preset is selected and corresponding attributes are mapped accordingly.
4. user review is required to confirm validity of individual features (and avoid bulk loading of unverified data)

[Nakaner]

Imports are a tricky subject and can go wrong easily.

Are you aware that imports of third-party data must be discussed and documented properly beforehand?

The goal of OSM is not to be a large collection of so-called "authoritative" data. Our goal is a community-driven project which picks out the best pieces from all sources. Unfortunately, being an "authoritative" data set does not mean that the data set is free of errors.

Importing is not just uploading a shape file. First of all, the license of the data has to be checked. OSM has some requirements to the license of the data.

Often some data exists already in that area. This data has to be conflated, the history of the existing features should to be kept. The imported data should be checked if it fits between the existing OSM data. If OSM data has an offest due to the usage of the best available but still suboptimal imagery (Bing with offset of a few meters), either the OSM data or the "authoritative" data has to be modified. For example, most trees do not grow in the middle of an road (except traffic islands ;-)). Those trees have to be checked by the users who import the data. Maybe the roads has been removed or it is a new road and the authoritative data is outdated? There is a wiki page about this problem which focus on mechanical edits but it partially applies to imports, too.

From my point of view, importing data should still has a minimum level of difficulty. People should be aware that they are doing something "special". I fear that offering a too easy way to import data will lead towards to much undiscussed imports which have to be reverted. This angers those who care for the data and those whose imports are reverted.

I would like to ask you to get in touch with the community on the Imports mailing list before investing to much time into a feature which might attract endless anger. Please also take the time and read a few import discussions on that mailing list beforehand. You will quickly learn what the main problems of imports are.

[planemad]

This looks like a super powerful feature and i'm excited by how this could greatly enrich the OSM data with unique identifiers and enables more OSM data users to contribute back to the project easily.

The biggest risk is probably the potential to replace existing higher quality data that is contributed by volunteers with authoritative but lower quality information as @Nakaner outlines. This is the hardest part, and a good UI or workflow for imports should allow the user to make a clear assessment of data quality of the various sources in view to conflate successfully.

[amandasaurus]

"authoritative data" in OSM means surveyed data from the local community. You're talking about "government data". Often gov data is high quality, but not always. iD is currently very good for adding authoritative data (in the OSM sense).

It's very interesting to see this, and I do think there are ways to improve imports, but I worry that this suggestion will make it too easy for someone to just blitz some data into OSM, without looking at it first. I don't think iD has (any?) good validation functionality, so this could result in a lot of bad data where someone adds the data on top of existing OSM data.

I worry about people not knowing about copyright. Many people have some strange (and wrong) ideas about copyright, so a click through "yeah this is OK" could lead to copyright problems in the OSM data.

You'll have to find a way of ensuring that there has been a discussion on the OSM imports@ lists for every use of this..

[slibby]

Thanks for the immediate feedback, @rory @planemad @Nakaner

We have tried to consider the risks of making it "easier" to add existing data to OSM with this feature from the beginning, and to me, it breaks down to several different areas where we might provide a "brake" or impedance that limits users and tries to prevent this feature from being used for that purpose.

1. Copyright

I am by no means an OSM Copyright Discussion expert, but our initial plan was to have as clear of a warning or click-through as possible, requiring users to validate the availability of source data before they can add the service to the map. I believe we have the same risks currently with the TMS or XYZ Tile Layer scenario - users can add any imagery layer without any proof of copyright, and use that to digitize their features, then submit the data with no record of the original source layer. I don't think this is necessarily a perfect approach but in a scenario where as @rory says many people aren't even aware of the need for copyright checks on data, my preference is to get more and better data into OSM (where it is needed, especially) and trust our community to moderate the copyright concerns.

Another idea that I have floated is to keep this feature hidden behind a feature flag or a warning screen of some type within the application. I do think that the less-experienced iD editor would be unlikely to stumble across this and add a set of data from a Map Service without clear intent, so hopefully it is already seen as a tool for intentioned and educated editors, and we should do our best to provide them with clear guidance, limitations, and restrictions on copyright.

2. Overlapping existing data

I think we're wide open to suggestions on this, one idea was to build some logic into the tool to disallow saving of edits if certain conditions are met (i.e. new building footprints overlap existing building footprints from OSM, new roads are not connected to the existing road network or other "checks" that could be implemented). In some cases where data has been provided by users, government or official-source data may be significantly better, so we would like to still offer the option to replace a single or multiple features if we can make sure that the users aren't going through this haphazardly.

3. Number of features

My feeling is that it would be fair to limit the number of features that can be added with this method to a certain number or a certain geographic area. That is difficult to do when you could be adding logging roads over a large area vs. a hundred point addresses in a small neighborhood, but some method to limit the number of features imported would be essential. Currently, the tool has two import methods - import globally or import in current view, which covers the need to import large geometries in some areas with the desire to focus edits locally where possible.

Last, personally, I would like to try to keep this feature from being labeled as an "easy-import" feature or being directly associated to the concept of OSM bulk imports though that is likely inevitable. There is a lot of history and discussion around bulk imports with a history of compelling arguments from different points of view, and this is intentionally not a tool to make it easier to execute the kind of imports that get justifiably reviewed by the OSM import list - large geographic areas of a specific feature type (all TIGER Lines, all buildings in NYC, etc.) or specific areas with imports of several feature types (MassGIS, city of Friday etc.).

Some example use cases that we have envisioned for this:

1. Adding building footprints for a new housing development in an area with existing OSM building footprints from the community or prior imports
2. Adding trails and access paths from a city government for a park or area without existing data of those types
3. Adding water bodies to rural areas that are available from a county Map Service but do not have a community nearby to map them.
4. Allowing a not-fully-mapped municipality in an area not frequently covered by OSM to provide their own data to the community following their own review.

[SomeoneElseOSM]

Isn't the lack of response on Esri/arcgis-osm-editor#104 (comment) at least slightly worrying here? Surely until that is answered any "Arcgis services" should be considered unusable, at least within iD, where we don't expect users to have as much knowledge about the relationships between different bits of OSM data as we might elsewhere.

[woodpeck]

Imports can have a number of problems that must not be ignored. The concept of "data import because the OSM community is small" is particularly problematic; for OSM to thrive it is necessary that you have people on the ground who take ownership and who maintain and amend the data. Imports work well when they are done with and for a local community; they are a liability when done in lieu of a local community. This is a social problem rather than a technical one; people often don't understand OSM enough to get that. Care must be taken not to send out the wrong signals.

But I would like to focus on the technical side for now and mention a few typical issues that will need addressing in this context. When people have copied information from their GIS systems into OSM in the past, the following problems have occurred frequently:

• Properties from the existing system copied into OSM that made no sense in OSM, for example we have objects in OSM that have tags like AREA (in sq m), or lots of information that is internal to whoever maintained the data and useless in OSM. This can include custom identifiers/keys; generally the importing of ID numbers to link OSM data with external data is frowned upon (unless the ID numbers are somehow visible when you visit the object in situ).
• Over-noded geometries. We have had lots of building imports where - for whatever reason, potentially a projection issue - a rectangular building suddenly had a two or even three digit number of points. That is undesirable and was often the cause for lots of cleanup work by the community (if one existed).
• Duplication of edges (bad) or points (worse). Where a mesh-type data set (e.g. landuse) was imported, it would sometimes happen that along a shared border, both neighbouring objects would have their own OSM nodes instead of sharing nodes, because the original "simple features" data set didn't do topology. For longer shared edges, OSM would even expect the neighbouring polygons to be dissolved into multipolygons that share a "way" object as their common boundary.
• Apparent disregard for already-existing features in OSM. This can come in two colours; one would be to replace an existing object with a newly imported one without a thorough examination (you might be replacing a better building outline with one of lower quality, but you might also replace a low-quality building outline with a better one and overlook that the low-quality outline had tags about the roof shape, thereby losing that information). The other would be to add new objects (e.g. there were no buildings at all in OSM so I don't have to check) but ignore the fact that the newly-added data collides with other features - if your buildings sits on a road then it's on you to either fix the road or fix the building.

[slibby]

@SomeoneElseOSM Yes, I do lament the lack of response on that issue but I also know it is still being actively pursued on the Esri side. However, that issue is specifically related to a service provided by Esri directly (Esri has to contend with the data license for all the imagery), where this issue's functionality is intended to allow the addition of data provided by other organizations (municipalities, states, etc.) who are simply using Esri software to expose the data. It's akin to adding support for adding WFS layers (from anywhere) to iD.

[jgravois]

@woodpeck, thanks for the specific and candid feedback and concrete examples of challenges with previous imports!

Properties from the existing system copied into OSM that made no sense in OSM

our current design for this only grants an opportunity to map attributes when users have selected an OSM preset and then limits users to mapping to existing relevant tags. the information you've provided about outside identifiers is helpful and we can ensure that they are stripped as well.

Over-noded geometries.

we can definitely pursue checking vertex counts and simplifying/squaring when necessary.

Apparent disregard for already-existing features in OSM

The tool in its current form certainly doesn't allow for wholesale replacement and imported features must be reviewed individually prior to inclusion in a changeset. our hope with this was to explicitly require QA with respect to existing data.

[mapmeld]

This is an ongoing process, but I'll write about some protections which are baked into this tool for a deliberate and careful process of bringing in this data:

• licenses - before importing data from a new source or new license, the user must check a box that the source and license are compatible with OSM. The messaging and information here could be improved to educate users about what they're doing.
• presets - as mentioned earlier in the thread, helping users import their data with meaningful tags
• working in small areas - partially due to browser limitations, you can't import a whole city at a time... for building footprints you would be zoomed into a neighborhood level
• individual approval - currently the user is approving features individually before they are added to the Save dialog - I'd like to make this review process more clear in the next few days
• overlap - for features where this is relevant (currently buildings) we avoid importing a polygon if it would overlap with an existing polygon on OSM.

Still in concept phase, but ultimately I would like to have this work a little like HOT's Task Manager - an expert sets up the service, area, license, and data tag mapping, and then individuals can come and edit small areas to that standard.

[simonpoole]

Most of the technical and other issues have already been touched on (well maybe nobody has pointed out that adding data also implies cleaning up if the whole thing is a screw up which is a lot more work than in a typical layered GIS system, but I digress). so I just want to touch on point 1. Copyright by @slibby.

In, in the mean time many years of, my experience the absolute last people that are able to understand if their own terms of use are compatible with both our contributor and distribution (ODbL) terms are typical government GIS offices (which I assume is the market ESRI is targeting here), self declaration as only verification is not a good idea.

Further it is not true that we do not police background layers and similar that can be used indirectly to add data by tracing of features or by individual feature copying (currently mainly limited to JOSM). We both vet the most used sources of such layers and have technical safeguards available in all major editors including logging of used sources. Naturally this is more difficult than detecting direct imports, but that is no reason to take a stance of you can't do it, so we will ignore you. As has already been pointed out ESRI itself is a good example of our processes working.

If you have proof of non-approved sources being used, please forward the information to the DWG data@osmfoundation.org which will take the appropriate steps to rectify the situation.

[Nakaner]

@mapmeld wrote:

This is an ongoing process, but I'll write about some protections which are baked into this tool for a deliberate and careful process of bringing in this data:

• licenses - before importing data from a new source or new license, the user must check a box that the source and license are compatible with OSM. The messaging and information here could be improved to educate users about what they're doing.
• presets - as mentioned earlier in the thread, helping users import their data with meaningful tags
• working in small areas - partially due to browser limitations, you can't import a whole city at a time... for building footprints you would be zoomed into a neighborhood level
• individual approval - currently the user is approving features individually before they are added to the Save dialog - I'd like to make this review process more clear in the next few days
• overlap - for features where this is relevant (currently buildings) we avoid importing a polygon if it would overlap with an existing polygon on OSM.

How do you ensure that the users of this feature are aware of the Import Guidelines and don not violate the guideline? If you make a workflow easier, it is your responsibility that you do not lead your users into an area which looks nice but is dangerous.

[jgravois]

@Nakaner the OSM wiki uses the following definition of Automated edit.

... "changes made to OpenStreetMap content with no or very limited human oversight".

(emphasis mine)

the protections we are putting in place (to forbid importing overlapping features and require review and approval for each individual new feature) are intended to ensure that the human oversight remains meaningful. is it possible that a user can click buttons and approve individual problematic edits one at a time? or course! but it is also possible for folks to sketch poor edits by hand with equal velocity. yet you're attempting to apply the same standard as you do to bulk edits made without manual review.

lots of folks have chimed in here with valid concerns that we all share. we are attempting to articulate our plans to address these things individually, but other specific suggestions would also be sincerely appreciated.

[slibby]

@simonpoole I had a question about the logging of imagery sources - is the mandatory imagery_used changeset tag the primary method used in iD for this currently, or are there other behind the scenes telemetry or logging? Perhaps this provides a way to save a similar reference from this proposed feature - a recording that the changeset includes features from GeoService:https://test.example.com/MapServer so that there is a chain of custody like with the imagery layers.

[stevage]

I'd just like to jump in here as someone who has done a lot of OSM editing (mostly with Potlatch2), has lots of experience with open data, and indeed, currently manages open data for a local government, with lots of useful spatial data.

I'm really in favour of this kind of tool. There seems to be an assumption in some of the above negative comments that competent OSM users dont need easy-to-use tools, and such tools will only encourage clueless newbies to carry out harmful imports.

Currently I don't have any way to import data, since I don't use JOSM. I'd love to be able to import lots of POIs, building outlines etc, on a case-by-case basis. This would be really handy.

An easy way of dealing with the copyright issue is by whitelisting a known set of approved endpoints.

(The only issue for me with this specific tool is we don't currently expose any ArcGIS services, so this is a bit useless to me. If it could import GeoJSON it would be useful.)

[amandasaurus]

overlap - for features where this is relevant (currently buildings) we avoid importing a polygon if it would overlap with an existing polygon on OSM.

Umm... are you really sure about that? Because if so, you won't be able to import any buildings. Countries are mapped in OSM. Relation 148,838 is the United States of America. Does that mean you don't import any buildings which overlap with that polygon? You're allowed to import buildings which overlap with boundary=administrative polygons.

Additionally you should probably check for other polygons, not just buildings. What if someone wants to import lakes that overlap existing OSM buildings? What about riverbanks and roads?

[mapmeld]

@rory - when writing that I was going back and forth on how generic to make the statement. What the feature actually does we can avoid overlapping duplicate data on OSM. We currently have this enabled for conflicting building polygons, preferring any existing OSM version. If it's relevant we could also apply it to another iD Editor preset, such as water polygons.

It's also only capable of working with the data in the browser viewer... we don't search the planet for polygons.

[Nakaner]

@jgravois wrote:

@Nakaner the OSM wiki uses the following definition of Automated edit.

... "changes made to OpenStreetMap content with no or very limited human oversight".

(emphasis mine)

Have you read the Import Guidelines? What your tool is going to make easier is an import, not an automated edit. I don't summarize the guideline for you, please read it on your own.

[amandasaurus]

we can avoid overlapping duplicate data on OSM. We currently have this enabled for conflicting building polygons

OK, that makes more sense. 🙂 It's good to ensure you don't put buildings on top of buildings, but there are many other things to be careful with, buildings/roads (w/o layers/tunnels/bridges).

If you were to write this a JOSM plugin, instead of for iD, you'd benefit from JOSM's existing validators, which covers "overlapping buildings" and much more.

[bhousel]

Hey @slibby, @mapmeld @jgravois, I am really excited to take a look at what you've built, and I appreciate that you opened this issue for discussion before submitting a PR, especially considering that imports are a controversial topic, and this work has the potential to have a larger effect on OpenStreetMap than a typical change to iD.

I'm also glad to hear the comments from @rory, @Nakaner, @woodpeck, @SomeoneElseOSM, @simonpoole, and @stevage (hope I am not forgetting anybody). Many of you are leaders within the OSM community and work with the DWG on policing and cleaning up after imports, so have more experience than anybody when it comes to dealing with the problems caused by bad imports.

Here are my overall impressions, (I will follow up with more direct responses to some of the items):

1. Everyone here agrees that not all imports are bad. There are appropriate situations for them, and it seems the typical use case for this tool is "authority source of data wants to contribute their own license-compatible data to OpenStreetMap, in accordance with community rules on preannouncing the import and allowing the data to be thoroughly reviewed". I don't think the tool is intended for people to do "rogue" imports and we can include safeguards to prevent this.

2. Adding an import tool to the editor can help us be more efficient in reviewing imports and guiding people towards good import behavior. For example, if the tool is used, here are some things that iD could do:

• For starters, just adding a changeset tag like import=yes, and having it flagged in OSMCha within minutes would be huge for reviewers. Everyone would know immediately if an import is happening, rather than responding later only if somebody happens to notice or complain to the DWG.
• Why stop there? We could change the save screen to include more fields. e.g. if you want to ensure that the import has been pre-announced - add a field with a link to the import mailing list discussion or github ticket or slack channel where this has happened. If you want to ensure that the license guidelines were followed, add a field for that too. We can make these fields required. They can be stored in changeset tags. We can make source=* tag required too, or anything else we think of.
• Storing more metadata in changeset tags lets us measure and track the effect of imports on OpenStreetMap in the long term. Think of what we might want to measure here (Pearson's law: "that which is measured, improves")
• It's a good idea to test an import against the OSM dev server first before running it live. iD has a source switcher to make this easy.
• Could we enforce that the tool is only available if the user is logged in with a dedicated import/bot account? Maybe - might require some additions to openstreetmap-website and account management, but it's doable.

3. It's not just about ArcGIS - It was mentioned on Support multiple layers for managing imports #2586 that the tool could be adapted for GeoJSON or Shapefile use (I guess drag and drop), which is an already identified need for some groups.

4. The technical issues are real - see @woodpeck's comment above especially, in regards to overnoded ways, connectivity, overlaps, and conflation with existing OSM features. It looks like from @mapmeld's comment that they have incorporated some of this into the tool already.

5. Validation is coming to iD - This is a separate issue, but some work has been done on bringing more JOSM-style validation into iD, and I'm looking forward to incorporate that into iD soonish. Sorry I can't say more, but it is relevant to the discussion here, and running a validation step on all the imported things will be something that iD will be able to do. If strong validation is seen as a blocker to community acceptance of an iD import tool, I hope we will have news coming soon that makes everyone happy.

6. Not having a tool like this just means that people will continue to do whatever they want. This means more rogue undiscussed imports and special-purpose forks of iD.

So, I'm looking at this as an opportunity to improve how importing is done. Let's keep the discussion going and I'm looking forward to a PR that we can test out (against the OSM dev server of course!)

[jonahadkins]

Hi! 👋

Absolutely ❤️ the ideas and discussion here. I've broken up my comments based on my experience / 2 cents..

Adding GeoService Layer

I've done quite a few (approved) imports with locality provided data from ArcGIS Online/Open Data. As you can imagine sometimes the data is "interesting" and requires a bit of pre-processing. I've also experimented with using AGS Open Data as reference in updating / verifying military boundaries

I've always thought the wealth of authoritative data / information that organizations make available in AGOL could be super useful as reference for normal tracing/verification in iD.
A few examples & use cases:

• in Virginia the state makes available imagery services that are great for updating in out-of-the-way localities
• HIFLD Open has federal lands contains accurate locations of US military installations and sites
• Virginia DCR provides conservation lands managed by multiple agencies within the state - great for filling in boundary gaps when it's not clear due to bad imagery.
• NCOneMap provides parcel data for the entire state of North Carolina.

While some of the sources are authoritative, they're not necessarily great for importing due to over-generalization, or way too many vertices for example. But they are super helpful for providing the missing context or reference needed for the casual mapper.

TL:DR Please oh please make it easy to Add GeoService Layer in iD!

Importing GeoService Layer

I also think extending the Add GeoService Layer functionality to map attributes, and import features is excellent! I think the current limitations with some additional ones discussed above when editing in iD would help enforce the necessary rules to prohibit bad or erroneous importing. A mapper wouldn't be able to import an new state boundary or really long road, but a bite size chunk of buildings, pond, or maybe just map attributes from overlapping features?

If the implementation is viewed as an import and has been discussed with community and the import guidelines are followed, it's up to the user which tool they would use to perform the import. The discussed functionality here would make iD another import tool and only make the import process smoother when using an AGOL data source.

Great work @slibby @mapmeld @jgravois and anyone else i left off...

[Nakaner]

@bhousel wrote:

• Why stop there? We could change the save screen to include more fields. e.g. if you want to ensure that the import has been pre-announced - add a field with a link to the import mailing list discussion or github ticket or slack channel where this has happened. If you want to ensure that the license guidelines were followed, add a field for that too. We can make these fields required. They can be stored in changeset tags. We can make source=* tag required too, or anything else we think of.

Adding a changeset tag with the URL of the discussion on the Imports mailing list is a great idea. If that is done, import=yes is just eye candy because having a tag import_discussion=<URL> implies import=yes.

@bhousel wrote:

Validation is coming to iD - This is a separate issue, but some work has been done on bringing more JOSM-style validation into iD, and I'm looking forward to incorporate that into iD soonish. Sorry I can't say more, but it is relevant to the discussion here, and running a validation step on all the imported things will be something that iD will be able to do. If strong validation is seen as a blocker to community acceptance of an iD import tool, I hope we will have news coming soon that makes everyone happy.

I don't think that it is a blocker but if someone proposes on the Imports mailing list that he is going to use iD for an import, people will tell him that JOSM has a validator. ;-)

OT: Is there already an issue for tracking the progress of the validator feature? There is a dozen of issues where people complain about broken multipolygons, broken route relations etc. but no overall validation issue.

[jharpster]

Great discussion and comments so far. Assuming the license behind the data sources of the GeoServices layer are compatible with ODbL and that the process prohibits people from large scale imports or overwriting existing OSM data, this could be a great addition to iD.

I don't see much risk in users trying to edit complex boundary relations using a GeoService overlay. Beyond the relatively easy cases of adding missing building footprints or roads, it would be valuable to transfer missing attributes from 'Authoritative' sources to data already in OSM.

[simonpoole]

@slibby wrt imagery_used this may or may not be appropriate for non imagery sources, but I would suggest if you don't want to use imagery_used you should use a differently named tag for logging arcgis sources. The related topic is that you either need to support the black list from the OSM API, or a similar mechanism to block sources.

@ALL we have a number of people commenting on this issue that are not regular participants in such discussions, so that we can all understand who is coming from where, could everybody state their affiliation, if any, with ESRI when commenting?

Further: we should be clear that while improved tooling is great (@stevage ESRI brought the issue of easy imports forward as one of the goals of the PR, not anybody else), this does amount to support of not just any proprietary API, but that of the de facto monopolist in GIS-space. Now while in the end it is up to Mapbox to decide and there are other proprietary APIs supported in iD with no open counterpart, it would seem that this proposed PR should be balanced out with full support for WMS and WTMS layers so that other companies have a fighting (well not really) chance of supporting their customers using iD.

Affiliation with ESRI: none, I do live in likely the sole country in which ESRI has only 95% market share in the public administration market segment instead of 100%, so I'm likely biased.

[nickpeihl]

I am the GIS Coordinator for San Juan County, WA (San Juan islands) and I think this is an amazing idea that I've been hoping for for years!

The San Juan Islands are a busy summer tourist destination in the PNW but have a very small year-round population (~16,000). Unfortunately, aggregate mapping services like Google, Bing, and Esri have historically had very bad maps of our region which often leads to confusion for both tourists and locals. The County itself has very accurate maps and the data is public domain, but most map services (Esri being the exception) have not been willing to *mport our authoritative data. Instead, they require us to submit change requests for each road or address.

I (with some help from two friends) have corrected ~95% of County roads in OSM by tediously moving TIGER imported nodes or tracing new roads in iD using our street basemap with the goal of having at an openly licensed and reasonable quality mapping service in our rural area that tourists and locals may be able to use.

The proposal here and in #2586 would vastly improve the speed and quality of data into OSM for my County and likely many other areas. I hope both proposals can move forward together cohesively and I would be happy to contribute to them.

[slibby]

I work for Esri as stated in my Github bio, though this work is not part of my paid responsibilities so I would prefer to consider myself part of the OSM community for the purposes of this issue and eventual PR, though that may not be very convincing.

@simonpoole is the blacklist that you referred to the elements available here? http://api.openstreetmap.org/api/0.6/capabilities
If so, I'll add that to our feature list of what we want to implement in v.1

[stevage]

@simonpoole

ESRI brought the issue of easy imports forward as one of the goals of the PR, not anybody else), this does amount to support of not just any proprietary API, but that of the de facto monopolist in GIS-space. Now while in the end it is up to Mapbox to decide and there are other proprietary APIs supported in iD with no open counterpart, it would seem that this proposed PR should be balanced out with full support for WMS and WTMS layers so that other companies have a fighting (well not really) chance of supporting their customers using iD.

Esri has very high penetration in government GIS systems in Australia, but much less in public-facing open data. There are a few government bodies using ArcGIS Open Data (for example) but much more common is uploading spatial datasets to open data platforms such as data.gov.au (example, where they can be accessed by WFS or GeoJSON (no Esri here). Some councils, including where I work, use Socrata, which supports GeoJSON, KML and Shapefile but not WFS (or Esri web services) - example. Of course, there are lots of raster tile endpoints, both Esri and GeoServer, too - see nationalmap.gov.au.

So, the situation you describe does not really represent the current state of open spatial data in Australia, at least.

So anyway, if this issue could be reframed as "Selectively import features from open data endpoints such as WFS, GeoJSON and ArcGIS Feature Services", with an extensible import mechanism, that would be awesome.

[bhousel]

@Nakaner wrote:

OT: Is there already an issue for tracking the progress of the validator feature? There is a dozen of issues where people complain about broken multipolygons, broken route relations etc. but no overall validation issue.

There isn't a single issue tracking the validator feature, but there are a bunch of issues tagged , and #3130 is probably the best one that lists a bunch of things that we could validate.

[wuhland]

It seems like there are some differing viewpoints on this thread about how to define what this tool is actually doing.
@slibby noted near the top that:

I would like to try to keep this feature from being labeled as an "easy-import" feature or being directly associated to the concept of OSM bulk imports though that is likely inevitable.

And yea, it seems like in most commenters here are treating this feature as an import tool:
@Nakaner:

What your tool is going to make easier is an import, not an automated edit.

But is importing what this tool is actually doing?

The import section of the OSM Wiki defines it this way:

Importing (also known as Bulk Importing) is the process of uploading external data to OSM.

Based on this definition I think that there is a case to be made that this is not an import feature at all.
If we consider an individual point or way to be the core feature around which the OSM dataset is constructed then it strikes me that this feature absolutely not allowing for the importation of or bulk importation of data. I think what it is doing should rather be considered enabling the review and addition of individual datum.

It might seem like splitting hairs but there is an important difference between the two. The fact that individual users would be reviewing each addition to the OSM dataset is something that I feel is largely being discounted in this conversation.

Certainly, bulk imports can be problematic. There are also a lot of ways people make poor decisions when interacting with OSM on an elemental basis but what we are talking about here is closer to the latter than the former. OSM is an open project I think we all accept that people make both bad and good decisions adding to it. If the argument here is that this tool will make it easier to make poor decisions than we should be up front about that and not just write it off as a bulk import tool.

I'm new to this conversation so for transparencies sake I'll say that I do not work for ESRI nor does my job depend on ESRI's good favor in any way. I work for USAID and have been a proponent of OS there for some time. I have also been wanting to see a feature like this for a while now, dont really care where it comes from.

[stevage]

Yeah. From the initial description:

The iD editor will load the vector features of the current map extent as temporary objects in the browser window (not immediately submitted as edits to OSM). The contributor can then select objects generated from the ArcGIS Layer, inspect and alter any tag values or geometries and submit them as edits to OSM (additions or replacements for features).

I would define the two options like:

• Bulk import: many features transferred directly from source to OSM, then hopefully reviewed
• Individual import: one or several features loaded from source into an OSM editor, reviewed, then saved to OSM.

[slibby]

Thanks again to everyone who responded to this Issue in great detail. I will be presenting on this topic tomorrow (Friday) at State of the Map US in Boulder, and our hope is to open a pull request shortly afterward, showing the work we've done to take these comments and suggestions into account. I would recommend we move further comments to that PR thread once it's opened. My slides for the presentation are available here, and a recording of the session will also be made: https://docs.google.com/presentation/d/1oV0gKRiqpINp4frwoMYLzCF2dsR4ZqDDxyRF8xWHTJM/edit?usp=sharing

edit: video of the talk is here: https://www.youtube.com/playlist?list=PLqjPa29lMiE2k2Sp5L5rb6ntduG9dt0te

[mboeringa]

edit: video of the talk is here: https://www.youtube.com/playlist?list=PLqjPa29lMiE2k2Sp5L5rb6ntduG9dt0te

That is the link for the playlist of all talks of State of the Map US 2017.

For easier access, the direct link to the video of the presentation @slibby gave that shows the functionality being discussed in this thread, is here:
https://www.youtube.com/watch?v=GUii8jWH_so

[simonpoole]

Shouldn't this be closed as ESRI has added the functionality to RapId, circumventing any possible outcome of the discussion on this fork?

[slibby]

I think that depends on whether RapId is accepted as widely-available editing client. This code is pretty stale and likely VERY out of date from the current state if iD, so I'm inclined to agree with closing the Issue.

More info on the RapId work here: https://www.esri.com/arcgis-blog/products/arcgis-living-atlas/mapping/arcgis-data-support-in-osm-editors/

Others are welcome to comment on the closed issue if you prefer to re-open it.