Layers - Pitch

[westnordost]

Idea

The idea is to make available a tool that makes contributing and maintaining data easy and efficient to do on a survey, focussing on one aspect of the map at a time.

Consideration

On the smartphone, there are three issues that further challenge usability:

• a small screen severely limits the number of things that can be displayed at once
• a fat finger as pointing device is not only much less precise than a computer mouse, the same finger obscures part of the screen, most importantly the part where the tap (or drag) happens
• the user is currently out and about

StreetComplete is so easy and quick to use, because amongst other things, it severely limits what data the user can see and change.
The usability of "fully-featured" editors like iD, JOSM and Vespucci all suffer from that the data they work with is so complex, as they show all of it at the same time. New mappers, especially from GIS backgrounds are often surprised that there is no inherent concept of layers in OpenStreetMap.

For mappers who want to map/update "everything" on a survey, there is and there will probably never be a more efficient method than taking a lot of photos (or for mapillary) and then adding all the info visible on the photos to the map via JOSM.

The target audience for this idea are mappers who want to focus on completing and keeping up-to-date one aspect of OpenStreetMap efficiently at a time, i.e. per survey.

This requires the app to be able to

• show the current data in an uncluttered and clear form to the user, only that "layer"
• and the ability to add/update this data quickly and easily so that it can be done on the go without stopping and so that subsequent data entry at the computer at home is not necessary

Rough UI Concept

The UI / and input form can and should be customized for each layer as long as it enhances the overview and makes the input easier.

1. the background map should probably be desaturated. No quests are displayed. At least, the data for the layer must stand out in (flashy) colors.
2. the legend lists what each color stands for. Ideally, this should be retractable into a tappable icon, maybe have it expanded every time one enters this layer
3. selection is highlighted in some way. Maybe like in iD (yellow glow) or maybe as if it is floating above the map (shadow).
4. the show/edit form for the selected element. Maybe always show this section - empty or with a hint if nothing is selected to reduce jarring motions during usage, like in JOSM
5. for layers where new (POIs) can be added, maybe this UI: display a huge crosshair which marks the point where a new point is added after pressing the (+) button.

Switching to this other part of the app would be possible through the main menu. It would lead to a list of available layers, one can select a layer, that list slides to the left to reveal the map with the appropriate layer.

Colors

For all the layers, there should be a consistent color scheme at least for "no" and "not specified yet". I am thinking, it should be...

• transparent or grey for "no"/"N/A", f.e. not lit, no cycleway, ...
• red for "not specified yet"
  ... because missing data should stand out, rather than data where a particular feature does not exist

Possibilities

I did some prototyping with Tangram and found out that:

1. it is not possible to display textures (reasonably) on ways. So this means, there can't be any graphics for the lane count, the different cycleway options etc onto the street ways. It's colors (, widths, dashing-patterns, outlines) only
2. it would be possible to have repeating graphics on the ways, much like the oneway-arrows. This might replace a legend.
3. It is possible to display lines on the left and on the right side of ways

Too bad for point 1, so the display will be less fancy. However, this means it could maybe be streamlined so that it is easier to add new layers.

Layers

Why layers at all, if focussed contribution and maintenance is already possible with the quest system? One can simply deactivate all quests except ones where one wants to do the survey (f.e. only cycleway quests). Well:
For some data, it is necessary to see the big picture in order to be able to fill in the blanks and/or notice that certain data is outdated. A good example are housenumbers, POIs (a shop is missing / wrong name). For some others, the big picture is necessary because the data is part of a connected network: A good example are speed limits and specifically slow zones, lit ways and paths, cycleway infrastructure etc.

So, there are at least two additional prerequisites for layers:

1. use where the big picture / surrounding data of the same type is necessary or at least very helpful
2. use for data that can only be recorded or is best recorded on-site - because this app is to be used on a survey

Layer ideas

Certainly there are many more, just covering the most valued ones.

These are the most important because these are things that can only be effectively recorded on-site:

1. Addresses: Edit street and housenumber of addresses and add new addresses efficiently. Unclear how the street address part could be made efficient.
2. POIs: Show and let the user add/edit POIs (shops, amenities, ...) types and maybe names with the help of the id presets
3. Maxspeed: While quite important, I wouldn't want to touch this with the current prevalent tagging scheme(s). Difficulty here is that two types of infos should be displayed at the same time
4. Lit: Might be a candidate for the first layer because it is the layer with the least caveats I'd think
5. Smoothness: Important to see the big picture here to put what is "good" smoothness etc. into context

This is less important because while they can often only be effectively recorded on-site, it is not necessary to see the big picture:
6. Surface

These are also less important because while it might be very helpful to see the big picture, they do not necessarily need to be recorded on-site:
7. Cycleway infrastructure: Edit cycleway availability on roads or mark them as cycleway=separate, maybe edit whether sidepaths allow cyclists/have cycle tracks. There are many open ends here, one could also allow changing classification of roads to/from bicycle roads, show and allow to add bicycle stand POIs etc, though it is better to start as limited as possible.
8. Sidewalks: Same as cycleway infrastructure but certainly easier to do
9. Lanes: Difficult here to display it properly
10. Parking lanes

Technical

Now comes the hard part. The really really hard part. Thinking how the UI is best going to be is nice and dandy, but the technical part must work basically so different from the rest of the app that it could basically be another app.
This kind of edit mode would work more like a classic OpenStreetMap editor: Download data into memory, visualize and let the user edit that data, upload the changes at the end. StreetComplete doesn't do that at all at the moment. Here is a small excursion:

Current data flow

1. Download: The app memorizes and persists to a local database which rectangles of the map have been downloaded already (more precisely: tiles at zoom level 16) DownloadedTilesDao. Whenever the user's position changes, it is checked whether the area he is in currently has already been downloaded. If not, it is. X days after an area has been downloaded, the app deliberately forgets that this area has been downloaded, so that a download is triggered again. This way, the app ensures that the data is refreshed every X days.
2. Quest Creation: On each download, the data is analyzed for eligibility for all the quest types. The quests are created for all elements that are eligible to them.
3. Persistence: Both the quests and all elements that are referenced by at least one quest are persisted into a local database. All other elements are discarded. So at this point, you can simply close the app and the quests will be there next time you open it. You may take this for granted already, but JOSM for example doesn't do that (automatically) - nothing is persisted unless you deliberately save it to disc.
4. Quest display: The map view in turn also memorizes which rectangles of the map have been in view already. If the user scrolls over an area that haven't been in view already, that section is loaded from database and the quests therein added to the map.
5. Quest solving: When a quest is solved, it is marked as solved and persisted to the database. The OSM element it refers to is not changed at all at this point, this is also in contrast to other OSM editors which work directly on that data. Instead, a kind of diff is created and persisted.
6. Upload: On upload, the diff is applied to the persisted element fetched from the database, then uploaded - one element at a time. The updated element (from the server) is then persisted.

So what this means is, that the answers the user gives does not result in a change to the data, only after upload. This is okay, since that changed data is not shown anywhere. The downside that can already now be observed with this approach is that quests are not immediately (re)created after splitting a way (offline) and quests that conditionally follow after another quest has been answered are also not created until upload.

Required data flow

Above section in a nutshell: Downloaded OSM data is not changed until upload. This approach will not work for the layers as the user wants to see his change immediately, not after upload. Also, he'd want to immediately continue after splitting a way.

So, what is necessary is

1. Download: at least also the elements used for any layer must also be persisted so that they are available offline.
2. Persistence: They must be persisted in a way that allows efficient spatial access to them. In other words, they need to be sorted into a tiles raster. This touches upon Consider quest dependencies #2438 (comment) , some work done for the layers is the same work that would need to be done to remove the technical limitation that follow-up quests are not displayed until another download happens
3. Display: The map view for the layers will work similarly to the map view for the quests (see point 4 in previous section), only that not the quests are loaded dynamically from database, but the OSM elements referenced by the layer definition
4. Editing: To enable editing, the concept that the downloaded OSM element is not changed within the app must be broken. Still pondering how to do this... will probably involve having two versions of an (edited) element: The original downloaded version and the local version (=projected server version).
5. Upload: I'd like to stick to the concept of keeping around diffs for each element and applying these diffs just before upload for automatic conflict resolution. So, the local edited version would likely only serve display purposes. However, there may be unknown complexities with the undo function here.

MVP

An MVP would be to have at least one layer available at all, reachable through the UI. Selecting and viewing the current data should work.
It would be very good if editing of this data also already works but it is not absolutely necessary for a first version.

It would be nice to have the layer definition in some kind of streamlined fashion so that it is easier to add new layers, but it is not necessary for a first version.
Specifically splitting ways and creating/deleting POIs would be a feature that would likely not be available from the start.

So, a good MVP should have basic editing capabilities, in a pinch online-only would do, but editing is simply necessary to see if this idea gets any followers. Both MapContrib and MapComplete, who have similar approaches, did not really lift off. So before sinking too much time into this idea, it would be best to get a rudimentary prototype going ASAP and then seeing how the idea is being embraced or not.

[matkoniecz]

Why layers at all, if focussed contribution and maintenance is already possible with the quest system? One can simply deactivate all quests except ones where one wants to do the survey (f.e. only cycleway quests). Well:

Also, if someone knows that data become outdated (big redesign of cycleways, changes in lighting system) then it would enable to update it.

It would also allow to review what was added by others.

[westnordost]

Please feel free to comment. Also if you think you'd use it and in what way you would use it. If you can think of other layers to add, or an efficient interface etc..

It is not 100% clear yet if I do this or not, since the technical part is somewhat of an Herculean task and I am not sure if I still got enough time for it. Positive feedback can help. I did plan to do this as part of the BMBF grant but only got time to plan and occupy myself (with the details) it since this week.

[smichel17]

I don't doubt that this is useful, but…

Thinking how the UI is best going to be is nice and dandy, but the technical part must work basically so different from the rest of the app that it could basically be another app.

In this case, why not actually make it a different app?

• The UI is also different enough from StreetComplete that you would need a way to switch between the UIs.
  This means you must either:
  • Add a control to the main StreetComplete interface, probably the menu. This is some added clutter.
  • Add a setting to switch between the two modes. This seems a slower way to change between modes than using the "recent apps" screen to change between two apps.
• If there will be some parts shared, you can pull them out into a separate library.
  • The easiest way to do this is first move the shared features into a module inside the same code base, and then move the module into a different library when you are ready (without this, rapid iteration is a real PITA).
    • Red Moon has an example of a module within the same code base — see timepickerpreference https://github.com/LibreShift/red-moon
    • Actually, you could keep the other app's code base in the same repo, to eliminate the overhead of bumping library versions. This has other downsides, though.
• Or, instead of making a different app entirely, maybe it would make sense to take Vespucci's design in this direction.

---

All that said, I think I see a less extreme solution than what you are proposing, that can fit within StreetComplete more easily.

1. Implement quest presets (Add quest presets #1654)*.
   • The quest presets will serve the same purpose as layers.
2. Add my suggestion in Show all quests for element at once #124, so you can switch between quests available for a certain element.
   • This will be the UI for choosing which tag to edit on a given object.
3. Add a new option under the "resurvey intervals" setting called "bulk edit mode".
   • This shows every quest unconditionally.
   • When bulk edit mode is enabled, changesets are never created automatically. Instead, the upload button is always shown, regardless of the automatic upload setting. Pushing the upload button saves every pending change in one changeset.
     • Not sure about the technical implementation of this one — you can probably think of improvements.
   • Optional: When in bulk edit mode, change the available presets to match the "layers" you had in mind.
     • Also the map display, as you mentioned.
4. If it is important to be able to create new elements, use a modified version of the interface for creating a note.

*By the way, this feature is exactly the type of thing I am talking about in #1654 when I have said I want to think deeply about quest presets and make sure we have considered ALL of the use cases. Same for #1914, and other issues mentioned at feature: quest display.

Anyway, these are the goals you said we need:

• show the current data in an uncluttered and clear form to the user, only that "layer"
• and the ability to add/update this data quickly and easily so that it can be done on the go without stopping and so that subsequent data entry at the computer at home is not necessary

I think appropriate presets would accomplish this, perhaps along with an indicator of what quests are available, as discussed in #124 (comment).

It is not quite as nice as what you propose, but I think it is a much better ratio of effort to reward.

[matkoniecz]

In this case, why not actually make it a different app?

Download, login, installation would need to happen in both apps. Also, adding it in SC immediately has significant user base.

[westnordost]

Implement quest presets (#1654)*.

To be honest, this turned into a wall of text so much that I stopped following the discussion. I'll just wait out if #2457 will come to any sort of tangible conclusion.

[rouelibre1]

I would certainly use it for micro-mapping tasks when I want to complete some streets / neighborhoods.

My first use case is always the same thing that I like doing : bicycle parking. It could work well for benches, fyre hydrants or any other urban furniture.

Right now, I've customized OsmAnd+ UI for that purpose (= adding the missing bicycle parking POIs), and I use SC to survey the existing POIs where some tags (bicycle_parking, covered, capacity) are missing.

To give an idea, this is how my OsmAnd+ looks right now

I use a lot the feature to save geolocated photo/audio notes in the app and then process them at home on my computer.

Also, the ultimate thing that I'm missing is a way to quickly visualise if and when a street or area was "completed" for a certain type of POI (= 100% of POI of type X from this street/area are supposed to be in OSM), and also define & assign survey areas as tasks to different people, like some kind of task manager to organize the surveying of a city for example, with the ultimate goal of making it 100% complete & accurate on one specific aspect of the map.

[westnordost]

Also, the ultimate thing that I'm missing is a way to quickly visualise if and when a street or area was "completed" for a certain type of POI

Well, that's just not in the OSM data. But anyway, you can assign survey tasks, indirectly: Just open a note and phrase it as a question (add a "?"). StreetComplete users will then see the note and contribute to it. F.e. you could ask for photos to be taken etc.

The OsmAnd thing is interesting. Can you actually tap on the bicycle parkings there and change the data? How does the interface look? Can this be done with OsmAnd also for non-POIs? (i.e., (cycle)ways)

[rouelibre1]

Well, that's just not in the OSM data.

Indeed, I was just dreaming up about the ultimate task management tool to organized perfect "cartoparties" (as we call them in french) between small groups of highly motivated users. But I don't believe it's StreetComplete's mission.

The OsmAnd thing is interesting. Can you actually tap on the bicycle parkings there and change the data? How does the interface look? Can this be done with OsmAnd also for non-POIs? (i.e., (cycle)ways)

I don't believe the app allows editing of non-POI data, the menu of Osmand I'm using to display these is called 'POI overlay', and the native app plugin to edit osm data refers specifically to POIs.
The next steps in editing the POI are no so pleasant but it works, here are a few screenshots

You can find the app on F-droid to try it out if you want, it gives you access for free to the 'OsmAnd live' feature to refresh osm data as much as you want (on Play store, you either have to go with the free version of the app that retrieves osm data updates monthly, or pay a subscription to get updates more frequently). In the app, you need to activate the plugin 'OpenStreetMap editing'. The plugin 'Audio/video notes' is pretty interesting too and pleasant to use.

[smichel17]

Download, login, installation would need to happen in both apps. Also, adding it in SC immediately has significant user base.

I suppose that's fair, although I wonder how big a barrier it really is given that this feature seems targeted at power users.

An alternative is that (according to this very old stackoverflow question, at least) it is possible for one app to put multiple entries in the launcher. I think if this were set up with care (making sure they always use different windows), it could behave to the user as two separate apps, which share some information.

[mnalis]

The idea is to make available a tool that makes contributing and maintaining data easy and efficient to do on a survey, focussing on one aspect of the map at a time.

So, if I get this correctly, the idea is basically having Quest "presets" (#1654) renamed to "layers" with minimal number of quests per layer and with each layer having it's own (CSS-alike) styling (or am I missing something crucial here)?

If so, I'd instead suggesting separate user-customizable presets (so user can put one or more quests in it, as they wish to map), and separate (also user-customizable) styling. OsmAnd does it is similar way, as that part of it IMHO works very nicely (OsmAnd ships with few presets and styles filled with defaults, but users are free to modify them and/or add new ones customized to their liking).

BTW if implemented, could multiple layers be used (and customized?) at once, in true GIS-style (thus increasing complexity), or only one-layer-at-the-time (thus highly decreasing usability)? The first (complex) one would be useful to me (unlike the limited one), but I'm afraid majority of current SC users (non-expert-OSM/GIS users) who are there for simplicity would not be happy with the idea of turning SC into such a GIS system.

Also, I see few problems with implementing this idea, and some notes:

• if whole of SC moves to that GIS idea instead of current way of doing things - it would highly inconvenience current users used to (and/or preferring) current way of doing things, and I would never recommend alienating current userbase.
• if GIS-way is added in addition to current way of doing things in SC, it will make it much more complicated and make users choose between two radically different concepts in same app (and also it will split developer resources to supporting two separate interfaces) would also probable be highly confusing
• maybe the most kind way to users would be new app as suggested (while firstly isolating common parts into separate library to avoid duplicating bugs/code etc). But this would also split developer resources to supporting two separate interfaces, leaving less time to attend to each of the codebases.

[smichel17]

new app as suggested

My suggestion was a different app, which does not necessarily mean a new app. I already mentioned this in passing but I want to emphasize it: I think this idea would be a great UI for Vespucci.

@westnordost As you rightly point out, Vespucci suffers from showing too much data on screen at once. The ability to filter to different "layers" would make it so much more usable. It would be amazing. I would reinstall the app.

But StreetComplete already has a concept to limit and simplify the data shown, and that concept is the Quest. Everything in StreetComplete is quests, from its data structures to its UI. You've been reluctant to implement even multi-part quests because of this (#2127 (comment) and related), and so it is quite confusing to me that you now propose an even larger departure.

---

Edit:

Forgot to mention, it seems like this feature is targeted at "power users" — historically not StreetComplete's target demographic, but they are Vespucci's demographic. The two apps are not really in competition, except to the extent that Vespucci's complicated UX makes it hard to use even for power users… who then turn to StreetComplete. A usability improvement to Vespucci would help SC too, because then power users will have less need of it, so SC can focus on being the best "introduction to OSM" app that it can be.

And I want to add that I totally sympathize with the technical side of things — SC's code base is nice to work in, and it's much easier to work with what you know or build green field than jump into an unfamiliar code base. It just doesn't sway my opinion about which is the right design decision :)

[westnordost]

@mnalis Well, no not really it is not the same as quest presets. As @smichel17 noted, it is quite a departure from the quest concept. But StreetComplete is not defined by its quests, its defined by its ease of use. The new concept is still easy to use, while covering a use case that is more attractive for power/craft mappers because it is more efficient.

The quest system has its limits of what can be done because for each quest it requires an algorithm to determine whether a data point is missing or not. This is simple in some cases (lit tag on highway=... is missing) but can be extremely hard and not so suitable for others (finding where housenumbers are missing, see #2464).

Taking the example of housenumbers, an interface like shown in the mockup above would be by far more efficient and unproblematic than the quest version of it (i.e. not dependent on building type bein set first, not disabled in certain countries, not having a complex algorithm to determine if a housenumber is really missing etc ...) because of one reason: The user can have this overview himself, and see by himself if something is missing, no algorithm required. This is one big advantage all the other OSM editors have over StreetComplete.

@smichel17 In regards to Vespucci - it wouldn't surprise me if you could already do that in Vespucci somehow, after all you can also add custom presets etc. as far as I know. The general UX approach of Vespucci (and JOSM) is to grant the user maximum flexibility. So, if this would be implemented there, it would probably look and work very similarly to filters in JOSM:

1. Each user has to specify his custom filter by himself and
2. There is no streamlined form/input that would make it easier to edit this one data point the filter has been created for

The problem about Vespucci is that it can do so many things, that even if such a feature would be added, it would easily be buried amongst the many other settings and customizations one can do.
Imagine a StreetComplete where there are no predefined quests but each user defines his own quest types by writing overpass wizard queries and defining a simple UI for these. Nobody would use that! Know OpenMapKit?

This is why I initially wrote that such a layers feature would need to have such a "handcrafted" UI and smart filters just like the quests: To preserve the ease of use and to not confront the users with such customization.

[westnordost]

I am just going to tag @pietervdvn , author of MapComplete here because the approach is quite similar, maybe he is interested in reading this and might have an opinion about it. Same with @GuillaumeAmat, author of MapContrib. @zlant created the parking lanes viewer, which is a very similar concept as the one here.
Finally, the idea could be interesting for @simonpoole , author of Vespucci, as @smichel17 mentioned that maybe such an idea could be a better fit to be integrated into Vespucci in an UI overhaul or so.