Wayback Imagery

[Bonkles]

Description

This feature will add ESRI's Wayback imagery catalog to Rapid's basemap imagery panel (B).

Caption: screenshot of ESRI Wayback Imagery App

Background

ESRI Wayback imagery consists of multiple metadata and WMTS imagery endpoints, taken at different dates since 2014. Unlike other imagery endpoints such as Bing, Mapbox, or ESRI World Clarity, the wayback endpoints are NOT continually refreshed with newer imagery. This allows us to 'step back in time' to see what a place looked like at a given wayback date.

This article provides a helpful summary:
Wayback imagery Article: https://www.esri.com/about/newsroom/arcwatch/turn-back-the-clock-with-the-world-imagery-wayback-archive/

Example App: https://livingatlas.arcgis.com/wayback/#active=12457&mapCenter=-115.299%2C36.064%2C14
Example App Code: https://github.com/vannizhang/wayback

ArcGIS Online Group referenced by the app/code: https://esri.maps.arcgis.com/home/group.html?id=0f3189e1d1414edfad860b697b7d8311&start=1&view=list#content

Feature Requirements

The feature should:

• Display Wayback imagery as a single option in the background imagery panel (b).
• Load the available wayback imagery endpoints at runtime, not from a static configuration file.
• honor the existing keyboard commands for next/prev imagery:
  • If the 'next' or 'previous' is hit we should go forwards/backwards in time one increment.
  • IF there is no more wayback imagery we should go forwards/backwards to the next/prev imagery item in the list.
• display the vintage of the imagery in the Background Info panel (shift+cmd+B).

The feature should NOT display each wayback imagery as its own item in the background imagery list- this would lead to a very very long list.

[Bonkles]

@RitaDee for our initial investigation, let's understand a few things about the API. See if you can do the following:

Basic API Understanding - single image layer

• Look through the ArcGIS Online catalog for wayback and find one layer's worth of imagery at a specific date.
• Plug that imagery URL into Rapid's "custom data" background imagery option:

• See the imagery loaded into Rapid.
• See if you can modify imagery.json to include that same layer by copying and modifying the ESRI World Clarity entry. You'll need to keep the 'default' flag set to 'true' or it may not display.
• Open the Background Imagery Panel with cmd-shift-B. Does this display the correct information?
• Can you use a utility like curl or postman to construct an API call to obtain the correct imagery vintage for this same layer? (Remember that the metadata is obtained through a completely separate API in the ArcGIS online group)

Getting the list of imagery / metadata endpoints programmatically

As part of this feature, we will need to load the metadata and imagery layer listings at start time- we don't want to hard-code them into a configuration file or in the code- as ESRI releases new imagery in wayback, we should be able to pick them up without modifying/rebuilding the feature.

This can be accomplished through the ArcGIS Online API for Groups:
https://developers.arcgis.com/rest/users-groups-and-items/search.htm

Which explains the details about how to fetch a given group's entities. Here's a query that will get you started:

https://esri.maps.arcgis.com/sharing/rest/content/groups/0f3189e1d1414edfad860b697b7d8311/search?f=json&num=100&sortField=title&sortOrder=asc&start=1

This API should return a large JSON blob of the first 100 entries. Hint: Copy this blob, paste it into VSCode, and use the built-in formatter to 'nicely' print the blob so it's easier to read/reason about.

**Investigation for Programmatic metadata/imagery endpoint gathering **
When all is said and done, Rapid should load the imagery endpoints programmatically.

• Write an API call that contains a list of the first 100 imagery endpoints. (hint: you will need to modify the above query to add a 'filter' param. Check the 'type' returned by the query!).
• Pick one of the items returned in the previous item's query. See if you can construct a query to obtain the metadata from that layer (hint: look at the unfiltered query you started with- it contains entries for the metadata for each layer, too!).

[Bonkles]

@RitaDee let me know when you are finished with the above, and we can discuss next steps. :)

[tordans]

Thats an interesting dataset which I never looked at in details. Looking at I now here are a few thoughts…

License: (I find it helpful to document those legal train of thought, that why I wrote it down.)
I could not find an explicit "OK" from Esri in in the Wiki. But someone mentioned that since they documented the usage in OSM channels we can assume a "go ahead". I also think we can assume since the product is called "World Imagery Wayback" and the Wiki grants permission for all "World Imagery" the Wayback is included.
And now I find something very explicit: The World Imagery Wayback App links to detailed pages on each release (example) which has a section on "Data Collection and Editing" which links to a UseCases Doc which then links to a PDF which mentions OSM a few time including "Users acknowledge that any vector data contributed to OSM is then governed by and released under the OpenStreetMap License (e.g. ODbL). "
Not easy to find from an OSM point of view but very clear in the end.

UseCase: I wonder what the UseCase for the older images actually is. From an OSM mapping point of view, why would I go back more than ~5 years when I have more recent images? I recently argued to keep older areal images of Berlin in ELI because shadows and light is different. But once one goes back more than a few years, the risk of adding wrong data increases. So I wonder if the UI should guard for that by limiting the number of years that on can go back … or showing the vintage more prominently … or giving some kind of guidance when using the layer.

UI: While using the Viewer I noticed that the "change map on hover" is great to discover the right vintage. Which makes me think that visual discovery is very important for this. Which I guess is a +1 to add good keyboard shortcut support like described above so it becomes very easy to change the layer fast.

OHM: Looks like OpenHistoricalMap is also interested in the feature.

[Bonkles]

@tordans thanks for the investigation- and to answer a couple of your points, we're not sure exactly which vintages of imagery to serve- I don't think we'll need to show the user an exhaustive list of all the different dates available.

A couple of use cases are:

1. Tree coverage. Perhaps the latest imagery has a lot of tree cover and a previous vintage taken in fall/winter does not, allowing you to see the sidewalks in a city better.
2. Building demolitions/new construction. If your regular imagery is out of date there might be a building where you think a park or empty lot should be. If a building was knocked down, seeing some older imagery showing it still standing gives mappers the signal they need to delete the building, rather than wonder if their imagery is out of date.
3. Changes over time: If I see most of the buildings in an area mapped, but 2-3 aren't mapped, wayback imagery might give me an idea of why- perhaps those three of those buildings were all built in the past year?

I agree that going back to 2014 is perhaps not something most folks will need. Thinking about this a bit more, maybe we default to showing the most recent 'N' vintages, with the option for folks to click a checkbox that shows them every layer possible, just in case they do need to see all the dates.

And finally, we'll build in some metadata calls to ensure that the map shows very clearly what date the user is on.

[Bonkles]

@RitaDee here's a PR that implements this feature in iD: OpenHistoricalMap/iD#181

[RitaDee]

@RitaDee here's a PR that implements this feature in iD: OpenHistoricalMap/iD#181

Thank you for this, @Bonkles. I will study the implementation.

[1ec5]

here's a PR that implements this feature in iD: OpenHistoricalMap/iD#181

Hi, I’m the troublemaker who posted OpenHistoricalMap/iD#181 the other day. You’re more than welcome to crib from it, if the differences between Rapid and iD don’t make that too difficult.

I agree that going back to 2014 is perhaps not something most folks will need. Thinking about this a bit more, maybe we default to showing the most recent 'N' vintages, with the option for folks to click a checkbox that shows them every layer possible, just in case they do need to see all the dates.

There’s an old discussion about imagery layer retention in osmlab/editor-layer-index#1446. Part of what drove that discussion is that there’s a new TIGER Roads overlay every year, and some people were skeptical that TIGER from more than a couple years ago would be of any value. (I do find it valuable sometimes to compare a roadway with something closer to the original TIGER import.) For OHM, maximizing access to historical imagery is more of a priority, so I didn’t consider filtering old layers at all.

It would be really cool to replicate the “Only versions with local changes” option in the Wayback application (note: Apache-licensed code). This option is useful because Esri releases World Imagery multiple times a year but obviously their partners haven’t captured new imagery for the entire globe each time. Typically there’s a lag, which the metadata endpoints will indicate. I didn’t get as far as the “local changes” filter, but I assume it has something to do with the same endpoint that iD uses to get the layer’s vintage.

But someone mentioned that since they documented the usage in OSM channels we can assume a "go ahead".

For what it’s worth, @dkensok is a member of the team at Esri that produces the Wayback product. He’s been telling anyone who will listen that this imagery is out there and urging mappers to make use of it. But yes, the use cases document is important for getting the permission in writing.

[Bonkles]

Tanner! Use https://github.com/vannizhang/wayback-core for replacing some of the existing PR functionality.

Also known bug- repeated UI elements.