AGOL Backup Tool

[stdavis]

Although Esri recently added a recycle bin to AGOL, it only protects against item deletion for 14 days. There is still no backup for bad (intentional or not) edits to data.

We should build an automated tool that backs up AGOL data on a schedule similar to our traditional database backup schedule.

The data could be backed up to cloud storage. It could back up feature layers to FGDBs, uploads as their native type, and other items (web maps, apps, etc) to JSON files. The format for backup should be decided based upon what would make it easiest to recreate the items in AGOL. Expired backups would be cleaned up as part of the update process.

Inputs:

• credentials
• org
• tag name that designates which items get backed up

It would be awesome to have a turn-key solution ready to deploy for other clients. I know that DEQ would be interested as well as UDOT (Keaton) and DWR (Buck). This could also perhaps turn into an open source project that has appeal to a broader AGOL audience.

It would also be cool to have some sort of restore option.

This seems to be the type of thing that Esri recommends.

@steveoh Was correct to point out that we should not be pushing folks (DEQ is the most recent) to move their single source of truth for data to AGOL without also having a solution for backing up data. We have also started to migrate a few layers to AGOL exclusively for which we should have backups.

Existing Solutions:

Backup Your ArcGIS from GEO Jobe
CivicLens Backup Utility
Backup Manager

[steveoh]

I suggest that we start small and prioritize the most important data types that are the most difficult to recreate first with restoring the data being as turn key as backing it up.

If we all agree with that, hub services are a simple http APIi download request as a fgdb.

AGOL only services are a bit more difficult but we can use the createReplica endpoint if it's available or page through the query results.

Jake would know the best formats for restoring the data but I'd like to plug GeoParquet for storage. Ideally we could share some code with palletjack to load the parquet and restore services.

[stdavis]

I can see the advantage of attempting to use the hub downloads as the first approach since they already have downloads cached and simple API. I worry a bit about how many times we have heard from users that the hub downloads are broken or out-of-date. I use the arcgis python api to query features into a feature dataset and then into a dataframe but I'm guessing that would not work for larger datasets. Perhaps requiring that the "allow others to export" checkbox be checked may need to happen so that we have access to the create replica endpoint as a backup to hub.

[steveoh]

Should we resolve this with a link to https://github.com/agrc/project-moonwalk since it's no longer an idea?