Proposal: a /sync side-car for pulling data from another Gaia hub

[jcnelson]

Proposal

Users will periodically need to move their data between Gaia hubs. The user can do so today by listing all their files, fetching them all, and re-uploading them all. However, this is needlessly expensive for the user, and especially painful on mobile -- the user needs to have 2x network capacity to move their data over. This proposal would add a POST /sync/{:address}/{:filename} endpoint that would allow an authorized user to instruct their private Gaia hub to pull a file from another location and store it under the given bucket.

API

The Gaia hub would have a POST /sync/{:address}/{:filename} endpoint that would allow the user to drive synchronization of their private Gaia hub using just list-files. To do so, the user would list each file with list-files, and for each file, would call POST /sync/{:address}/{:filename} to instruct the Gaia hub to asynchronously download and store the file from an upstream source.

The API endpoint would take as its POST body the URL to the original file. When received, the Gaia hub would immediately reply with an HTTP 204 and queue up the file to be fetched. In the background, the Gaia hub would dequeue files from its /sync/{:address} queue and fetch them. The user can later call GET /sync/{:address}/{:filename} to get the status of a particular file's download, or call GET /sync/{:address}?page={:page} to get the paginated statuses of all in-flight files.

If a file fails to download, it will be dequeued and no further action will be taken. It is the client's responsibility to re-queue failed files and to make sure that all files have transferred (e.g. such as by listing them on the target hub).

For security, the caller must have a signed auth token whose address is {:address} (or has an association token that grants them this ability), and the token must have the sync scope.

Implementation

I would recommend that if we built this, it should be a side-car service (possibly part of the Gaia admin side-car). This is because it maintains state of its own, and the Gaia hub proper should not have its resources consumed by doing work in the background.

Thoughts?