Add metis support

[graft]

Magma should use metis as its file service component.

Roadmapping:
Moved to planning

[graft]

The main reformations that must happen:

1. Magma does not accept File posts to /update. Instead, it returns a Metis upload-authorized URL. The client does whatever it wants with this URL and is responsible for uploading the file.

2. Metis notifies Magma that there is a file in the appropriate place via a notification endpoint, which it can reach via HMAC. Magma fills in the matching record.

[graft]

To use Metis, magma should be able to do a few things:

1. Hold a path on metis in a file or image attribute. This uses the /:project_name/magma bucket on metis owned by user 'magma'. This path allows magma to generate get and put URLs that can be handed off to someone else to download or upload the file at that path. This doesn't require any new functionality from Metis or magma.

2. Move data from Metis into that path. Instead of providing a file to the /update endpoint, we send an existing Metis file path. Magma performs a move operation metis, transferring the file from wherever it is to Magma's preferred destination. Metis currently doesn't allow moves across buckets, so perhaps Metis needs a new command: /transfer, which lets you move across buckets.

How should we authenticate? We need to use the user's token in (2) because they must have the rights to access the file, as determined by Metis based on the bucket permissions and so on. However, Metis also needs to know that Magma approves of the transfer, so that users can't overwrite files using /transfer on Metis without Magma's approval. It would be nice to achieve both of these things in a single transfer.

We can do this if we have an endpoint that requires BOTH a valid token AND an Hmac-signed URL. With such an endpoint, we can transfer a file from bucket1/path1, where bucket1.owner = 'metis', to magma/path2, where magma.owner = 'magma'. So: the user requests Magma to upload from a specified Metis location. Magma uses the user's token, signs a url, and requests from Metis the transfer operation.

[graft]

Using the transfer mechanism above we can avoid the necessity of requiring upload notification from Metis to Magma. The Magma upload cycle can instead look like this:

1. Post to /update with the revision "upload" for a file or image attribute. Magma replies with an upload URL for /:project_name/magma/uploads/<tmp_file_name> based on the model, attribute etc.

2. The user puts the file at the intended location.

3. When complete, the user posts to /update with the revision "metis://:project_name/magma/uploads/<tmp_file_name>". Magma moves the file using the Metis transfer endpoint and updates the DB when successful.

OR

1. Post /update with the location of an existing file in metis, skipping steps 1 and 2.

[graft]

When in (3) or (1) above, how does Magma communicate with Metis?

1. The user updates Magma with some metis locations for files. This means a POST to update with file attributes of the form "metis://some/file/location", which brings us into the update endpoint.
2. Magma censors the update. If there are censored inputs found, the entire update is rejected.
3. Magma validates the update. Using Magma::Validation, Magma verifies the data according to model valiations. That is, it checks the identifier, checks dictionaries, etc. Here again the entire update is rejected, with errors, if invalid input is found.
4. Magma posts to Metis asking to link (copy) all files to a new location controlled by Magma. This means POST { links: old_files.zip(new_files).to_h } to metis/link_files. The POST to Metis uses the user's token to authenticate. In addition the URL is hmac-signed by Magma.
5. Metis determines that (1) the files exist (2) the user has rights to access them and (3) that Magma has signed as the bucket owner. If not, it rejects to Magma, which rejects the entire update.
6. Metis links the files to the location specified by Magma and notifies Magma the files have been moved.
7. Magma updates the appropriate records with the new file locations.

Step 7 takes place in #upsert in Magma::Loader, called from #dispatch_record_set. Step 3 takes place in #validate, also called from #dispatch_record_set. This means step 4 must also occur in #dispatch_record_set. This is nice because we can collect all link requests from the entire record set. How do we do this?

There is a #metis_link method that is called in #dispatch_record_set, after #validate and before #upsert. It iterates over records, finds file attributes and assembles them into 'old_file' => 'new_file' pairs (old in any project bucket, new in the magma bucket). It then uses a Etna::Client to post the complete link hash to Metis. If Metis replies favorably, it continues loading, otherwise it raises a Magma::LoadFailed error with the incorrect old_files.

It seems like this behavior should actually live on the FileAttribute somehow, a problem for refactoring.

[graft]

Closed by #180