Handle uploads of existing s3 objects

[perrygeo]

It should be possible to accept an s3 uri as an upload object. With a file we transfer the data to s3 with stage and then calling create to start the processing. With an s3 uri we skip the transfer and call create directly

Questions

• Does this work across accounts?
• If not, there may be an intermediate option where we can use boto to transfer between s3 buckets (advantage: no need to pull data locally) then call create.

cc @rclark

[perrygeo]

So a few notes on what I've found so far:

The upload API appears to only accept https URIs, not s3://

$ mapbox upload perrygeo.test3 s3://mapbox/playground/perrygeo/rgb.tif
Error: status code 500
{"message":"Internal Server Error"}

If I upload from mapbox bucket with standard permissions (ie not public readable), I get a 422 error - "Could not access URL". If I change perms to public readable, I can make it work once

$ mapbox upload perrygeo.test1 https://s3.amazonaws.com/mapbox/playground/perrygeo/rgb.tif

However, subsequent uploads with that url fail

$ mapbox upload perrygeo.test2 https://s3.amazonaws.com/mapbox/playground/perrygeo/rgb.tif
Error: status code 409
{"message":"The provided url has already been used by another upload. Fetch new credentials."}

Trying to access publicly accessible data from a non-mapbox bucket in a different region fails

$ mapbox upload perrygeo.test4 https://s3-us-west-2.amazonaws.com/perrygeo-gdal/test1.tif
Error: status code 500
{"message":"Internal Server Error"}

Still haven't tested non-mapbox, public in us-east to see if region plays a role.

So my current understanding is that direct s3 upload is only available to public objects in mapbox buckets - @rclark does this behavior make sense given what you know of the Upload API?

[perrygeo]

From @rclark via chat:

• public/private objects in tilestream-tilesets-production bucket: OK!
• public objects in any other mapbox-owned bucket: OK!
• public/private objects in non-mapbox-owned buckets: No dice!