Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle uploads of existing s3 objects #49

Closed
perrygeo opened this issue Jan 20, 2016 · 2 comments
Closed

Handle uploads of existing s3 objects #49

perrygeo opened this issue Jan 20, 2016 · 2 comments
Milestone

Comments

@perrygeo
Copy link
Contributor

It should be possible to accept an s3 uri as an upload object. With a file we transfer the data to s3 with stage and then calling create to start the processing. With an s3 uri we skip the transfer and call create directly

Questions

  • Does this work across accounts?
  • If not, there may be an intermediate option where we can use boto to transfer between s3 buckets (advantage: no need to pull data locally) then call create.

cc @rclark

@perrygeo
Copy link
Contributor Author

So a few notes on what I've found so far:

The upload API appears to only accept https URIs, not s3://

$ mapbox upload perrygeo.test3 s3://mapbox/playground/perrygeo/rgb.tif
Error: status code 500
{"message":"Internal Server Error"}

If I upload from mapbox bucket with standard permissions (ie not public readable), I get a 422 error - "Could not access URL". If I change perms to public readable, I can make it work once

$ mapbox upload perrygeo.test1 https://s3.amazonaws.com/mapbox/playground/perrygeo/rgb.tif

However, subsequent uploads with that url fail

$ mapbox upload perrygeo.test2 https://s3.amazonaws.com/mapbox/playground/perrygeo/rgb.tif
Error: status code 409
{"message":"The provided url has already been used by another upload. Fetch new credentials."}

Trying to access publicly accessible data from a non-mapbox bucket in a different region fails

$ mapbox upload perrygeo.test4 https://s3-us-west-2.amazonaws.com/perrygeo-gdal/test1.tif
Error: status code 500
{"message":"Internal Server Error"}

Still haven't tested non-mapbox, public in us-east to see if region plays a role.

So my current understanding is that direct s3 upload is only available to public objects in mapbox buckets - @rclark does this behavior make sense given what you know of the Upload API?

@perrygeo
Copy link
Contributor Author

From @rclark via chat:

  • public/private objects in tilestream-tilesets-production bucket: OK!
  • public objects in any other mapbox-owned bucket: OK!
  • public/private objects in non-mapbox-owned buckets: No dice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants