Add support for S3 Bucket URL #118

astromoose · 2018-05-03T12:46:57Z

Enhancement Suggestion

Afternoon,

I'd love to use Rome but can't use S3 for big-silly-company reasons. I can, however set up a Minio (https://docs.minio.io/) instance locally which is basically an S3 compatible object store that runs within a docker container.

Getting rome to work with Minio would mean supporting an s3 url in .aws/config rather than a region. Something like:

[default]
endpoint =  https://play.minio.io:9000

Why would the enhancement be useful to most users

Supporting s3-compatible endpoints would open up other similar object storage systems and services. I know s3 can be super cheap, but some companies (esp. european ones) can't use AWS for compliance reasons.

The text was updated successfully, but these errors were encountered:

tmspzz · 2018-05-03T12:54:05Z

Can you share parts of the file system between host and minio?

If so you can just use local in your Romefile.

[Cache]
  local = ~/shared/path/with/minio

Also does the config for minio require this endpoint to be in .aws/config?

astromoose · 2018-05-03T13:00:09Z

Thanks for getting back to me so quickly! I'd already considered local/shared storage via smb/nfs.

We've got about ~40 developers using Carthage.

S3/Minio fits our needs a lot better as then we don't need to set up client shares on those 40 machines all over the world, we can just distribute S3 key/secrets per team.

tmspzz · 2018-05-03T13:02:17Z

What does it mean that minio is compatible with S3? The project uses a haskell equivalent of the amanzon SDK, thus it expects a region to operate.

tmspzz · 2018-05-03T13:05:19Z

You seem to be in luck anyways https://hackage.haskell.org/package/minio-hs

tmspzz · 2018-05-03T21:04:26Z

@astromoose please try pre-release https://github.com/blender/Rome/releases/tag/v0.15.0.40

If it works as you expect, I will merge and make a full release.

Can I have the pleasure of knowing what company this is for and add it to the list of companies using Rome (if you end up doing so)? Doesn't seem like a business sensitive information to me.

You know... I just worked for free for your employer :)

astromoose · 2018-05-04T05:41:29Z

Awesome, thanks! I’ll get Rome integrated in our CI today.

I work for Daimler TSS. When we’re saving time on builds you’re more than welcome to mention us on the list! I’ll let you know :)

tmspzz · 2018-05-04T05:53:03Z

Cool, @astromoose

follow the instructions here: https://github.com/blender/Rome/tree/feature/minio#setting-up-endpoint-override-for-minio

astromoose · 2018-05-04T10:12:56Z

Hoi hoi, looking very good but not quite there. rome can talk to the minio instance but is erroring on upload:

API: PutObject(bucket=rome, object=PureLayout/iOS/PureLayout.framework-7753452d69924e3699418ca9399e739dd7163683.zip)
Time: 10:00:17 UTC 05/04/2018
RemoteHost: xx.xx.71.31:60400
Error: stat /data/minio/rome/PureLayout/iOS: no such file or directory
       1: cmd/fs-v1-helpers.go:167:cmd.fsStat()
       2: cmd/fs-v1-helpers.go:232:cmd.fsStatFile()
       3: cmd/fs-v1.go:663:cmd.(*FSObjects).parentDirIsObject.func1()
       4: cmd/fs-v1.go:671:cmd.(*FSObjects).parentDirIsObject()
       5: cmd/fs-v1.go:734:cmd.(*FSObjects).putObject()
       6: cmd/fs-v1.go:689:cmd.(*FSObjects).PutObject()
       7: cmd/object-handlers.go:667:cmd.(ObjectLayer).PutObject-fm()
       8: cmd/object-handlers.go:744:cmd.objectAPIHandlers.PutObjectHandler()
       9: cmd/api-router.go:77:cmd.(objectAPIHandlers).PutObjectHandler-fm()
      10: net/http/server.go:1947:http.HandlerFunc.ServeHTTP()

API: PutObject(bucket=rome, object=PureLayout/iOS/PureLayout.framework-7753452d69924e3699418ca9399e739dd7163683.zip)
Time: 10:00:17 UTC 05/04/2018
RemoteHost: xx.xx.71.31:60400
Error: stat /data/minio/rome/PureLayout: no such file or directory
      <snip, same trace as above>

API: PutObject(bucket=rome, object=PureLayout/iOS/1DB2FFCC-7B7B-39A3-BCB0-D130A8BCC8F7.bcsymbolmap-7753452d69924e3699418ca9399e739dd7163683.zip)
Time: 10:03:48 UTC 05/04/2018
RemoteHost: xx.xx.71.31:60404
Error: read tcp 172.17.0.13:9000->xx.xx.71.31:60404: read: connection reset by peer
       1: cmd/fs-v1-helpers.go:326:cmd.fsCreateFile()
       2: cmd/fs-v1.go:779:cmd.(*FSObjects).putObject()
       3: cmd/fs-v1.go:689:cmd.(*FSObjects).PutObject()
       4: cmd/object-handlers.go:667:cmd.(ObjectLayer).PutObject-fm()
       5: cmd/object-handlers.go:744:cmd.objectAPIHandlers.PutObjectHandler()
       6: cmd/api-router.go:77:cmd.(objectAPIHandlers).PutObjectHandler-fm()
       7: net/http/server.go:1947:http.HandlerFunc.ServeHTTP()

API: PutObject(bucket=rome, object=PureLayout/iOS/7B70AD08-3413-3F7C-A298-B25EDEAD6885.bcsymbolmap-7753452d69924e3699418ca9399e739dd7163683.zip)
Time: 10:04:58 UTC 05/04/2018
RemoteHost: xx.xx.71.31:60406
Error: read tcp 172.17.0.13:9000->xx.xx.71.31:60406: read: connection reset by peer
      <snip, same as above>

I'll check these errors aren't related to minio running in a docker container though, those connection resets look weird.

tmspzz · 2018-05-04T10:14:54Z

I have tried with the public instance at https://play.minio.io:9000 and both upload and download work. You have to create the bucked in minio first.

astromoose · 2018-05-04T10:15:38Z

((OT: I love the release names, tell me you have some kind of random 'Consul of Rome' generator or something? :D))

tmspzz · 2018-05-04T10:20:39Z

It's less hi tech than that, I just follow the list at https://en.wikipedia.org/wiki/List_of_Roman_consuls

The errors above seem to me like Minio issues. It is trying to get attributes of some paths but it failing to do so:

stat /data/minio/rome/PureLayout/iOS

man stat

Maybe Minio cannot write to the file system?

tmspzz · 2018-05-07T11:43:42Z

@astromoose did you have any chance to check your Minio setup?

astromoose · 2018-05-08T12:09:06Z

Hoi @blender - still working on it. I'm seeing problems with calculated sha256's on file uploads, which deffo looks like minio issues.

Will keep you posted :) 👍

tmspzz · 2018-05-08T12:11:28Z

@astromoose please do.

tmspzz · 2018-05-17T12:19:30Z

@astromoose Any luck? I am thinking of merging this anyways (others features still work) and then make a new release.

thii · 2018-05-18T02:24:26Z

@blender Is that possible to fetch the region and endpoint via environment variables (e.g. AWS_REGION and AWS_ENDPOINT) instead of the ~/.aws/config file? Different consumer machines may have different configs.

tmspzz · 2018-05-18T06:00:55Z

@thii it should be already supported

thii · 2018-05-18T06:22:03Z

We're also using an S3 compatible object storage (not Minio). I've just tested with Rome 0.15.40 but it is still looking for the config file.

$ AWS_REGION=us-east-1 AWS_ENDPOINT=https://ourcorp.com rome download --platform iOS
rome: /Users/admin/.aws/config: openFile: does not exist (No such file or directory)

thii · 2018-05-18T06:40:56Z

By the way, when I added the config file, it still couldn't download from our remote cache.

Error: could not download OurFramework : Unexpected Error
Error: could not download OurFramework.dSYM : Unexpected Error

We are currently using Rome for the local cache only and use s3cmd to sync with our object storage.

tmspzz · 2018-05-18T07:01:04Z

@thii Sorry AWS_ENDPOINT is not supported, but I can add it if we all think it's a good name (doesn't appear to have a proper variable name in AWS CLI tools docs). I think it's a good name. AWS_REGION should be supported.

Could you tell me what S3 compatible cache you're using so that I can add it to the docs once this works?

thii · 2018-05-18T07:07:53Z

@blender From what I know, we're using Ceph.

tmspzz · 2018-05-18T07:12:06Z

@thii so turns out AWS_REGION is not supported as env variable.

I'll add both AWS_REGION and AWS_ENDPOINT

tmspzz · 2018-05-22T14:26:07Z

@thii a pre-release that supports both env variables is available here https://github.com/blender/Rome/releases/tag/v0.15.0.41

Please let me know if this works and I will make a full release.

thii · 2018-05-22T15:03:22Z

@blender Looks like it does pull the config values from the env vars instead of the config file, but it still returns an Unexpected Error when trying to download from our cache storage.

tmspzz · 2018-05-22T15:15:24Z

@thii can you paste the output of this command

AWS_REGION=us-east-1 AWS_ENDPOINT=https://ourcorp.com rome download --platform iOS

thii · 2018-05-22T15:27:21Z

I know this doesn't give you much info, but here it is.

Error: could not download OurFramework : Unexpected Error
Error: could not download OurFramework.dSYM : Unexpected Error
Error: Cannot retrieve symbolmaps ids for OurFramework
Error: could not download .OurFramework.version : Unexpected Error

There might be something different in our cache storage.

tmspzz · 2018-05-22T15:32:24Z

@thii this seems to be a communication problem with your store. Have you tried with the aws cli?

tmspzz · 2018-05-22T16:10:43Z

@thii there was some error prettifying, I remove it so you can get the error messages are they are. See https://github.com/blender/Rome/releases/tag/v0.15.0.42

thii · 2018-05-23T05:07:57Z

@blender It works now, after I explicitly specified the port to 443. Seems it fallbacks to port 9000 if not being specified, even though the endpoint starts with https. I suggest trying to get the port from the endpoint first before falling back to the default one.

tmspzz · 2018-05-23T05:15:06Z

That is was i was doing but clearly it's not working :) I'll fix that

tmspzz · 2018-05-23T05:55:06Z

@thii Can you please try this? https://github.com/blender/Rome/releases/tag/v0.15.0.43 should fix the default port for https if no port is speficied.

I'll promote this to a full release and merge if it works.

thii · 2018-05-23T07:59:22Z

@blender It works! Thanks. Please release.

tmspzz · 2018-05-23T08:22:59Z

We're done 🎉

brew upgrade rome

tmspzz mentioned this issue May 3, 2018

Add Minio support via AWS Service Endpoint override #119

Merged

tmspzz added enhancement in-progress labels May 4, 2018

tmspzz mentioned this issue May 15, 2018

Support for Bintray #120

Closed

tmspzz removed the in-progress label May 23, 2018

tmspzz closed this as completed May 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for S3 Bucket URL #118

Add support for S3 Bucket URL #118

astromoose commented May 3, 2018 •

edited

Loading

tmspzz commented May 3, 2018 •

edited

Loading

astromoose commented May 3, 2018

tmspzz commented May 3, 2018 •

edited

Loading

tmspzz commented May 3, 2018

tmspzz commented May 3, 2018

astromoose commented May 4, 2018

tmspzz commented May 4, 2018

astromoose commented May 4, 2018

tmspzz commented May 4, 2018 •

edited

Loading

astromoose commented May 4, 2018

tmspzz commented May 4, 2018 •

edited

Loading

tmspzz commented May 7, 2018

astromoose commented May 8, 2018

tmspzz commented May 8, 2018

tmspzz commented May 17, 2018

thii commented May 18, 2018

tmspzz commented May 18, 2018

thii commented May 18, 2018

thii commented May 18, 2018

tmspzz commented May 18, 2018

thii commented May 18, 2018

tmspzz commented May 18, 2018

tmspzz commented May 22, 2018

thii commented May 22, 2018

tmspzz commented May 22, 2018 •

edited

Loading

thii commented May 22, 2018 •

edited

Loading

tmspzz commented May 22, 2018

tmspzz commented May 22, 2018

thii commented May 23, 2018

tmspzz commented May 23, 2018

tmspzz commented May 23, 2018 •

edited

Loading

thii commented May 23, 2018

tmspzz commented May 23, 2018

Add support for S3 Bucket URL #118

Add support for S3 Bucket URL #118

Comments

astromoose commented May 3, 2018 • edited Loading

tmspzz commented May 3, 2018 • edited Loading

astromoose commented May 3, 2018

tmspzz commented May 3, 2018 • edited Loading

tmspzz commented May 3, 2018

tmspzz commented May 3, 2018

astromoose commented May 4, 2018

tmspzz commented May 4, 2018

astromoose commented May 4, 2018

tmspzz commented May 4, 2018 • edited Loading

astromoose commented May 4, 2018

tmspzz commented May 4, 2018 • edited Loading

tmspzz commented May 7, 2018

astromoose commented May 8, 2018

tmspzz commented May 8, 2018

tmspzz commented May 17, 2018

thii commented May 18, 2018

tmspzz commented May 18, 2018

thii commented May 18, 2018

thii commented May 18, 2018

tmspzz commented May 18, 2018

thii commented May 18, 2018

tmspzz commented May 18, 2018

tmspzz commented May 22, 2018

thii commented May 22, 2018

tmspzz commented May 22, 2018 • edited Loading

thii commented May 22, 2018 • edited Loading

tmspzz commented May 22, 2018

tmspzz commented May 22, 2018

thii commented May 23, 2018

tmspzz commented May 23, 2018

tmspzz commented May 23, 2018 • edited Loading

thii commented May 23, 2018

tmspzz commented May 23, 2018

astromoose commented May 3, 2018 •

edited

Loading

tmspzz commented May 3, 2018 •

edited

Loading

tmspzz commented May 3, 2018 •

edited

Loading

tmspzz commented May 4, 2018 •

edited

Loading

tmspzz commented May 4, 2018 •

edited

Loading

tmspzz commented May 22, 2018 •

edited

Loading

thii commented May 22, 2018 •

edited

Loading

tmspzz commented May 23, 2018 •

edited

Loading