New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 Region #400

Closed
shreyu86 opened this Issue May 30, 2014 · 83 comments

Comments

Projects
None yet
@shreyu86
Contributor

shreyu86 commented May 30, 2014

Following is the problem I noticed with 0.7.0
When I specify s3_region as us-west-2 in my config, registry is up but I cannot pull or push.

If I comment out s3_region then everything is fine. I have confirmed on S3 that the region of my bucket is us-west-2
image

Has anyone else experienced this? Logs show that registry is connecting to BUCKET_NAME.s3-us-west-2.amazonaws.com but I cannot pull or push.

I would appreciate any help/pointers on this issue.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 May 30, 2014

Member

Hi @shreyu86

Any error message in the logs when you try to push / pull?
Any error message at startup?

Thanks,

  • Olivier
Member

dmp42 commented May 30, 2014

Hi @shreyu86

Any error message in the logs when you try to push / pull?
Any error message at startup?

Thanks,

  • Olivier

@dmp42 dmp42 modified the milestone: Next May 30, 2014

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 May 30, 2014

Contributor

I get 502 trying to pull or push also if I ping registry I get a 404. Logs
don't say anything special from which I can deduce something is broken even
tough logging is at debug level.
On May 30, 2014 12:08 PM, "Mangled Deutz" notifications@github.com wrote:

Hi @shreyu86 https://github.com/shreyu86

Any error message in the logs when you try to push / pull?
Any error message at startup?

Thanks,

  • Olivier


Reply to this email directly or view it on GitHub
#400 (comment)
.

Contributor

shreyu86 commented May 30, 2014

I get 502 trying to pull or push also if I ping registry I get a 404. Logs
don't say anything special from which I can deduce something is broken even
tough logging is at debug level.
On May 30, 2014 12:08 PM, "Mangled Deutz" notifications@github.com wrote:

Hi @shreyu86 https://github.com/shreyu86

Any error message in the logs when you try to push / pull?
Any error message at startup?

Thanks,

  • Olivier


Reply to this email directly or view it on GitHub
#400 (comment)
.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 May 30, 2014

Member

also if I ping registry I get a 404

What do you mean by that?

Logs don't say anything special from which I can deduce something is broken even tough logging is at debug level

Still, can you gist your registry logs / output?

Thanks a lot.

Member

dmp42 commented May 30, 2014

also if I ping registry I get a 404

What do you mean by that?

Logs don't say anything special from which I can deduce something is broken even tough logging is at debug level

Still, can you gist your registry logs / output?

Thanks a lot.

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 May 30, 2014

Contributor

I'll send the gists soon (out for lunch). Thanks for the fast reply.
On May 30, 2014 12:18 PM, "Mangled Deutz" notifications@github.com wrote:

also if I ping registry I get a 404

What do you mean by that?

Logs don't say anything special from which I can deduce something is
broken even tough logging is at debug level

Still, can you gist your registry logs / output?

Thanks a lot.


Reply to this email directly or view it on GitHub
#400 (comment)
.

Contributor

shreyu86 commented May 30, 2014

I'll send the gists soon (out for lunch). Thanks for the fast reply.
On May 30, 2014 12:18 PM, "Mangled Deutz" notifications@github.com wrote:

also if I ping registry I get a 404

What do you mean by that?

Logs don't say anything special from which I can deduce something is
broken even tough logging is at debug level

Still, can you gist your registry logs / output?

Thanks a lot.


Reply to this email directly or view it on GitHub
#400 (comment)
.

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 May 30, 2014

Contributor

This is the gist of the logs: https://gist.github.com/shreyu86/cc4ae0f6b7f4e329438a have redacted some contents.

also if I ping registry I get a err not 404 but I cant see the page which shows registry version and setting flavor.

when I do a ping using curl here is the response I get:

curl -X GET HOST:5000/v1/_ping
curl: (56) Recv failure: Connection reset by peer

Contributor

shreyu86 commented May 30, 2014

This is the gist of the logs: https://gist.github.com/shreyu86/cc4ae0f6b7f4e329438a have redacted some contents.

also if I ping registry I get a err not 404 but I cant see the page which shows registry version and setting flavor.

when I do a ping using curl here is the response I get:

curl -X GET HOST:5000/v1/_ping
curl: (56) Recv failure: Connection reset by peer

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 May 30, 2014

Member

So, /ping is not 404, right? Instead, the server appears not to be started / bound.

Let me try tomorrow with a s3 region and see if that works for me.

Keep me posted if anything new meanwhile.

Best.

Member

dmp42 commented May 30, 2014

So, /ping is not 404, right? Instead, the server appears not to be started / bound.

Let me try tomorrow with a s3 region and see if that works for me.

Keep me posted if anything new meanwhile.

Best.

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 May 30, 2014

Contributor

Thanks! No Ping is not 404, that was a confusion, as of now I have commented out the S3 region. Registry seems to be running fine without s3_region.

Also in similar situations I used following method to connect to S3:

import boto.s3.connection
def connect_to_s3(settings):

    return boto.s3.connection.S3Connection(
        aws_access_key_id=settings.aws.auth.access_key_id,
        aws_secret_access_key=settings.aws.auth.secret_access_key,
        host=settings.s3.host
    )

In this method assume settings to be a dot dictionary object.
host here being s3-us-west-2.amazonaws.com for west-2 and I use this connection object returned above to do all the S3 operations, I have not faced any issues with the above method.

Also I am out of sync with the latest changes (couple of them major) so opened an issue instead of PR.

Will surely dig into this more.

Contributor

shreyu86 commented May 30, 2014

Thanks! No Ping is not 404, that was a confusion, as of now I have commented out the S3 region. Registry seems to be running fine without s3_region.

Also in similar situations I used following method to connect to S3:

import boto.s3.connection
def connect_to_s3(settings):

    return boto.s3.connection.S3Connection(
        aws_access_key_id=settings.aws.auth.access_key_id,
        aws_secret_access_key=settings.aws.auth.secret_access_key,
        host=settings.s3.host
    )

In this method assume settings to be a dot dictionary object.
host here being s3-us-west-2.amazonaws.com for west-2 and I use this connection object returned above to do all the S3 operations, I have not faced any issues with the above method.

Also I am out of sync with the latest changes (couple of them major) so opened an issue instead of PR.

Will surely dig into this more.

@dmp42 dmp42 added this to the 0.8 milestone May 30, 2014

@dmp42 dmp42 added the bug label May 30, 2014

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 Jun 2, 2014

Contributor

Just curious @dmp42 were you able to repro the issue?

Contributor

shreyu86 commented Jun 2, 2014

Just curious @dmp42 were you able to repro the issue?

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jun 2, 2014

Member

I haven't had time yet to get to it, sorry for that - will sure do later today and keep you posted though!

Member

dmp42 commented Jun 2, 2014

I haven't had time yet to get to it, sorry for that - will sure do later today and keep you posted though!

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 Jun 2, 2014

Contributor

Thanks!
On Jun 2, 2014 7:39 AM, "Mangled Deutz" notifications@github.com wrote:

I haven't had time yet to get to it, sorry for that - will sure do later
today and keep you posted though!


Reply to this email directly or view it on GitHub
#400 (comment)
.

Contributor

shreyu86 commented Jun 2, 2014

Thanks!
On Jun 2, 2014 7:39 AM, "Mangled Deutz" notifications@github.com wrote:

I haven't had time yet to get to it, sorry for that - will sure do later
today and keep you posted though!


Reply to this email directly or view it on GitHub
#400 (comment)
.

skarnik-rmn added a commit to skarnik-rmn/docker-registry that referenced this issue Jun 2, 2014

@dmp42 dmp42 referenced this issue Jun 2, 2014

Closed

Solution for #400 #405

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 Jun 2, 2014

Contributor

I've opened #405 as a solution to this issue. I have patched our private
registry and can confirm that everything (Push, Pull, Search) operations
are working normally after this change.

-Shreyas

Shreyas

On Mon, Jun 2, 2014 at 7:40 AM, Shreyas Karnik shreyu86@gmail.com wrote:

Thanks!
On Jun 2, 2014 7:39 AM, "Mangled Deutz" notifications@github.com wrote:

I haven't had time yet to get to it, sorry for that - will sure do later
today and keep you posted though!


Reply to this email directly or view it on GitHub
#400 (comment)
.

Contributor

shreyu86 commented Jun 2, 2014

I've opened #405 as a solution to this issue. I have patched our private
registry and can confirm that everything (Push, Pull, Search) operations
are working normally after this change.

-Shreyas

Shreyas

On Mon, Jun 2, 2014 at 7:40 AM, Shreyas Karnik shreyu86@gmail.com wrote:

Thanks!
On Jun 2, 2014 7:39 AM, "Mangled Deutz" notifications@github.com wrote:

I haven't had time yet to get to it, sorry for that - will sure do later
today and keep you posted though!


Reply to this email directly or view it on GitHub
#400 (comment)
.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jun 2, 2014

Member

@shreyu86

I just created a new bucket, in the us-west-2 region - I'm using us-west-2 as a region in the configuration and it works for me...

So, we need to dig deeper.

Can you provide with additional infos?

  • the logs you copied (the "with region" scenario) seems to be cut too early - what happens exactly? it hangs there and does nothing more? it crashes? or is there more after that?
  • what version of the registry are you running? (I assume 0.7.0) - did you try and/or have the problem with 0.6.9?
  • are you running it from the docker container? or from a pip install?

Thanks a lot!

Member

dmp42 commented Jun 2, 2014

@shreyu86

I just created a new bucket, in the us-west-2 region - I'm using us-west-2 as a region in the configuration and it works for me...

So, we need to dig deeper.

Can you provide with additional infos?

  • the logs you copied (the "with region" scenario) seems to be cut too early - what happens exactly? it hangs there and does nothing more? it crashes? or is there more after that?
  • what version of the registry are you running? (I assume 0.7.0) - did you try and/or have the problem with 0.6.9?
  • are you running it from the docker container? or from a pip install?

Thanks a lot!

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 Jun 2, 2014

Contributor

@dmp42 here is the information you requested.

  • the logs you copied (the "with region" scenario) seems to be cut too early - what happens exactly? it hangs there and does nothing more? it crashes? or is there more after that?
    • It just hangs at those logs, nothing further happens.
  • what version of the registry are you running? (I assume 0.7.0) - did you try and/or have the problem with 0.6.9?
    • I am running 0.7.0 I did not try 0.6.9.
  • are you running it from the docker container? or from a pip install?
    • I am running it inside docker container pulled from index.docker.io

Thanks for looking into this.

Contributor

shreyu86 commented Jun 2, 2014

@dmp42 here is the information you requested.

  • the logs you copied (the "with region" scenario) seems to be cut too early - what happens exactly? it hangs there and does nothing more? it crashes? or is there more after that?
    • It just hangs at those logs, nothing further happens.
  • what version of the registry are you running? (I assume 0.7.0) - did you try and/or have the problem with 0.6.9?
    • I am running 0.7.0 I did not try 0.6.9.
  • are you running it from the docker container? or from a pip install?
    • I am running it inside docker container pulled from index.docker.io

Thanks for looking into this.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jun 2, 2014

Member

Can you get this gist: https://gist.github.com/dmp42/8436a9be2c569bd75965

And try run it from where you are? (obviously replace bucketname = 'XXXX'
awsid = 'XXX'
awssec = 'XXX' with appropriate values).

Also, can you put debug = 2 inside your boto.cfg file (in your home)?

Thanks a lot!

Member

dmp42 commented Jun 2, 2014

Can you get this gist: https://gist.github.com/dmp42/8436a9be2c569bd75965

And try run it from where you are? (obviously replace bucketname = 'XXXX'
awsid = 'XXX'
awssec = 'XXX' with appropriate values).

Also, can you put debug = 2 inside your boto.cfg file (in your home)?

Thanks a lot!

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 Jun 2, 2014

Contributor

It worked. Here is the output:

Gonna connect without region
0.257856845856
Gonna connect with region
0.0846657752991

Contributor

shreyu86 commented Jun 2, 2014

It worked. Here is the output:

Gonna connect without region
0.257856845856
Gonna connect with region
0.0846657752991

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jun 2, 2014

Member

This is getting really weird.

@shreyu86:

  • do you use the configuration variable storage_redirect?
  • can you copy the exact command lines you are using to launch the registry, both with region and without region
  • same thing for your configuration files, with and without region

Sorry for the long walk, but I see no other way to get to the bottom of this...

Member

dmp42 commented Jun 2, 2014

This is getting really weird.

@shreyu86:

  • do you use the configuration variable storage_redirect?
  • can you copy the exact command lines you are using to launch the registry, both with region and without region
  • same thing for your configuration files, with and without region

Sorry for the long walk, but I see no other way to get to the bottom of this...

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 Jun 2, 2014

Contributor
  • do you use the configuration variable storage_redirect?
    • no it is set to false
  • can you copy the exact command lines you are using to launch the registry, both with region and without region
  • command lines are:
# Commands to invoke registry
 docker run -d -p 5000:5000 -v /etc/docker-registry/config.yml:/opt/docker-registry/config.yml -e SETTINGS_FLAVOR=prod -e DOCKER_REGISTRY_CONFIG=/opt/docker-registry/config/config.yml -e GUNICORN_WORKERS=10 registry:latest
  • same thing for your configuration files, with and without region:
    • for this I edit the configuration file variable s3_region to set a region I set it to us-west-2 and to connect without region I comment it out.

Here are my redacted settings just in case you need to take a look at those:

https://gist.github.com/shreyu86/52652677440596e669b4

Contributor

shreyu86 commented Jun 2, 2014

  • do you use the configuration variable storage_redirect?
    • no it is set to false
  • can you copy the exact command lines you are using to launch the registry, both with region and without region
  • command lines are:
# Commands to invoke registry
 docker run -d -p 5000:5000 -v /etc/docker-registry/config.yml:/opt/docker-registry/config.yml -e SETTINGS_FLAVOR=prod -e DOCKER_REGISTRY_CONFIG=/opt/docker-registry/config/config.yml -e GUNICORN_WORKERS=10 registry:latest
  • same thing for your configuration files, with and without region:
    • for this I edit the configuration file variable s3_region to set a region I set it to us-west-2 and to connect without region I comment it out.

Here are my redacted settings just in case you need to take a look at those:

https://gist.github.com/shreyu86/52652677440596e669b4

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jun 2, 2014

Member
# Commands to invoke registry
 docker run -d -p 5000:5000 -v /etc/docker-registry/config.yml:/opt/docker-registry/config.yml -e SETTINGS_FLAVOR=prod -e DOCKER_REGISTRY_CONFIG=/opt/docker-registry/config/config.yml

Unless I'm mistaken, this can't work: you are telling the registry to use /opt/docker-registry/config/config.yml while you mount /opt/docker-registry/config.yml

Is this a typo or did you move some stuff and were launching with this?

Thanks.

Member

dmp42 commented Jun 2, 2014

# Commands to invoke registry
 docker run -d -p 5000:5000 -v /etc/docker-registry/config.yml:/opt/docker-registry/config.yml -e SETTINGS_FLAVOR=prod -e DOCKER_REGISTRY_CONFIG=/opt/docker-registry/config/config.yml

Unless I'm mistaken, this can't work: you are telling the registry to use /opt/docker-registry/config/config.yml while you mount /opt/docker-registry/config.yml

Is this a typo or did you move some stuff and were launching with this?

Thanks.

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 Jun 2, 2014

Contributor

Err typo in cleaning and redacting.

Correct set of commands are

docker run -d -p 5000:5000 -v
/etc/docker-registry/config.yml:/opt/docker-registry/config/config.yml -e
SETTINGS_FLAVOR=prod -e
DOCKER_REGISTRY_CONFIG=/opt/docker-registry/config/config.yml -e
GUNICORN_WORKERS=10 registry:latest
Contributor

shreyu86 commented Jun 2, 2014

Err typo in cleaning and redacting.

Correct set of commands are

docker run -d -p 5000:5000 -v
/etc/docker-registry/config.yml:/opt/docker-registry/config/config.yml -e
SETTINGS_FLAVOR=prod -e
DOCKER_REGISTRY_CONFIG=/opt/docker-registry/config/config.yml -e
GUNICORN_WORKERS=10 registry:latest
@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jun 2, 2014

Member

Ok... this doesn't make any sense...

If you successfully connected using the test script I provided, there is absolutely no reason why the registry wouldn't connect. This is almost exactly the same code...

One last shot: did you run the test script on the same machine that you use to run the registry?

Otherwise, I see no resolution to this :-(

As I can't reproduce on my own setup - there is only two possibilities:

  • we find how to reproduce
  • you manage to debug deep inside boto to see what happens
  • you trust me enough to grant me a temporary access to your bucket

Sorry for not being able to be more helpful...

Member

dmp42 commented Jun 2, 2014

Ok... this doesn't make any sense...

If you successfully connected using the test script I provided, there is absolutely no reason why the registry wouldn't connect. This is almost exactly the same code...

One last shot: did you run the test script on the same machine that you use to run the registry?

Otherwise, I see no resolution to this :-(

As I can't reproduce on my own setup - there is only two possibilities:

  • we find how to reproduce
  • you manage to debug deep inside boto to see what happens
  • you trust me enough to grant me a temporary access to your bucket

Sorry for not being able to be more helpful...

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 Jun 2, 2014

Contributor

One last shot: did you run the test script on the same machine that you use to run the registry?
Yes both on my dev machine and on the registry machine inside of ec2 here is the output for the registry machine:

Gonna connect without region
0.0115849971771
Gonna connect with region
0.0101661682129

I guess inside of EC2 it is a minimal difference with and without regions but not sure about its impact.

I will try my hand at debugging boto.

Regarding the bucket, will have to jump through a lot of permission issues so right now will try to debug this issue at my end and post findings here.

Thanks for the help, also I will try it with a fresh S3 bucket as whenever I update the registry I use the same bucket, I've been using private registry since a long time so will do this test with a fresh S3 bucket and see if I face the same issue or not.

I will surely post any findings here. Thanks @dmp42 I really appreciate your help.

Contributor

shreyu86 commented Jun 2, 2014

One last shot: did you run the test script on the same machine that you use to run the registry?
Yes both on my dev machine and on the registry machine inside of ec2 here is the output for the registry machine:

Gonna connect without region
0.0115849971771
Gonna connect with region
0.0101661682129

I guess inside of EC2 it is a minimal difference with and without regions but not sure about its impact.

I will try my hand at debugging boto.

Regarding the bucket, will have to jump through a lot of permission issues so right now will try to debug this issue at my end and post findings here.

Thanks for the help, also I will try it with a fresh S3 bucket as whenever I update the registry I use the same bucket, I've been using private registry since a long time so will do this test with a fresh S3 bucket and see if I face the same issue or not.

I will surely post any findings here. Thanks @dmp42 I really appreciate your help.

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 Jun 2, 2014

Contributor

Actually I think I am OK with not using explicit region declaration in the registry config because while debugging this issue I found that boto redirects the bucket to the underlying region, some logs (not in that order from registry logs)

2014-06-02 17:00:27,119 DEBUG: Host: BUCKET.s3.amazonaws.com
2014-06-02 17:00:27,210 DEBUG: Redirecting: http://BUCKET-us-west-2.amazonaws.com/

I think right now this is one short term optimization I can make, but will visit this during off hours to get to the bottom.

Contributor

shreyu86 commented Jun 2, 2014

Actually I think I am OK with not using explicit region declaration in the registry config because while debugging this issue I found that boto redirects the bucket to the underlying region, some logs (not in that order from registry logs)

2014-06-02 17:00:27,119 DEBUG: Host: BUCKET.s3.amazonaws.com
2014-06-02 17:00:27,210 DEBUG: Redirecting: http://BUCKET-us-west-2.amazonaws.com/

I think right now this is one short term optimization I can make, but will visit this during off hours to get to the bottom.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jun 2, 2014

Member

@shreyu86 Ok - please keep me update of any progress on this.

Member

dmp42 commented Jun 2, 2014

@shreyu86 Ok - please keep me update of any progress on this.

@ddeaguiar

This comment has been minimized.

Show comment
Hide comment
@ddeaguiar

ddeaguiar Jun 5, 2014

I'm also running into this issue. I'm using registry:latest on docker.io. Here's my startup command:

docker run \
       --rm \
       -it \
       -e SETTINGS_FLAVOR=s3 \
       -e AWS_REGION=us-east-1 \
       -e AWS_BUCKET=my-bucket \
       -e AWS_ENCRYPT=true \
       -e AWS_SECURE=true \
       -e STORAGE_PATH=/registry \
       -e AWS_SECRET=REDACTED \
       -e AWS_KEY=SECRET \
       -e SEARCH_BACKEND=sqlalchemy \
       -P \
       registry

It seems that setting the region doesn't solve the broken_pipe error I'm getting on push.

ddeaguiar commented Jun 5, 2014

I'm also running into this issue. I'm using registry:latest on docker.io. Here's my startup command:

docker run \
       --rm \
       -it \
       -e SETTINGS_FLAVOR=s3 \
       -e AWS_REGION=us-east-1 \
       -e AWS_BUCKET=my-bucket \
       -e AWS_ENCRYPT=true \
       -e AWS_SECURE=true \
       -e STORAGE_PATH=/registry \
       -e AWS_SECRET=REDACTED \
       -e AWS_KEY=SECRET \
       -e SEARCH_BACKEND=sqlalchemy \
       -P \
       registry

It seems that setting the region doesn't solve the broken_pipe error I'm getting on push.

@phemmer

This comment has been minimized.

Show comment
Hide comment
@phemmer

phemmer Jun 5, 2014

Contributor

Having this issue as well. When I start with a region set, it just hangs during startup. Without the region it starts up fine.

# docker run \
  -e SETTINGS_FLAVOR=s3 \
  -e AWS_REGION=us-west-1 \
  -e AWS_BUCKET=cloudcom-cliff-packages \
  -e AWS_KEY=FOO \
  -e AWS_SECRET=BAR \
  -e AWS_PATH=/docker/registry \
  -e STORAGE_PATH=/docker/registry \
  -e SEARCH_BACKEND=sqlalchemy  \
  -t -i \
  registry:0.7.0                
2014-06-05 19:53:16,282 DEBUG: Will return docker-registry.drivers.s3.Storage
2014-06-05 19:53:16,284 DEBUG: Using access key provided by client.
2014-06-05 19:53:16,284 DEBUG: Using secret key provided by client.
2014-06-05 19:53:16,285 DEBUG: path=/
2014-06-05 19:53:16,285 DEBUG: auth_path=/cloudcom-cliff-packages/
2014-06-05 19:53:16,285 DEBUG: Method: HEAD
2014-06-05 19:53:16,285 DEBUG: Path: /
2014-06-05 19:53:16,285 DEBUG: Data: 
2014-06-05 19:53:16,286 DEBUG: Headers: {}
2014-06-05 19:53:16,286 DEBUG: Host: cloudcom-cliff-packages.s3-us-west-1.amazonaws.com
2014-06-05 19:53:16,286 DEBUG: Port: 80
2014-06-05 19:53:16,286 DEBUG: Params: {}
2014-06-05 19:53:16,286 DEBUG: establishing HTTP connection: kwargs={'port': 80, 'timeout': 70}
2014-06-05 19:53:16,286 DEBUG: Token: None
2014-06-05 19:53:16,287 DEBUG: StringToSign:
HEAD


Thu, 05 Jun 2014 19:53:16 GMT
/cloudcom-cliff-packages/
2014-06-05 19:53:16,287 DEBUG: Signature:
AWS AKIAI72GUHBWOGEM6O7A:UCa/+gWFxmm3+zg7r7YEedATvXI=
Contributor

phemmer commented Jun 5, 2014

Having this issue as well. When I start with a region set, it just hangs during startup. Without the region it starts up fine.

# docker run \
  -e SETTINGS_FLAVOR=s3 \
  -e AWS_REGION=us-west-1 \
  -e AWS_BUCKET=cloudcom-cliff-packages \
  -e AWS_KEY=FOO \
  -e AWS_SECRET=BAR \
  -e AWS_PATH=/docker/registry \
  -e STORAGE_PATH=/docker/registry \
  -e SEARCH_BACKEND=sqlalchemy  \
  -t -i \
  registry:0.7.0                
2014-06-05 19:53:16,282 DEBUG: Will return docker-registry.drivers.s3.Storage
2014-06-05 19:53:16,284 DEBUG: Using access key provided by client.
2014-06-05 19:53:16,284 DEBUG: Using secret key provided by client.
2014-06-05 19:53:16,285 DEBUG: path=/
2014-06-05 19:53:16,285 DEBUG: auth_path=/cloudcom-cliff-packages/
2014-06-05 19:53:16,285 DEBUG: Method: HEAD
2014-06-05 19:53:16,285 DEBUG: Path: /
2014-06-05 19:53:16,285 DEBUG: Data: 
2014-06-05 19:53:16,286 DEBUG: Headers: {}
2014-06-05 19:53:16,286 DEBUG: Host: cloudcom-cliff-packages.s3-us-west-1.amazonaws.com
2014-06-05 19:53:16,286 DEBUG: Port: 80
2014-06-05 19:53:16,286 DEBUG: Params: {}
2014-06-05 19:53:16,286 DEBUG: establishing HTTP connection: kwargs={'port': 80, 'timeout': 70}
2014-06-05 19:53:16,286 DEBUG: Token: None
2014-06-05 19:53:16,287 DEBUG: StringToSign:
HEAD


Thu, 05 Jun 2014 19:53:16 GMT
/cloudcom-cliff-packages/
2014-06-05 19:53:16,287 DEBUG: Signature:
AWS AKIAI72GUHBWOGEM6O7A:UCa/+gWFxmm3+zg7r7YEedATvXI=
@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jun 5, 2014

Member

@ddeaguiar or @phemmer

By any chance, would any of you be able to grant me a (temporary) access to your problematic bucket?

Like I said earlier, I can't reproduce here, and that would really help diagnosing this.

Thanks!

Member

dmp42 commented Jun 5, 2014

@ddeaguiar or @phemmer

By any chance, would any of you be able to grant me a (temporary) access to your problematic bucket?

Like I said earlier, I can't reproduce here, and that would really help diagnosing this.

Thanks!

@phemmer

This comment has been minimized.

Show comment
Hide comment
@phemmer

phemmer Jun 6, 2014

Contributor

I created a brand new bucket (bucket currently completely empty), and a brand new set of credentials. The credentials below are functional, but have read only access to the bucket.

docker run \
  -e SETTINGS_FLAVOR=s3 \
  -e AWS_BUCKET=docker-registry-400-test \
  -e AWS_KEY=AKIAI4CU3BUIRSLWC54Q \
  -e AWS_SECRET=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 \
  -e AWS_PATH=/docker/registry \
  -e STORAGE_PATH=/docker/registry \
  -e SEARCH_BACKEND=sqlalchemy \
  -e AWS_REGION=us-west-1 \
  -t -i \
  registry:0.7.0
2014-06-06 01:49:27,853 DEBUG: Will return docker-registry.drivers.s3.Storage
2014-06-06 01:49:27,854 DEBUG: Using access key provided by client.
2014-06-06 01:49:27,854 DEBUG: Using secret key provided by client.
2014-06-06 01:49:27,854 DEBUG: path=/
2014-06-06 01:49:27,854 DEBUG: auth_path=/docker-registry-400-test/
2014-06-06 01:49:27,854 DEBUG: Method: HEAD
2014-06-06 01:49:27,854 DEBUG: Path: /
2014-06-06 01:49:27,854 DEBUG: Data: 
2014-06-06 01:49:27,854 DEBUG: Headers: {}
2014-06-06 01:49:27,855 DEBUG: Host: docker-registry-400-test.s3-us-west-1.amazonaws.com
2014-06-06 01:49:27,855 DEBUG: Port: 80
2014-06-06 01:49:27,855 DEBUG: Params: {}
2014-06-06 01:49:27,855 DEBUG: establishing HTTP connection: kwargs={'port': 80, 'timeout': 70}
2014-06-06 01:49:27,855 DEBUG: Token: None
2014-06-06 01:49:27,855 DEBUG: StringToSign:
HEAD


Fri, 06 Jun 2014 01:49:27 GMT
/docker-registry-400-test/
2014-06-06 01:49:27,855 DEBUG: Signature:
AWS AKIAI4CU3BUIRSLWC54Q:AVbAuKqhCl0fLI9uLjz7mws8zSY=
^CKeyboardInterrupt
** [Bugsnag] No API key configured, couldn't notify
Traceback (most recent call last):
  File "/usr/local/bin/docker-registry", line 9, in <module>
    load_entry_point('docker-registry==0.7.0', 'console_scripts', 'docker-registry')()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 351, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2363, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2088, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/run.py", line 16, in <module>
    from .tags import *  # noqa
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/tags.py", line 21, in <module>
    store = storage.load()
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/storage/__init__.py", line 36, in load
    _storage[kind] = engine.fetch(kind)(None, config=cfg)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/drivers/s3.py", line 65, in __init__
    super(Storage, self).__init__(path, config)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/boto.py", line 135, in __init__
    self._config.boto_bucket)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 471, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 490, in head_bucket
    response = self.make_request('HEAD', bucket_name, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 633, in make_request
    retry_handler=retry_handler
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1046, in make_request
    retry_handler=retry_handler)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 922, in _mexe
    request.body, request.headers)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 570, in create_connection
    for res in getaddrinfo(host, port, 0 if has_ipv6 else AF_INET, SOCK_STREAM):
  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 621, in getaddrinfo
    return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
  File "/usr/local/lib/python2.7/dist-packages/gevent/resolver_thread.py", line 34, in getaddrinfo
    return self.pool.apply_e(self.expected_errors, _socket.getaddrinfo, args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gevent/threadpool.py", line 222, in apply_e
    success, result = self.spawn(wrap_errors, expected_errors, function, args, kwargs).get()
  File "/usr/local/lib/python2.7/dist-packages/gevent/event.py", line 233, in get
    result = self.hub.switch()
  File "/usr/local/lib/python2.7/dist-packages/gevent/hub.py", line 331, in switch
    return greenlet.switch(self)
KeyboardInterrupt

Also, this was working just fine as of 3964f29. I don't know what commit broke it, but I was running a custom build from that commit until when I tried to upgrade today.

Contributor

phemmer commented Jun 6, 2014

I created a brand new bucket (bucket currently completely empty), and a brand new set of credentials. The credentials below are functional, but have read only access to the bucket.

docker run \
  -e SETTINGS_FLAVOR=s3 \
  -e AWS_BUCKET=docker-registry-400-test \
  -e AWS_KEY=AKIAI4CU3BUIRSLWC54Q \
  -e AWS_SECRET=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 \
  -e AWS_PATH=/docker/registry \
  -e STORAGE_PATH=/docker/registry \
  -e SEARCH_BACKEND=sqlalchemy \
  -e AWS_REGION=us-west-1 \
  -t -i \
  registry:0.7.0
2014-06-06 01:49:27,853 DEBUG: Will return docker-registry.drivers.s3.Storage
2014-06-06 01:49:27,854 DEBUG: Using access key provided by client.
2014-06-06 01:49:27,854 DEBUG: Using secret key provided by client.
2014-06-06 01:49:27,854 DEBUG: path=/
2014-06-06 01:49:27,854 DEBUG: auth_path=/docker-registry-400-test/
2014-06-06 01:49:27,854 DEBUG: Method: HEAD
2014-06-06 01:49:27,854 DEBUG: Path: /
2014-06-06 01:49:27,854 DEBUG: Data: 
2014-06-06 01:49:27,854 DEBUG: Headers: {}
2014-06-06 01:49:27,855 DEBUG: Host: docker-registry-400-test.s3-us-west-1.amazonaws.com
2014-06-06 01:49:27,855 DEBUG: Port: 80
2014-06-06 01:49:27,855 DEBUG: Params: {}
2014-06-06 01:49:27,855 DEBUG: establishing HTTP connection: kwargs={'port': 80, 'timeout': 70}
2014-06-06 01:49:27,855 DEBUG: Token: None
2014-06-06 01:49:27,855 DEBUG: StringToSign:
HEAD


Fri, 06 Jun 2014 01:49:27 GMT
/docker-registry-400-test/
2014-06-06 01:49:27,855 DEBUG: Signature:
AWS AKIAI4CU3BUIRSLWC54Q:AVbAuKqhCl0fLI9uLjz7mws8zSY=
^CKeyboardInterrupt
** [Bugsnag] No API key configured, couldn't notify
Traceback (most recent call last):
  File "/usr/local/bin/docker-registry", line 9, in <module>
    load_entry_point('docker-registry==0.7.0', 'console_scripts', 'docker-registry')()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 351, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2363, in load_entry_point
    return ep.load()
  File "/usr/lib/python2.7/dist-packages/pkg_resources.py", line 2088, in load
    entry = __import__(self.module_name, globals(),globals(), ['__name__'])
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/run.py", line 16, in <module>
    from .tags import *  # noqa
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/tags.py", line 21, in <module>
    store = storage.load()
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/storage/__init__.py", line 36, in load
    _storage[kind] = engine.fetch(kind)(None, config=cfg)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/drivers/s3.py", line 65, in __init__
    super(Storage, self).__init__(path, config)
  File "/usr/local/lib/python2.7/dist-packages/docker_registry/core/boto.py", line 135, in __init__
    self._config.boto_bucket)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 471, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 490, in head_bucket
    response = self.make_request('HEAD', bucket_name, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 633, in make_request
    retry_handler=retry_handler
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1046, in make_request
    retry_handler=retry_handler)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 922, in _mexe
    request.body, request.headers)
  File "/usr/lib/python2.7/httplib.py", line 973, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 791, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 772, in connect
    self.timeout, self.source_address)
  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 570, in create_connection
    for res in getaddrinfo(host, port, 0 if has_ipv6 else AF_INET, SOCK_STREAM):
  File "/usr/local/lib/python2.7/dist-packages/gevent/socket.py", line 621, in getaddrinfo
    return get_hub().resolver.getaddrinfo(host, port, family, socktype, proto, flags)
  File "/usr/local/lib/python2.7/dist-packages/gevent/resolver_thread.py", line 34, in getaddrinfo
    return self.pool.apply_e(self.expected_errors, _socket.getaddrinfo, args, kwargs)
  File "/usr/local/lib/python2.7/dist-packages/gevent/threadpool.py", line 222, in apply_e
    success, result = self.spawn(wrap_errors, expected_errors, function, args, kwargs).get()
  File "/usr/local/lib/python2.7/dist-packages/gevent/event.py", line 233, in get
    result = self.hub.switch()
  File "/usr/local/lib/python2.7/dist-packages/gevent/hub.py", line 331, in switch
    return greenlet.switch(self)
KeyboardInterrupt

Also, this was working just fine as of 3964f29. I don't know what commit broke it, but I was running a custom build from that commit until when I tried to upgrade today.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jun 6, 2014

Member

Thanks a ton! I'll look into it ASAP.

About what changed from then - mainly we upgraded boto to the latest version.

Member

dmp42 commented Jun 6, 2014

Thanks a ton! I'll look into it ASAP.

About what changed from then - mainly we upgraded boto to the latest version.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Jun 6, 2014

Member

Works from outside the container (vs: pip install docker-registry) but doesn't from inside the container...

I'm thinking:

  • libevent issue
  • a/o networking issue
Member

dmp42 commented Jun 6, 2014

Works from outside the container (vs: pip install docker-registry) but doesn't from inside the container...

I'm thinking:

  • libevent issue
  • a/o networking issue
@josephg

This comment has been minimized.

Show comment
Hide comment
@josephg

josephg Jun 17, 2014

+1 I'm having the same issue.

josephg commented Jun 17, 2014

+1 I'm having the same issue.

@danielschonfeld

This comment has been minimized.

Show comment
Hide comment
@danielschonfeld

danielschonfeld Jun 19, 2014

+1 Here.... same issue, tried 0.7.0 and 0.7.3 both are affected.

Interestingly enough, if you omit the AWS_REGION from the environment, the container fails instead of hanging with an boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden trying to call return self.head_bucket(bucket_name, headers=headers). Adding the AWS_REGION=us-east-1 makes it hang.

danielschonfeld commented Jun 19, 2014

+1 Here.... same issue, tried 0.7.0 and 0.7.3 both are affected.

Interestingly enough, if you omit the AWS_REGION from the environment, the container fails instead of hanging with an boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden trying to call return self.head_bucket(bucket_name, headers=headers). Adding the AWS_REGION=us-east-1 makes it hang.

@phemmer

This comment has been minimized.

Show comment
Hide comment
@phemmer

phemmer Jun 19, 2014

Contributor

@danielschonfeld Make sure your key & secret have permission to access the bucket. Removing AWS_REGION is the current workaround for this issue, so you've got something else going on for that 403 error.

Contributor

phemmer commented Jun 19, 2014

@danielschonfeld Make sure your key & secret have permission to access the bucket. Removing AWS_REGION is the current workaround for this issue, so you've got something else going on for that 403 error.

@danielschonfeld

This comment has been minimized.

Show comment
Hide comment
@danielschonfeld

danielschonfeld Jun 19, 2014

@phemmer my key/secret belong to an IAM user who is part of a group that has the "Administrator" policy which looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    }
  ]
}

I've also used that key/secret combination with s3cmd on my macbook to verify i can upload, read and list files and directories inside that s3 bucket

danielschonfeld commented Jun 19, 2014

@phemmer my key/secret belong to an IAM user who is part of a group that has the "Administrator" policy which looks like this:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    }
  ]
}

I've also used that key/secret combination with s3cmd on my macbook to verify i can upload, read and list files and directories inside that s3 bucket

@dmp42 dmp42 modified the milestones: 0.8, ASAP Aug 7, 2014

@mattheworiordan

This comment has been minimized.

Show comment
Hide comment
@mattheworiordan

mattheworiordan Aug 8, 2014

I have a bucket in eu-west-1 and was unable to get Docker Registry to start, it would hang every time without any indication of what was wrong. The only way I could get this to work without hanging was to:

Create the file /etc/boto.cfg with:

[Boto]
metadata_service_timeout = 2.0
metadata_service_num_attempts = 3

[S3]
region = <%= ENV['DOCKER_S3_REGION'] %>

And then my registry config.yml is configured as follows under the s3 section:

s3: &s3
  <<: *common
  storage: s3
  # s3_region: Providing this will cause Registry to fail, this is configured in /etc/boto.cfg instead
  s3_bucket: _env:DOCKER_S3_BUCKET
  boto_bucket: _env:DOCKER_S3_BUCKET
  storage_path: _env:DOCKER_S3_PREFIX
  s3_encrypt: false
  s3_secure: true
  s3_access_key: _env:AWS_ACCESS_KEY_ID
  s3_secret_key: _env:AWS_SECRET_ACCESS_KEY

Note s3_region is deliberately left out, but set within the boto settings. This is an odd bug and shoudl be fixed in docker registry.

mattheworiordan commented Aug 8, 2014

I have a bucket in eu-west-1 and was unable to get Docker Registry to start, it would hang every time without any indication of what was wrong. The only way I could get this to work without hanging was to:

Create the file /etc/boto.cfg with:

[Boto]
metadata_service_timeout = 2.0
metadata_service_num_attempts = 3

[S3]
region = <%= ENV['DOCKER_S3_REGION'] %>

And then my registry config.yml is configured as follows under the s3 section:

s3: &s3
  <<: *common
  storage: s3
  # s3_region: Providing this will cause Registry to fail, this is configured in /etc/boto.cfg instead
  s3_bucket: _env:DOCKER_S3_BUCKET
  boto_bucket: _env:DOCKER_S3_BUCKET
  storage_path: _env:DOCKER_S3_PREFIX
  s3_encrypt: false
  s3_secure: true
  s3_access_key: _env:AWS_ACCESS_KEY_ID
  s3_secret_key: _env:AWS_SECRET_ACCESS_KEY

Note s3_region is deliberately left out, but set within the boto settings. This is an odd bug and shoudl be fixed in docker registry.

@gerhard

This comment has been minimized.

Show comment
Hide comment
@gerhard

gerhard Aug 9, 2014

@mattheworiordan thanks, suggested fix worked a charm. I was getting this on latest stable, v0.7.3

gerhard commented Aug 9, 2014

@mattheworiordan thanks, suggested fix worked a charm. I was getting this on latest stable, v0.7.3

@kmccormack

This comment has been minimized.

Show comment
Hide comment
@kmccormack

kmccormack Aug 14, 2014

Hi, @mattheworiordan's fix didn't work for me on v0.7.3. I ended up spending a lot of time debugging and then just hacked the /docker_registry/drivers/s3.py file to hard-code the endpoint I'm interested in. So, I changed the following lines (96-99):

        return boto.s3.connection.S3Connection(
            self._config.s3_access_key,
            self._config.s3_secret_key,
            **kwargs)`

to

      return boto.s3.connection.S3Connection(
            aws_access_key_id=self._config.s3_access_key,
            aws_secret_access_key=self._config.s3_secret_key,
            host="s3-eu-west-1.amazonaws.com",
            **kwargs)`

and I didn't specify the AWS_REGION environment variable.

In order to try to reproduce the bug, I created the following small python script:

import gevent.monkey
gevent.monkey.patch_all()
import logging
logging.basicConfig(filename="boto.log", level=logging.DEBUG)
import boto
boto.config.set('Boto', 'debug', '2')
boto.config.set('Boto', 'is_secure', 'no')
import os
import boto.s3
s3 = boto.s3.connect_to_region(
    region_name='eu-west-1',
    aws_access_key_id=os.environ['AWS_KEY'],
    aws_secret_access_key=os.environ['AWS_SECRET'])
mybucket = s3.get_bucket('MY_BUCKET_IN_EU_WEST')`

Where MY_BUCKET_IN_EU_WEST is obviously a bucket in the problematic EU_WEST_1 region. This script worked fine when I ran it in the Docker registry container on the command line. I added lots of debug info to various python scripts to see what was going on and it seemed things just stopped in the gevent code at DNS resolution - It doesn't even end up making a HTTP request to Amazon. I got the same stack trace as @phemmer . My best guess is that something is happening when you use the region block of code in s3.py where the gevent.monkey code isn't properly swapping the calls for DNS resolution to use the gevent Socket instead of the standard Python Socket, but I'm not an expert on Python so can't be sure. Hope this helps anyone trying to resolve this bug or find a workaround.

kmccormack commented Aug 14, 2014

Hi, @mattheworiordan's fix didn't work for me on v0.7.3. I ended up spending a lot of time debugging and then just hacked the /docker_registry/drivers/s3.py file to hard-code the endpoint I'm interested in. So, I changed the following lines (96-99):

        return boto.s3.connection.S3Connection(
            self._config.s3_access_key,
            self._config.s3_secret_key,
            **kwargs)`

to

      return boto.s3.connection.S3Connection(
            aws_access_key_id=self._config.s3_access_key,
            aws_secret_access_key=self._config.s3_secret_key,
            host="s3-eu-west-1.amazonaws.com",
            **kwargs)`

and I didn't specify the AWS_REGION environment variable.

In order to try to reproduce the bug, I created the following small python script:

import gevent.monkey
gevent.monkey.patch_all()
import logging
logging.basicConfig(filename="boto.log", level=logging.DEBUG)
import boto
boto.config.set('Boto', 'debug', '2')
boto.config.set('Boto', 'is_secure', 'no')
import os
import boto.s3
s3 = boto.s3.connect_to_region(
    region_name='eu-west-1',
    aws_access_key_id=os.environ['AWS_KEY'],
    aws_secret_access_key=os.environ['AWS_SECRET'])
mybucket = s3.get_bucket('MY_BUCKET_IN_EU_WEST')`

Where MY_BUCKET_IN_EU_WEST is obviously a bucket in the problematic EU_WEST_1 region. This script worked fine when I ran it in the Docker registry container on the command line. I added lots of debug info to various python scripts to see what was going on and it seemed things just stopped in the gevent code at DNS resolution - It doesn't even end up making a HTTP request to Amazon. I got the same stack trace as @phemmer . My best guess is that something is happening when you use the region block of code in s3.py where the gevent.monkey code isn't properly swapping the calls for DNS resolution to use the gevent Socket instead of the standard Python Socket, but I'm not an expert on Python so can't be sure. Hope this helps anyone trying to resolve this bug or find a workaround.

@spesnova

This comment has been minimized.

Show comment
Hide comment
@spesnova

spesnova Aug 15, 2014

Here is my starting script. I don't specify region, and it works.
If I specify -e AWS_REGION=ap-northeast-1 \ also, it hangs me too.
I hope this helps.

#!/bin/bash
docker run \
   --name registry \
   -e SETTINGS_FLAVOR=prod \
   -e AWS_BUCKET=<your-s3-bucker-name> \
   -e AWS_KEY=<your-aws-key> \
   -e AWS_SECRET=<your-aws-secret> \
   -e DOCKER_REGISTRY_CONFIG=/docker-registry/config/config_sample.yml \
   -e STORAGE_PATH=/ \
   -e SEARCH_BACKEND=sqlalchemy \
   -e GUNICORN_WORKERS=2 \
   -p 5000:5000 \
   -d \
   registry:0.7.3 \
   docker-registry

spesnova commented Aug 15, 2014

Here is my starting script. I don't specify region, and it works.
If I specify -e AWS_REGION=ap-northeast-1 \ also, it hangs me too.
I hope this helps.

#!/bin/bash
docker run \
   --name registry \
   -e SETTINGS_FLAVOR=prod \
   -e AWS_BUCKET=<your-s3-bucker-name> \
   -e AWS_KEY=<your-aws-key> \
   -e AWS_SECRET=<your-aws-secret> \
   -e DOCKER_REGISTRY_CONFIG=/docker-registry/config/config_sample.yml \
   -e STORAGE_PATH=/ \
   -e SEARCH_BACKEND=sqlalchemy \
   -e GUNICORN_WORKERS=2 \
   -p 5000:5000 \
   -d \
   registry:0.7.3 \
   docker-registry
@ProbablyRusty

This comment has been minimized.

Show comment
Hide comment
@ProbablyRusty

ProbablyRusty Aug 26, 2014

Contributor

I am seeing the same issue. The container launch hangs when specifying AWS_REGION, and works OK when AWS_REGION is not specified. I have seen this issue on every version of docker-registry >0.6.9.

Contributor

ProbablyRusty commented Aug 26, 2014

I am seeing the same issue. The container launch hangs when specifying AWS_REGION, and works OK when AWS_REGION is not specified. I have seen this issue on every version of docker-registry >0.6.9.

@stongo

This comment has been minimized.

Show comment
Hide comment
@stongo

stongo Aug 27, 2014

Having same issue running from python install outside docker, so definitely not related to being inside a docker container as suggested by some

stongo commented Aug 27, 2014

Having same issue running from python install outside docker, so definitely not related to being inside a docker container as suggested by some

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Aug 27, 2014

Member

Then it's a library version problem. Had it running ok outside of a container here. You are likely running Ubuntu 14:04?

Member

dmp42 commented Aug 27, 2014

Then it's a library version problem. Had it running ok outside of a container here. You are likely running Ubuntu 14:04?

@mattheworiordan

This comment has been minimized.

Show comment
Hide comment
@mattheworiordan

mattheworiordan Aug 27, 2014

I tried it outside the container and had the same issue

mattheworiordan commented Aug 27, 2014

I tried it outside the container and had the same issue

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Aug 27, 2014

Member

@mattheworiordan read my comment please

Member

dmp42 commented Aug 27, 2014

@mattheworiordan read my comment please

@mattheworiordan

This comment has been minimized.

Show comment
Hide comment
@mattheworiordan

mattheworiordan Aug 27, 2014

@dmp42 apologies, misread @stongo's comment. I too am on Ubuntu 14.04

mattheworiordan commented Aug 27, 2014

@dmp42 apologies, misread @stongo's comment. I too am on Ubuntu 14.04

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Aug 27, 2014

Member

Same python libraries, different system deps. Pointing at libevent-dev ?

Can anyone do tests with different ubuntu versions?

Member

dmp42 commented Aug 27, 2014

Same python libraries, different system deps. Pointing at libevent-dev ?

Can anyone do tests with different ubuntu versions?

@stongo

This comment has been minimized.

Show comment
Hide comment
@stongo

stongo Aug 27, 2014

using Ubuntu 12.04

stongo commented Aug 27, 2014

using Ubuntu 12.04

@stongo

This comment has been minimized.

Show comment
Hide comment
@stongo

stongo Aug 27, 2014

Actually just compared my error more closely and it seems to be a slightly different error I'm experiencing.

stongo commented Aug 27, 2014

Actually just compared my error more closely and it seems to be a slightly different error I'm experiencing.

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 Sep 5, 2014

Contributor

@dmp42 any official update/workaround on this?

Contributor

shreyu86 commented Sep 5, 2014

@dmp42 any official update/workaround on this?

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Sep 5, 2014

Member

@shreyu86 for now, I would support specifying the region inside the boto.cfg file.

Member

dmp42 commented Sep 5, 2014

@shreyu86 for now, I would support specifying the region inside the boto.cfg file.

@shreyu86

This comment has been minimized.

Show comment
Hide comment
@shreyu86

shreyu86 Sep 6, 2014

Contributor

Thanks!

Shreyas

On Fri, Sep 5, 2014 at 4:32 PM, Olivier Gambier notifications@github.com
wrote:

@shreyu86 https://github.com/shreyu86 for now, I would support
specifying the region inside the boto.cfg file.


Reply to this email directly or view it on GitHub
#400 (comment)
.

Contributor

shreyu86 commented Sep 6, 2014

Thanks!

Shreyas

On Fri, Sep 5, 2014 at 4:32 PM, Olivier Gambier notifications@github.com
wrote:

@shreyu86 https://github.com/shreyu86 for now, I would support
specifying the region inside the boto.cfg file.


Reply to this email directly or view it on GitHub
#400 (comment)
.

@jvimr

This comment has been minimized.

Show comment
Hide comment
@jvimr

jvimr Sep 17, 2014

I've created a set of small dockerfiles that do the workaround (boto.conf && remove aws region from sampe_config.yml ) - https://github.com/jvimr/docker.registry

jvimr commented Sep 17, 2014

I've created a set of small dockerfiles that do the workaround (boto.conf && remove aws region from sampe_config.yml ) - https://github.com/jvimr/docker.registry

This was referenced Oct 7, 2014

@adamlc

This comment has been minimized.

Show comment
Hide comment
@adamlc

adamlc Oct 17, 2014

@jvimr you are a legend! works perfectly :)

adamlc commented Oct 17, 2014

@jvimr you are a legend! works perfectly :)

@chuegle

This comment has been minimized.

Show comment
Hide comment
@chuegle

chuegle Nov 1, 2014

This looks like a gevent bug dealing with unicode.

It's triggered by specifying the region in boto, which pulls the URL from the boto region file which has unicode strings. If you don't specify the region, it uses the default which is not unicode, so therefore it works.

It can be reproduced (in both docker/ubuntu and Mac OS X) by doing the following:

in a.py:
 import b
in b.py:
 import gevent.socket
 print gevent.socket.getaddrinfo(u"s3.amazonaws.com", 443)
python a.py # hangs

If you change the host to a non-unicode, it works.

Oddly, if you put the gevent.socket call in a.py instead of b.py, it works.

Also, if you have it in both a.py and b.py, it works.

If you put a unicode string in b.py and a normal string in a.py, it will hang.

This would be an ugly workaround:

        if self._config.s3_region is not None:
            return boto.s3.connection.S3Connection(
                host=str(boto.regioninfo.load_regions()['s3'][self._config.s3_region]),
                aws_access_key_id=self._config.s3_access_key,
                aws_secret_access_key=self._config.s3_secret_key,
                **kwargs)

or some way if specifying a host=str(host) in the kwargs w/o region.

Side note discovered while investigating:
The gevent.monkey.patch_all() ends up being called after most everything in requirements files are imported. It seems that python entry_point code is nice enough to import the requirements for you before importing the code to patch_all(). It should be less important in gevent 1.0+ that can handle threading, but something to be aware of.

ADDENDUM
A bit more research found a couple other fixes:

  • u'fix gevent'.encode('idna') - Prevent the deadlock by initializing outside of the getaddrinfo call.
  • install simplejson - Boto will use this if it exists, and json if not. simplejson returns its results a str() while json returns it as unicode()

I went into a bit more detail on how it works in boto in:
boto/boto#2179 (comment)

chuegle commented Nov 1, 2014

This looks like a gevent bug dealing with unicode.

It's triggered by specifying the region in boto, which pulls the URL from the boto region file which has unicode strings. If you don't specify the region, it uses the default which is not unicode, so therefore it works.

It can be reproduced (in both docker/ubuntu and Mac OS X) by doing the following:

in a.py:
 import b
in b.py:
 import gevent.socket
 print gevent.socket.getaddrinfo(u"s3.amazonaws.com", 443)
python a.py # hangs

If you change the host to a non-unicode, it works.

Oddly, if you put the gevent.socket call in a.py instead of b.py, it works.

Also, if you have it in both a.py and b.py, it works.

If you put a unicode string in b.py and a normal string in a.py, it will hang.

This would be an ugly workaround:

        if self._config.s3_region is not None:
            return boto.s3.connection.S3Connection(
                host=str(boto.regioninfo.load_regions()['s3'][self._config.s3_region]),
                aws_access_key_id=self._config.s3_access_key,
                aws_secret_access_key=self._config.s3_secret_key,
                **kwargs)

or some way if specifying a host=str(host) in the kwargs w/o region.

Side note discovered while investigating:
The gevent.monkey.patch_all() ends up being called after most everything in requirements files are imported. It seems that python entry_point code is nice enough to import the requirements for you before importing the code to patch_all(). It should be less important in gevent 1.0+ that can handle threading, but something to be aware of.

ADDENDUM
A bit more research found a couple other fixes:

  • u'fix gevent'.encode('idna') - Prevent the deadlock by initializing outside of the getaddrinfo call.
  • install simplejson - Boto will use this if it exists, and json if not. simplejson returns its results a str() while json returns it as unicode()

I went into a bit more detail on how it works in boto in:
boto/boto#2179 (comment)

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Nov 3, 2014

Member

@chuegle !!!!

Awesome research.

Is there an upstream (gevent) ticket for this? Otherwise, do you have an ETA for this to hit a boto release?

Member

dmp42 commented Nov 3, 2014

@chuegle !!!!

Awesome research.

Is there an upstream (gevent) ticket for this? Otherwise, do you have an ETA for this to hit a boto release?

@chuegle

This comment has been minimized.

Show comment
Hide comment
@chuegle

chuegle Nov 3, 2014

The gevent ticket (although there may be more than one):
gevent/gevent#349

As far as a boto eta, I haven't submitted a patch and haven't heard back on my comments there, so I can't really say. I'll see if I have time this week to implement one of the fixes for boto and do a pull request, but would still have no idea how long it is until the next rev is pushed or if this change would make it in.

chuegle commented Nov 3, 2014

The gevent ticket (although there may be more than one):
gevent/gevent#349

As far as a boto eta, I haven't submitted a patch and haven't heard back on my comments there, so I can't really say. I'll see if I have time this week to implement one of the fixes for boto and do a pull request, but would still have no idea how long it is until the next rev is pushed or if this change would make it in.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Nov 3, 2014

Member

Fixed thanks to @chuegle incredible work!

Member

dmp42 commented Nov 3, 2014

Fixed thanks to @chuegle incredible work!

@dmp42 dmp42 closed this Nov 3, 2014

@hartym

This comment has been minimized.

Show comment
Hide comment
@hartym

hartym Nov 18, 2014

Any idea when / version in which this will be made available ?

hartym commented Nov 18, 2014

Any idea when / version in which this will be made available ?

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Nov 18, 2014

Member

@hartym was released with 0.9 IIRC.
Are you still experiencing this issue?

Member

dmp42 commented Nov 18, 2014

@hartym was released with 0.9 IIRC.
Are you still experiencing this issue?

@hartym

This comment has been minimized.

Show comment
Hide comment
@hartym

hartym Nov 20, 2014

@dmp42 I could not start a registry container using 0.9/latest tag, backed on S3. The container is working fine with local filesystem backend, but S3 connection never worked. I could connect to the same bucket with the same aws keys using raw boto though. At best, the container was hanging after AWS ...somecrypticstring... line.

For now, I did put aside the setup of my own registry, I will dive more into it early next year. Not sure my problem is related to this ticket, but all pointers I found around were heading me here.

hartym commented Nov 20, 2014

@dmp42 I could not start a registry container using 0.9/latest tag, backed on S3. The container is working fine with local filesystem backend, but S3 connection never worked. I could connect to the same bucket with the same aws keys using raw boto though. At best, the container was hanging after AWS ...somecrypticstring... line.

For now, I did put aside the setup of my own registry, I will dive more into it early next year. Not sure my problem is related to this ticket, but all pointers I found around were heading me here.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Nov 20, 2014

Member

@hartym if you delay your deployment to next year, fair enough - if you still have time to test, can you copy your configuration, launch command, and log?

Thanks a lot.

Member

dmp42 commented Nov 20, 2014

@hartym if you delay your deployment to next year, fair enough - if you still have time to test, can you copy your configuration, launch command, and log?

Thanks a lot.

@hartym

This comment has been minimized.

Show comment
Hide comment
@hartym

hartym Nov 21, 2014

@dmp42 seems like the "latest" image was not the same as 0.9.0, and after running brand new tests, I achieved to run the registry (with S3 storage on europe region and sqlalchemy search backend). Thanks for your help and sorry for bothering.

hartym commented Nov 21, 2014

@dmp42 seems like the "latest" image was not the same as 0.9.0, and after running brand new tests, I achieved to run the registry (with S3 storage on europe region and sqlalchemy search backend). Thanks for your help and sorry for bothering.

@dmp42

This comment has been minimized.

Show comment
Hide comment
@dmp42

dmp42 Nov 21, 2014

Member

@hartym happy you got it working!

Member

dmp42 commented Nov 21, 2014

@hartym happy you got it working!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment