Auth Timeout #510

Open
owenhaynes opened this Issue Oct 28, 2015 · 14 comments

Projects

None yet

5 participants

@owenhaynes

When pushing big layers timeout of the authentication happens. eg layers over 1GB insize

@flavio
Member
flavio commented Oct 28, 2015

Can you provide more context? This should be an error of the registry, not of Portus.

@GrantStreetGroup

Portus sets the length of time that the token is considered valid.
See https://docs.docker.com/registry/spec/auth/token/

Maybe the push is going past that point?

@mssola
Contributor
mssola commented Oct 28, 2015

@GrantStreetGroup is right. The period of time in which the authentication token is valid spans for 5 minutes. @mad0house is the upload taking that long ?

@owenhaynes

Yes push can take that long as some times we have people pushing over WiFi
which can get congested at times, some layers of the image can range from
1gb to 1.9gb and docker v1.8 takes ages to buffer and then the server takes
a while as well.

Not had any pulls timeout yet but I assume that can happen as well
On 28 Oct 2015 3:16 pm, "Miquel Sabaté Solà" notifications@github.com
wrote:

@GrantStreetGroup https://github.com/GrantStreetGroup is right. The
period of time in which the authentication token is valid spans for 5
minutes. @mad0house https://github.com/mad0house is the upload taking
that long ?


Reply to this email directly or view it on GitHub
#510 (comment).

@mssola
Contributor
mssola commented Oct 28, 2015

@mad0house so, just for testing purposes, could you try to raise this timeout to something like 30 minutes and see whether that works for you ? You can do that by changing this line from 5.minutes to 30.minutes. Then, of course, restart services, etc.

@owenhaynes

Will test it for you tomorrow morning(GMT), If this does work can we make it a configuration item?
On 28 Oct 2015 5:47 pm, "Miquel Sabaté Solà" notifications@github.com
wrote:

@mad0house https://github.com/mad0house so, just for testing purposes,
could you try to raise this timeout to something like 30 minutes and see
whether that works for you ? You can do that by changing this line
https://github.com/SUSE/Portus/blob/master/lib/portus/jwt_token.rb#L33
from 5.minutes to 30.minutes.


Reply to this email directly or view it on GitHub
#510 (comment).

@mssola
Contributor
mssola commented Oct 28, 2015

@mad0house yes, sounds reasonable.

@mssola mssola added the needs info label Oct 28, 2015
@owenhaynes

change has been made will get back to you later on today to give it time to test.

@kidhasmoxy

I have the same issue, testing now.

Here's what I get from the registry logs for layers ~ 1GB:

172.17.46.121 - - [29/Oct/2015:13:35:53 +0000] "GET /v2/_catalog HTTP/1.0" 200 20 "" "Ruby"
time="2015-10-29T13:37:41Z" level=error msg="token not to be used before 1446125032 or after 1446125337 - currently 1446125861"
time="2015-10-29T13:37:41Z" level=warning msg="error authorizing context: invalid token" http.request.host="registry.demosnc.com:5000" http.request.id=7a72895f-3a71-487e-af81-8f9af5a9907c http.request.method=PATCH http.request.remoteaddr=68.228.156.33 http.request.uri="/v2/release/geneva/blobs/uploads/5e4174b7-9f34-49ab-a6b6-3b0c8c96c28e?_state=RRNl2YXESRKrLWHT3e4RcMyYH5u0tiN50cBbztE_GLd7Ik5hbWUiOiJyZWxlYXNlL2dlbmV2YSIsIlVVSUQiOiI1ZTQxNzRiNy05ZjM0LTQ5YWItYTZiNi0zYjBjOGM5NmMyOGUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTUtMTAtMjlUMTM6MjM6NTguMTc3MTg3NzIxWiJ9" http.request.useragent="docker/1.8.3 go/go1.4.2 git-commit/f4bf5c7 kernel/4.1.10-boot2docker os/linux arch/amd64" instance.id=34d12fcb-9cd8-4e0a-b706-564777af862f vars.name="release/geneva" vars.uuid=5e4174b7-9f34-49ab-a6b6-3b0c8c96c28e version=v2.1.1
172.17.46.121 - - [29/Oct/2015:13:37:41 +0000] "PATCH /v2/release/geneva/blobs/uploads/5e4174b7-9f34-49ab-a6b6-3b0c8c96c28e?_state=RRNl2YXESRKrLWHT3e4RcMyYH5u0tiN50cBbztE_GLd7Ik5hbWUiOiJyZWxlYXNlL2dlbmV2YSIsIlVVSUQiOiI1ZTQxNzRiNy05ZjM0LTQ5YWItYTZiNi0zYjBjOGM5NmMyOGUiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMTUtMTAtMjlUMTM6MjM6NTguMTc3MTg3NzIxWiJ9 HTTP/1.0" 401 235 "" "docker/1.8.3 go/go1.4.2 git-commit/f4bf5c7 kernel/4.1.10-boot2docker os/linux arch/amd64"

@owenhaynes

Not had any problems so far after the change.

@GrantStreetGroup

This sounds like an bug with 'docker push' If a layer is taking longer then the token timeout to upload then the command should re-authorise and get a new token.

@mssola
Contributor
mssola commented Oct 29, 2015

@GrantStreetGroup agreed.

I'm going to take a deeper look into this as soon as possible. For now, we can just provide a configurable option so we can work around this problem.

@mssola mssola added enhancement and removed needs info labels Oct 29, 2015
@mssola mssola self-assigned this Oct 29, 2015
@mssola mssola added a commit to mssola/Portus that referenced this issue Oct 29, 2015
@mssola mssola config: added a configurable option for the expiration time of a JWT …
…token

See the isse #510

Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>
2f08353
@kidhasmoxy

Increasing the token delay worked, but it's more of a workaround on the Portus side for the fact that the docker client isn't retrying the token request when it finishes the upload as @GrantStreetGroup mentioned.

Either way, better to have a config option for it than to wait for a docker client update.

@mssola
Contributor
mssola commented Oct 30, 2015

@kidhasmoxy yes, I totally agree. That's why in my PR, you can see the note on the configuration option saying that it will be deprecated in the future once this is fixed upstream ;)

@mssola mssola added a commit to mssola/Portus that referenced this issue Oct 30, 2015
@mssola mssola config: added a configurable option for the expiration time of a JWT …
…token

See the isse #510

Signed-off-by: Miquel Sabaté Solà <msabate@suse.com>
6fdeaa5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment