Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does caching proxy work on unavailable upstream? #3705

Open
Felixoid opened this issue Aug 2, 2022 · 5 comments
Open

Does caching proxy work on unavailable upstream? #3705

Felixoid opened this issue Aug 2, 2022 · 5 comments

Comments

@Felixoid
Copy link

Felixoid commented Aug 2, 2022

Hello. According to manual, the registry mirror requests if the images updated on the ustream.

We use docker very intensively in our CI, and quite often affected by issues like this:

Exception: Command ['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_cleanup_dir_after_bad_zk_conn/_instances_0/.env', '--project-name', 'roottestcleanupdirafterbadzkconn', '--file', '/ClickHouse/tests/integration/test_cleanup_dir_after_bad_zk_conn/_instances_0/node1/docker-compose.yml', '--file', '/compose/docker_compose_keeper.yml', 'pull'] return non-zero code 1: Pulling zoo2  ... 
Pulling zoo3  ... 
Pulling zoo1  ... 
Pulling node1 ... 
Pulling zoo3  ... error
Pulling zoo1  ... error
Pulling zoo2  ... error
Pulling node1 ... error

ERROR: for zoo3  Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

ERROR: for zoo1  Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

ERROR: for zoo2  Get "https://registry-1.docker.io/v2/": context deadline exceeded

ERROR: for node1  Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Get "https://registry-1.docker.io/v2/": context deadline exceeded
Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

And then a huge number of jobs fail.

Do I understand correctly, that registry:2 (the current latest) can't be setup to serve requests with what it has w/o failing?

p.s. It's a moved issue from distribution/distribution-library-image#141

@Felixoid
Copy link
Author

The issue affects us pretty badly. We have a vast testing infrastructure, so the whole thing fails as a card house when there's a connection issue between proxy and docker hub. Here another case ClickHouse/ClickHouse#40241 (comment)

@alexey-milovidov
Copy link

This issue affects us a few times a day.

@milosgajdos
Copy link
Member

Registry proxy has not seen much TLC in a while, unfortunately, but we are open to accepting PRs.

@Felixoid
Copy link
Author

Felixoid commented Aug 17, 2022

Can you point to a code where I could start my research, please?

As well, I've tried to find what TLC means, but it's still a bit unclear

@Jamstah
Copy link
Collaborator

Jamstah commented Aug 17, 2022

TLC=Tender Love and Care, it basically means no one has put any work into making it better for a while.

The code for a proxying registry is here: https://github.com/distribution/distribution/tree/main/registry/proxy

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants