New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Addons cannot be installed on v0.20.0 #243

Closed
giner opened this Issue Aug 10, 2018 · 5 comments

Comments

Projects
None yet
3 participants
@giner

giner commented Aug 10, 2018

What happened:
Starting from release 0.20.0 addons cannot be installed

What you expected to happen:
bosh -d cfcr run-errand apply-specs should install addons

How to reproduce it (as minimally and precisely as possible):
Run bosh -d cfcr run-errand apply-specs on fresh deploy.

Anything else we need to know?:
Some log extracts:

Instance   apply-addons/fe1ac694-d99d-499a-801c-33eb64d3b8b7  
Exit Code  1  
Stdout     Deploying /var/vcap/jobs/apply-specs/specs/kube-dns.yml  
           service/kube-dns created  
           serviceaccount/kube-dns created  
           configmap/kube-dns-auth created  
           configmap/kube-dns created  
           deployment.extensions/kube-dns created  
           Waiting for deployment "kube-dns" rollout to finish: 0 of 1 updated replicas are available...  
           failed to start all system specs after 1200 with exit code 1  
             
Stderr     error: deployment "kube-dns" exceeded its progress deadline  
             

1 errand(s)

Errand 'apply-specs' completed with error (exit code 1)

Exit code 1

It is trying to download image for "pause:3.1" from Internet instead of using preloaded one.

  Warning  FailedCreatePodSandBox  4m (x128 over 1h)  kubelet, 100.10.1.13  Failed create pod sandbox: rpc error: code = Unknown desc = failed pulling image "k8s.gcr.io/pause:3.1": Error response from daemon: Get https://k8s.gcr.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Workaround
Run this on all workers:

docker tag k8s.gcr.io/pause-amd64:3.1 k8s.gcr.io/pause:3.1

Why is this happening?
Because of this kubernetes/kubernetes#65920

How to fix?
Probably by stripping -amd64 here https://github.com/cloudfoundry-incubator/kubo-release/blob/fbe1d04/scripts/download_container_images#L11. May be there is a better way.

@cf-gitbot

This comment has been minimized.

Show comment
Hide comment
@cf-gitbot

cf-gitbot Aug 10, 2018

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/159706265

The labels on this github issue will be updated when the story is started.

cf-gitbot commented Aug 10, 2018

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/159706265

The labels on this github issue will be updated when the story is started.

@alex-slynko

This comment has been minimized.

Show comment
Hide comment
@alex-slynko

alex-slynko Aug 14, 2018

Member

Hi @giner

Thank you for reporting this issue. We have implemented the fix and expect to release it as part of CFCR 0.21

Member

alex-slynko commented Aug 14, 2018

Hi @giner

Thank you for reporting this issue. We have implemented the fix and expect to release it as part of CFCR 0.21

@giner

This comment has been minimized.

Show comment
Hide comment
@giner

giner Aug 23, 2018

@alex-slynko, any hints on when 0.21 is going to be cut?

giner commented Aug 23, 2018

@alex-slynko, any hints on when 0.21 is going to be cut?

@alex-slynko

This comment has been minimized.

Show comment
Hide comment
@alex-slynko

alex-slynko Aug 23, 2018

Member

@giner
We are trying to solve a very annoying issue that breaks upgrades in AWS. We will release as soon as we finish it.

You can check the progress in our tracker https://www.pivotaltracker.com/n/projects/2093412 in future

Member

alex-slynko commented Aug 23, 2018

@giner
We are trying to solve a very annoying issue that breaks upgrades in AWS. We will release as soon as we finish it.

You can check the progress in our tracker https://www.pivotaltracker.com/n/projects/2093412 in future

@alex-slynko

This comment has been minimized.

Show comment
Hide comment
@alex-slynko

alex-slynko Aug 30, 2018

Member

Hi @giner
We have released 0.21 that fixes that issue. Additionally, we have improved our tests, so we would see the issues like this earlier.
Thank you again for using CFCR and reporting this issue!

Member

alex-slynko commented Aug 30, 2018

Hi @giner
We have released 0.21 that fixes that issue. Additionally, we have improved our tests, so we would see the issues like this earlier.
Thank you again for using CFCR and reporting this issue!

@cf-gitbot cf-gitbot removed the accepted label Aug 30, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment