Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

STATUSCHECK_IMAGE_PULL_ERR for an image that was pushed to repo in build phase #4851

Closed
nelango opened this issue Oct 3, 2020 · 7 comments
Closed

STATUSCHECK_IMAGE_PULL_ERR for an image that was pushed to repo in build phase #4851

nelango opened this issue Oct 3, 2020 · 7 comments

Comments

@nelango
Copy link

nelango commented Oct 3, 2020

In my skaffold dev session, build phase succeed and images were pushed to the repo, but deployment failed with STATUSCHECK_IMAGE_PULL_ERR.

Expected behavior

Images should pulled successfully from the repo as they were just pushed to it in the build phase.

Actual behavior

Deployment phase with STATUSCHECK_IMAGE_PULL_ERR.
Logs (summary):

Running "Run on Kubernetes" configuration (.vscode/launch.json) in Run mode

To view more detailed logs, go to Output channel : "Run on Kubernetes - Detailed"

Update initiated due to file change
Build started for artifact go-hello-world
Build completed for artifact go-hello-world

Deploy started
Deploy complete

Status check started
Resource pod/go-hello-world-554bbbbc57-csf5j status failed with container server is waiting to start: gcr.io/cloud-code-285423/go-hello-world:latest@sha256:9cbb767c622d87f5fef073c11101e8c5838f82d93d3e892c58d234129b08de60 can't be pulled
Resource deployment/go-hello-world status failed with waiting for rollout to finish: 0 of 1 updated replicas are available...

Status check failed

Update failed with error code STATUSCHECK_IMAGE_PULL_ERR
exiting dev mode because first deploy failed: 1/1 deployment(s) failed
Exited with code 1.

Information

  • Skaffold version: v1.14.0
  • Operating system: MacOS
  • Contents of skaffold.yaml:
apiVersion: skaffold/v2beta4
kind: Config
build:
  tagPolicy:
    sha256: {}
  artifacts:
  - context: .
    image: go-hello-world
deploy:
  kubectl:
    manifests:
    - kubernetes-manifests/**
profiles:
- name: cloudbuild
  build:
    googleCloudBuild: {}

Steps to reproduce the behavior

Run Cloud Code sample K8 app - Go: Hello World on vscode - https://github.com/GoogleCloudPlatform/cloud-code-samples/tree/master/golang/go-hello-world
@nelango
Copy link
Author

nelango commented Oct 3, 2020 •
edited

Details logs:

Running: skaffold dev -v info --port-forward --rpc-http-port 60689 --filename /Users/nelango/Source/Repos/sample-extensions/hello-world-19/skaffold.yaml --default-repo gcr.io/cloud-code-285423
starting gRPC server on port 50051
starting gRPC HTTP server on port 60689
Skaffold &{Version:v1.14.0 ConfigVersion:skaffold/v2beta7 GitVersion: GitCommit:4b3ca59af505c4ab5d5b6960a194e1f6887018f8 GitTreeState:clean BuildDate:2020-09-02T19:40:37Z GoVersion:go1.14.2 Compiler:gc Platform:darwin/amd64}
Using kubectl context: gke_nelango-exp-0710_us-east1-b_test-cluster-2
Loaded Skaffold defaults from \"/Users/nelango/.skaffold/config\"
Listing files to watch...
 - go-hello-world
List generated in 1.147156108s
Generating tags...
 - go-hello-world -> gcr.io/cloud-code-285423/go-hello-world:latest
Checking cache...
Tags generated in 1.318985ms
 - go-hello-world: Not found. Building
Building [go-hello-world]...
Cache check complete in 4.636022ms
Sending build context to Docker daemon  56.83kB

Step 1/8 : FROM golang:1.15
1.15: Pulling from library/golang
57df1a1f1ad8: Already exists
71e126169501: Already exists
1af28a55c3f3: Already exists
03f1c9932170: Already exists
f4773b341423: Pulling fs layer
fb320882041b: Pulling fs layer
24b0ad6f9416: Pulling fs layer
24b0ad6f9416: Verifying Checksum
24b0ad6f9416: Download complete
f4773b341423: Verifying Checksum
f4773b341423: Download complete
fb320882041b: Verifying Checksum
fb320882041b: Download complete
f4773b341423: Pull complete
fb320882041b: Pull complete
24b0ad6f9416: Pull complete
Digest: sha256:da7ff43658854148b401f24075c0aa390e3b52187ab67cab0043f2b15e754a68
Status: Downloaded newer image for golang:1.15
 ---> 05c8f6d2538a
Step 2/8 : RUN go get -u -v github.com/go-delve/delve/cmd/dlv
 ---> Running in 935707d709d6
�[91mgithub.com/go-delve/delve (download)
�[0m�[91mgithub.com/go-delve/delve/vendor/golang.org/x/sys/internal/unsafeheader
�[0m�[91mgithub.com/go-delve/delve/vendor/github.com/hashicorp/golang-lru/simplelru
�[0m�[91mgithub.com/go-delve/delve/vendor/go.starlark.net/internal/spell
�[0m�[91mgithub.com/go-delve/delve/vendor/github.com/mattn/go-isatty
�[0m�[91mgithub.com/go-delve/delve/vendor/golang.org/x/sys/unix
�[0m�[91mgithub.com/go-delve/delve/pkg/goversion
�[0m�[91mgithub.com/go-delve/delve/vendor/gopkg.in/yaml.v2
�[0m�[91mgithub.com/go-delve/delve/vendor/github.com/cosiner/argv
�[0m�[91mgithub.com/go-delve/delve/pkg/astutil
�[0m�[91mgithub.com/go-delve/delve/pkg/dwarf/util
�[0m�[91mgithub.com/go-delve/delve/vendor/golang.org/x/arch/arm64/arm64asm
�[0m�[91mgithub.com/go-delve/delve/vendor/golang.org/x/arch/x86/x86asm
�[0m�[91mgithub.com/go-delve/delve/pkg/dwarf/frame
�[0m�[91mgithub.com/go-delve/delve/pkg/dwarf/op
�[0m�[91mgithub.com/go-delve/delve/pkg/dwarf/godwarf
�[0m�[91mgithub.com/go-delve/delve/pkg/dwarf/line
�[0m�[91mgithub.com/go-delve/delve/vendor/github.com/peterh/liner
�[0m�[91mgithub.com/go-delve/delve/pkg/dwarf/loclist
�[0m�[91mgithub.com/go-delve/delve/pkg/dwarf/reader
�[0m�[91mgithub.com/go-delve/delve/vendor/go.starlark.net/syntax
�[0m�[91mgithub.com/go-delve/delve/pkg/version
�[0m�[91mgithub.com/go-delve/delve/pkg/config
�[0m�[91mgithub.com/go-delve/delve/vendor/github.com/google/go-dap
�[0m�[91mgithub.com/go-delve/delve/vendor/github.com/spf13/pflag
�[0m�[91mgithub.com/go-delve/delve/vendor/github.com/russross/blackfriday
�[0m�[91mgithub.com/go-delve/delve/pkg/gobuild
�[0m�[91mgithub.com/go-delve/delve/vendor/github.com/sirupsen/logrus
�[0m�[91mgithub.com/go-delve/delve/vendor/go.starlark.net/resolve
�[0m�[91mgithub.com/go-delve/delve/vendor/github.com/cpuguy83/go-md2man/md2man
�[0m�[91mgithub.com/go-delve/delve/pkg/logflags
�[0m�[91mgithub.com/go-delve/delve/vendor/go.starlark.net/internal/compile
�[0m�[91mgithub.com/go-delve/delve/vendor/github.com/spf13/cobra
�[0m�[91mgithub.com/go-delve/delve/pkg/proc
�[0m�[91mgithub.com/go-delve/delve/vendor/go.starlark.net/starlark
�[0m�[91mgithub.com/go-delve/delve/vendor/github.com/spf13/cobra/doc
�[0m�[91mgithub.com/go-delve/delve/pkg/proc/winutil
github.com/go-delve/delve/pkg/proc/linutil
�[0m�[91mgithub.com/go-delve/delve/service/api
�[0m�[91mgithub.com/go-delve/delve/pkg/proc/core/minidump
�[0m�[91mgithub.com/go-delve/delve/pkg/proc/gdbserial
github.com/go-delve/delve/pkg/proc/native
�[0m�[91mgithub.com/go-delve/delve/pkg/proc/core
�[0m�[91mgithub.com/go-delve/delve/pkg/locspec
�[0m�[91mgithub.com/go-delve/delve/service/debugger
�[0m�[91mgithub.com/go-delve/delve/service
�[0m�[91mgithub.com/go-delve/delve/service/rpc2
github.com/go-delve/delve/service/rpc1
�[0m�[91mgithub.com/go-delve/delve/service/dap
�[0m�[91mgithub.com/go-delve/delve/service/rpccommon
�[0m�[91mgithub.com/go-delve/delve/pkg/terminal/starbind
�[0m�[91mgithub.com/go-delve/delve/pkg/terminal
�[0m�[91mgithub.com/go-delve/delve/cmd/dlv/cmds
�[0m�[91mgithub.com/go-delve/delve/cmd/dlv
�[0m ---> f74ceaeffe3e
Step 3/8 : WORKDIR /src/hello-world
 ---> Running in 348cf03ef571
 ---> c8543b9a5728
Step 4/8 : COPY go.mod go.sum ./
 ---> bdd9863572a3
Step 5/8 : RUN go mod download
 ---> Running in 69016bea60c8
 ---> 32be67b588eb
Step 6/8 : COPY . ./
 ---> 9e01ba0ba024
Step 7/8 : RUN go build -o /app -v ./cmd/hello-world
 ---> Running in 4da965a33104
�[91mhello-world/cmd/hello-world
�[0m ---> 9d02b6294453
Step 8/8 : ENTRYPOINT ["dlv", "exec", "/app", "--continue", "--accept-multiclient", "--api-version=2", "--headless", "--listen=:3000", "--log"]
 ---> Running in 3a3b3bcf8ed9
 ---> 98fc99e31230
Successfully built 98fc99e31230
Successfully tagged gcr.io/cloud-code-285423/go-hello-world:latest
The push refers to repository [gcr.io/cloud-code-285423/go-hello-world]
bd0e48b55a56: Preparing
97a96efd53bc: Preparing
fc22fcdb9f0b: Preparing
d73b25bca51b: Preparing
3ead95e6aebf: Preparing
0d870262aa71: Preparing
1ba6a3ca6c41: Preparing
d2588c27d938: Preparing
6b6e43b44148: Preparing
17bdf5e22660: Preparing
d37096232ed8: Preparing
6add0d2b5482: Preparing
4ef54afed780: Preparing
1ba6a3ca6c41: Waiting
d2588c27d938: Waiting
6b6e43b44148: Waiting
17bdf5e22660: Waiting
d37096232ed8: Waiting
6add0d2b5482: Waiting
4ef54afed780: Waiting
0d870262aa71: Waiting
3ead95e6aebf: Pushed
d73b25bca51b: Pushed
fc22fcdb9f0b: Pushed
97a96efd53bc: Pushed
1ba6a3ca6c41: Layer already exists
d2588c27d938: Layer already exists
6b6e43b44148: Layer already exists
17bdf5e22660: Layer already exists
d37096232ed8: Layer already exists
bd0e48b55a56: Pushed
6add0d2b5482: Layer already exists
4ef54afed780: Layer already exists
0d870262aa71: Pushed
latest: digest: sha256:9cbb767c622d87f5fef073c11101e8c5838f82d93d3e892c58d234129b08de60 size: 3047
Build complete in 1m9.708025519s
Test complete in 872ns
Tags used in deployment:
 - go-hello-world -> gcr.io/cloud-code-285423/go-hello-world:latest@sha256:9cbb767c622d87f5fef073c11101e8c5838f82d93d3e892c58d234129b08de60
Starting deploy...
 - deployment.apps/go-hello-world created
 - service/go-hello-world-external created
Deploy complete in 2.097508281s
Waiting for deployments to stabilize...
 - deployment/go-hello-world: container server is waiting to start: gcr.io/cloud-code-285423/go-hello-world:latest@sha256:9cbb767c622d87f5fef073c11101e8c5838f82d93d3e892c58d234129b08de60 can't be pulled
    - pod/go-hello-world-554bbbbc57-csf5j: container server is waiting to start: gcr.io/cloud-code-285423/go-hello-world:latest@sha256:9cbb767c622d87f5fef073c11101e8c5838f82d93d3e892c58d234129b08de60 can't be pulled
 - deployment/go-hello-world failed. Error: container server is waiting to start: gcr.io/cloud-code-285423/go-hello-world:latest@sha256:9cbb767c622d87f5fef073c11101e8c5838f82d93d3e892c58d234129b08de60 can't be pulled.
Cleaning up...
 - deployment.apps "go-hello-world" deleted
 - service "go-hello-world-external" deleted
Cleanup complete in 13.267826484s
exiting dev mode because first deploy failed: 1/1 deployment(s) failed
Exited with code 1.

@nelango
Copy link
Author

nelango commented Oct 5, 2020

@tejal29 looks like I might not have permission to assign. But this is the issue where your hypothesis was that registry project id and cluster project id are different. I can confirm that the hypothesis is correct.

@briandealwis
Copy link
Member

This is the same as #4132 — you need to authorize your cluster to pull from the other project by creating an ImagePullSecret.

Duplicate of #4132

@mycar98765
Copy link

Hi @nelango , @briandealwis , how was this issue resolved finally? I faced the same issue as per @nelango 's first post, but I guess not much on authorization issue as #4132.

Status check started
Resource pod/id status updated to In Progress
Resource deployment/project status updated to In Progress
Resource pod/id status updated to In Progress
Resource deployment/project status failed with waiting for rollout to finish: 0 of 1 updated replicas are available...
Status check failed

Update failed with error code STATUSCHECK_IMAGE_PULL_ERR
1/1 deployment(s) failed
Skaffold exited with code 1.
Cleaning up...
Finished clean up.

@briandealwis
Copy link
Member

@mycar98765 the GKE troubleshooting docs are really helpful:

https://cloud.google.com/kubernetes-engine/docs/troubleshooting#permission_denied_error

@mycar98765
Copy link

Hi @briandealwis, I followed the link you shared and did the following:

  1. I verified and confirmed the existence of the image under Buckets > artifacts.project-id.appspot.com > containers > images in Cloud Storage
  2. Back to your shared link, under the Container Registry tab, I replaced SERVICE_ACCOUNT_EMAIL and BUCKET_NAME with project-number@cloudbuild.gserviceaccount.com and artifacts.project-id.appspot.com
  3. I ran the above gsutil iam command in the terminal. In fact, prior to running the command, project-number@cloudbuild.gserviceaccount.com is already granted with roles/storage.objectViewer role
  4. I debugged it on the Cloud Run emulator.

Same error persists:

Status check started
Resource pod/id status updated to In Progress
Resource deployment/project status updated to In Progress
Resource pod/id status updated to In Progress
Resource deployment/project status failed with waiting for rollout to finish: 0 of 1 updated replicas are available...
Status check failed

Update failed with error code STATUSCHECK_IMAGE_PULL_ERR
1/1 deployment(s) failed
Skaffold exited with code 1.
Cleaning up...
Finished clean up.

@briandealwis
Copy link
Member

@mycar98765 I’m not sure how you are encountering this situation with the Cloud Run emulator as that runs in minikube and the image should be local and never pulled.

But let’s take this discussion to GoogleCloudPlatform/cloud-code-vscode#659

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@briandealwis @nelango @mycar98765