Cosign 2.2.1 Docker image missing /bin/sh

[Fredouye]

Description

Cosign 2.2.1 does not include the sh executable, and fails to run through a GitLab CI pipeline.

Extract of .gitlab-ci.yml :

sign-image:
  stage: push
  variables:
    SSL_CERT_FILE: ${VAULT_CACERT}
  image:
    name: gcr.io/projectsigstore/cosign:v2.2.1
    entrypoint: [""]
  script:
    - cosign login ${REGISTRY_URL} --username ${REGISTRY_USERNAME} --password ${REGISTRY_PASSWORD}
    - echo $COSIGN_PRIVATE_KEY | base64 -d > /tmp/cosign_private_key
    - cosign sign --key /tmp/cosign_private_key "${REGISTRY_URL}/${REGISTRY_PROJECT}/${IMAGE}:${VERSION}" --yes
    - rm /tmp/cosign_private_key -f

Result :

Using docker image sha256:aadc646735691b2cec958e1930c2b815f66f6b4aec58d4dc88e1211dd463083b for gcr.io/projectsigstore/cosign:v2.2.1 with digest gcr.io/projectsigstore/cosign@sha256:88498ed17e61605cd68a5fc9d1fcd756ae0ef2d5515417881d739654accf818f ...
Cleaning up project directory and file based variables 00:02
ERROR: Job failed (system failure): Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown (exec.go:78:2s)

Downgrading to Cosign 2.2.0 fixes the issue.

By the way, Cosign 2.2.1 Docker image is much smaller than 2.2.0 (107 Mb instead of 124 Mb).

Version

Cosign 2.2.1 Docker image (gcr.io/projectsigstore/cosign:v2.2.1).

[haydentherapper]

This was intentional, we switched to a newer distroless base image without a shell. I’m open to adding it back, but I believe this could be refactored to not need a shell.

Thoughts @cpanato @hectorj2f?

[Fredouye]

It seems a shell is needed to run Cosign from a GitLab CI pipeline.

For the same reason, I use the debug image for Kaniko :

The kaniko executor image is based on scratch and doesn't contain a shell. We provide gcr.io/kaniko-project/executor:debug, a debug image which consists of the kaniko executor image along with a busybox shell to enter.

[hectorj2f]

@Fredouye You could use cgr.dev/chainguard/cosign:latest-dev which includes a shell. Obviously I'd advocate to use gcr.io/projectsigstore/cosign:v2.2.1 but you can use the latest-dev image tag which is an explicit extension of the original image which includes `git, bash, and other packages.

[haydentherapper]

@Fredouye I'll take a look at creating a dev or debug image. I'm not familiar with GitLab CI, but from the example you've given, it seems like a shell is necessary. Maybe there's a way to do a multi-step process where the key and registry credentials are obtained and passed to Cosign invocation step?

[hectorj2f]

Yes, @haydentherapper is right! You should be able to run it in a multi-step process as an alternative solution.

Also there is a typo in your script cosign login.

[haydentherapper]

@imjasonh, is it possible to create two containers using ko where all configuration but the base images are different, while also controlling the name of the container (one to be cosign, one cosign-debug)?

Concretely, I would like the following config to work, but I don't think it does, because I can't specify the same main path twice, I think.

defaultBaseImage: gcr.io/distroless/static-debian12:nonroot
baseImageOverrides:
   <path-to-cosign-debug>: gcr.io/distroless/static-debian12:debug-nonroot

builds:
- id: cosign
  dir: .
  main: ./cmd/cosign
  env:
  - CGO_ENABLED=0
  flags:
  - -trimpath
  - -tags
  - "{{ .Env.GIT_HASH }}"
  - -tags
  - "{{ .Env.GIT_VERSION }}"
  ldflags:
  - -extldflags "-static"
  - "{{ .Env.LDFLAGS }}"

builds:
- id: cosign-debug
  dir: .
  main: ./cmd/cosign
  env:
  - CGO_ENABLED=0
  flags:
  - -trimpath
  - -tags
  - "{{ .Env.GIT_HASH }}"
  - -tags
  - "{{ .Env.GIT_VERSION }}"
  ldflags:
  - -extldflags "-static"
  - "{{ .Env.LDFLAGS }}"

[hectorj2f]

@haydentherapper We could also revert the baseImage to this one #3250.

[Fredouye]

@Fredouye Maybe there's a way to do a multi-step process where the key and registry credentials are obtained and passed to Cosign invocation step?

This is already the case in my GitLab CI pipeline (which is using Vault client, Hadolint, Gitleaks, Kaniko, Trivy, Crane and Cosign as a last step).

Using cgr.dev/chainguard/cosign:latest-dev should be fine, but I like using "fixed" tags (which are then managed by the Renovate tool).

[Fredouye]

@haydentherapper We could also revert the baseImage to this one #3250.

Maybe moving from gcr.io/distroless/base-debian12:nonroot (1.99 MB) to gcr.io/distroless/static-debian12:debug-nonroot (3.15 MB, which contains busybox) would be a viable solution ?

[loosebazooka]

Sorry I'm not super familiar here. Do all gitlab CI steps run off a shell directly if there any script directives?

[haydentherapper]

I'd ideally like to release two images, though I'm not familiar with ko and if it's possible to do so. Otherwise, yea, we could switch to gcr.io/distroless/static-debian12:debug-nonroot.

[haydentherapper]

@imjasonh @cpanato Bumping this, do you know if ko can generate two images based on the same main and control its name (so we can have cosign and cosign-dev)

[cpanato]

it cannot generate two images but we can build as the release pipeline two images :) I will work on that

[VenutNSA]

Hello!
Seconded:)
When can we expect a downloadable image with postfix *-dev?)

[haydentherapper]

It'll be once we cut a new release. Let's target end of week to give us a chance to merge open PRs.

[Fredouye]

Thanks for the 2.2.2 release, gcr.io/projectsigstore/cosign:v2.2.2-dev works fine in my GitLab CI pipeline.