Add rekor support for cosign

[xopham]

Describe the feature
With #107, basic Cosign support was added. In the spirit of extending that support and strengthening the signature verification, we could extend this to also support the rekor integration of cosign: https://github.com/sigstore/cosign/tree/v0.2.0#rekor-support

Optional: Implementation ideas
We would probably have to extend the cosign configuration in the helm/values.yaml and might reuse host key. However, integration should not be too hard an issue.

[tommyreilly]

+1 on rekor transparency log verification to be supported as a feature to be added to connaisseur to keep it in equilibrium with cosign capabilities

[clem844]

Hi,
Any update on support for self hosted Rekor server?
My guess is we just need a way to pass the --rekor-url arg to cosign (and also set the environment variable COSIGN_EXPERIMENTAL=1 for now)

[xopham]

technically yes, that flag was non-functional in the past however. We have been doing some preparational work in #540 to better support the features beyond simple signing-verifying and are aiming to add support for other rekor urls soon.

However, as far as I can see there may be multiple hosts required and the option to configure the rootCA file: https://docs.sigstore.dev/cosign/openid_signing#custom-infrastructure

are you using rekor and fulcio and would need to configure both?

[clem844]

No Fulcio/OIDC in my current setup but it may be integrated in the future

[xopham]

ah, never mind. As we only do verification, I'd hope we do not need to integrate with Fulcio/OIDC, as all relevant info is in rekor.

To best understand the desired behavior. Say the following is configured for a validator in helm/values.yaml:

  - name: cosign-rekor-validator
    type: cosign
    host: rekor.sigstore.dev  # rekor url
    trust_roots:
      - name: my-key
        key: |
          -----BEGIN PUBLIC KEY-----
          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEOXYta5TgdCwXTCnLU09W5T4M4r9f
          QQrqJuADP6U7g5r9ICgPSmZuRHP/1AYUfOQW3baveKsT969EfELKj1lfCA==
          -----END PUBLIC KEY-----

In that case, we'd enforce a rekor entry, i.e. a cosign signature that has not been added to rekor will be treated as invalid. We could, in case necessary, at a later point add a switch for that later.
Would that be the desired behavior?

[clem844]

Yes that would be perfect

[xopham]

and will that require configuration of the rootCA? https://docs.sigstore.dev/cosign/openid_signing#custom-root-cert

[clem844]

No, I can deal without

[xopham]

@clem844 the above PR should add rekor-support. However, I noticed that cosign does store trust roots in $HOME/.sigstore which has the following form when validating via:

COSIGN_EXPERIMENTAL=1 cosign verify --key cosign.pub --rekor-url rekor.sigstore.dev $IMAGE

The trust roots have the following form:

/home/xoph/.sigstore
└── root
    └── targets
        ├── artifact.pub
        ├── ctfe.pub
        ├── fulcio.crt.pem
        ├── fulcio_v1.crt.pem
        └── rekor.pub

From the above I am not sure how cosign would handle multiple different rekor instances. Do you have a private rekor instance to check how that affects trust roots when verifying against rekor.sigstore.dev and your private rekor instance?
Would be much appreicated.

[clem844]

I did not initialize cosign with my own TUF signed public key of my Rekor instance. It was initialized with public instances.
Nevertheless, it is validating the signature and transparency log... Not sure why.

Verification for registry-proxy.example.io/artifacts-signature:latest --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The claims were present in the transparency log
  - The signatures were integrated into the transparency log when the certificate was valid
  - The signatures were verified against the specified public key

I will keep looking. If I open a bug on cosign I will reference it here

[clem844]

issue opened: sigstore/cosign#1816

[xopham]

The feature can be tested now. I have built a testbranch from #637 : https://github.com/sse-secure-systems/connaisseur/tree/test-pr637

It is configured such that, you could use our testimages or configure your own using your own or the public rekor instance, via the validator's host key. Our testimages are:

• signed, but not in transparency log: docker.io/securesystemsengineering/testimage:rekor-cosigned-notl
• signed and added to transparency log: docker.io/securesystemsengineering/testimage:rekor-cosigned-notl
  So if you install Connaisseur from the testbranch, you should get the following:

> kubectl run rekor --image=docker.io/securesystemsengineering/testimage:rekor-cosigned-tl
pod/rekor created
> kubectl run norekor --image=docker.io/securesystemsengineering/testimage:rekor-cosigned-notl
Error from server: admission webhook "connaisseur-svc.connaisseur.svc" denied the request: Failed to find signature in transparency log.

Let me know if if works for you and meets your vision for the feature 🙂
Next up, will be keyless signing 🚀

[clem844]

I tested it with my private Rekor and I get the following error:

[2022-05-02 13:17:39,545] INFO: {'message': 'Cosign timed out.', 'context': {'trust_data_type': 'dev.cosignproject.cosign/signature', 'detection_mode': False}}
[2022-05-02 13:17:39,548] ERROR: {'message': 'Cosign timed out.', 'context': {'trust_data_type': 'dev.cosignproject.cosign/signature', 'detection_mode': False, 'user': 'system:admin', 'operation': 'CREATE', 'kind': 'Pod', 'name': 'test-image-rekor', 'namespace': 'default'}}

This error is the same whether the signature is logged into Rekor or not.
In Rekor logs, I do not see any new requests from cosign.

[xopham]

did it work with the public Rekor instance? Are you sure that there is a connection to the private rekor instance? like can it be curled from within the cluster?

[clem844]

Yes, I can reach my Rekor instance from the cluster.
I just tried with the public rekor.sigstore.dev and there is an SSL error:
ssl.SSLError: [SSL: SSLV3_ALERT_BAD_CERTIFICATE] sslv3 alert bad certificate (_ssl.c:997)
with also the same cosign timed out

[clem844]

My bad: the SSL error has nothing to do with this.
I tried with the branch [test-pr637](https://github.com/sse-secure-systems/connaisseur/tree/test-pr637) as is and the image docker.io/securesystemsengineering/testimage:rekor-cosigned-tl that should be accepted. It still hits the same error.
With the public Rekor I cannot check if requests are being made to it. Is there a way to have more logs from cosign?

[xopham]

That is strange. Because the exact same works for me on two different local setups using kind and microk8s.
You should be able to find all available logs via kubectl logs. You can increase the verbosity for testing via logLevel key in helm/values.yaml: https://github.com/sse-secure-systems/connaisseur/blob/master/helm/values.yaml#L170
not sure if that helps...

[clem844]

Ok, I tried again today with my local setup (k3d). It works both with the images you provide and the public Rekor and my own images with a self hosted Rekor.
On my remote cluster it's probably a proxy problem. This is the error:

socket.error 8
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/cheroot/server.py", line 1300, in communicate
    req.respond()
  File "/usr/local/lib/python3.10/site-packages/cheroot/server.py", line 1090, in respond
    self.server.gateway(self).respond()
  File "/usr/local/lib/python3.10/site-packages/cheroot/wsgi.py", line 145, in respond
    self.write(chunk)
  File "/usr/local/lib/python3.10/site-packages/cheroot/wsgi.py", line 231, in write
    self.req.write(chunk)
  File "/usr/local/lib/python3.10/site-packages/cheroot/server.py", line 1146, in write
    self.conn.wfile.write(chunk)
  File "/usr/local/lib/python3.10/site-packages/cheroot/makefile.py", line 438, in write
    res = super().write(val, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/cheroot/makefile.py", line 36, in write
    self._flush_unlocked()
  File "/usr/local/lib/python3.10/site-packages/cheroot/makefile.py", line 45, in _flush_unlocked
    n = self.raw.write(bytes(self._write_buf))
  File "/usr/local/lib/python3.10/socket.py", line 723, in write
    return self._sock.send(b)
  File "/usr/local/lib/python3.10/ssl.py", line 1205, in send
    return self._sslobj.write(data)
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:2396)

One suggestion: it would be nice to have the cosign logs in the console. They are currently only in connaisseur's pod

[xopham]

neat 🙂
we'll consider that when reworking the error handling, it may make things a bit messy though, but guess that'll have to be seen