feat: cosign easy mode

[miki725]

• Followed the steps in the contributor's guide: https://crashoverride.com/docs/other/contributing#filing-the-pull-request
• PR title uses semantic commit messages
• Filled out the template to a useful degree

Description

attestation requires 3 pieces of information:

• public key
• encrypted private key
• password to decrypt private key

Normally they are manually managed:

# import existing key
$ ls chalk.*
chalk.key.   chalk.pub
$ CHALK_PASSWORD=... chalk setup

# use attestation
$ CHALK_PASSWORD=.... chalk insert ...

this is cumbersome as it required to fully manually manage the key

We also have signing key backup service which allows to backup private key password however that still required to manage a consistent key material across the org. For example in CI you want all builds to use a consistent key.

To help with that this PR allows to fetch key material from an API (with auth of course). As such it will allow all places across org which require signing to fetch the key material and therefore allow to sign artifacts with easy - hence the cosign easy mode.

To make the configuration for managing key material much simpler this PR adds a notion of attestation key provider which is responsible for either minting or retrieving existing key material. The user can select which provider they would like to use via attestation.key_provider configuration value. In addition each key provider supports additional configuration values:

attestation {
  key_provider = "get"
  attestation_key_get {
    uri = "https://..."
    ...
  }
}

In the future as user requests come in we plan on adding more key providers such as aws secrets manager, hashicorp vault to make it very easy for orgs to add attestation to their builds and chalk will take care of the rest.

Changes

• there is no more chalk setup gen and chalk setup load. each key provider implements its own logic how to retrieve/generate keys. as such chalk setup does whatever the key provider supports
  • embed will attempt to load existing keys. else will generate new key-material. it will never overwrite existing key and will show error describing that existing key would need to be removed first.
  • backup same thing as embed
  • get always fetches key from API. local keys are never imported
• removed other cli flags from chalk setup. some of them werent honored anymore like --store-password. also CLI allowed to overwrite the key location config however as thats by key provider now, that option was removed
• there is no more interactive prompt for the password. to import existing key CHALK_PASSWORD env var needs to be used. this makes same UX for either importing or using chalk later on. this simplified a bunch of logic around key handling. also the prompt wasnt fully functional before. for example it didnt support ctrl-c during the prompt...

Testing

$ make tests args="test_command.py::test_setup test_command.py::test_setup_existing_keys -x"

TODO

• adding get() for dicts con4m#125
• update PR description. for now please see some of the commit messages
• address all TODOs in code
• update all signing related docs
• add how-to for the new get provider
• update all configs in chalkdust
• resign chalkmark in autocomplete scripts

[miki725]

everything below hasnt changes except some minor refactor with function signature changes/etc. the key provider changes are all above in this file

[ee7]

Would it be better for this to be the latest version? cosign 2.1.1 was released in June 2023.

I saw your mention of this issue, but why does that imply that we should pin an older version here? I notice that the other uses in this repo are not pinned:

chalk/Dockerfile

Line 2 in 6117a5b

FROM gcr.io/projectsigstore/cosign as cosign

chalk/tests/Dockerfile

Line 54 in 6117a5b

COPY --from=gcr.io/projectsigstore/cosign /ko-app/cosign /usr/local/bin/cosign

And chalk downloads the latest cosign version:

chalk/src/configs/attestation.c4m

Line 24 in 6117a5b

install_url := "https://github.com/sigstore/cosign/releases/latest/download/" + install_name

[miki725]

issue is when you download a key generated by latest version of cosign but attempt to use it with older version of cosign. hence explicitly pinning the test server to use older version so it can generate compatible keys with everything else which consumes it

no need to pin on chalk side as we want to test with latest cosign

[ee7]

I'll try to finish reviewing tomorrow, but here's some nits in the meantime. Some of them may not have been introduced by this PR - sorry about that.

Looks promising though.

[ee7]

In case it helps, because I think the official Nim docs could be clearer here: does the below code clarify the behavior you were seeing with subtypes?

With inheritance, we can convert a subtype to its parent type, and vice versa. We cannot convert a subtype to a subtype, but in particular note that the exact behavior depends on which memory management option is used. Chalk currently uses --mm:refc, which means there is a class of bugs that produce a Defect (an exception that should not be caught) at run time rather than a compile error. Before Nim 2.0, the default memory management strategy was refc; with Nim 2.0, the default is orc.

As an aside: one difference between inheritance and object variants is that the latter allows changing one object kind to another. That must be done by assigning to a new object, because mutating the discriminator alone is not allowed.

type
  Animal = ref object of RootObj
    weight = 0

  Cat = ref object of Animal
    lives = 9

  Dog = ref object of Animal
    isLoud: bool

var
  cat1 = Cat()
  cat2: Animal = Cat()
  dog = Dog()

doAssert $typeof(cat1) == "Cat"
doAssert $typeof(cat2) == "Animal"

# cat1 = dog      # Errors at compile time.
# cat1 = Animal() # Errors at compile time.
# cat2 = Animal() # OK

# There is a difference in behavior between ORC memory management
# (which is the default in Nim 2.0) and refc (which chalk currently uses).
when defined(gcOrc) or defined(gcArc):
  cat2 = Dog(cat2)
  doAssert $typeof(cat2) == "Animal" # not Dog.
  # echo cat2.isLoud # Errors at compile time. isLoud is undeclared for type Animal.
elif defined(gcRefc):
  doAssertRaises(ObjectConversionDefect):
    cat2 = Dog(cat2) # Raises defect at run time.

# Procedures use static dispatch.
# cat2, which is of type Animal, uses the proc for Animal.
proc getWeight(animal: Animal): int =
  animal.weight

proc getWeight(cat: Cat): int =
  42

doAssert getWeight(cat1) == 42
doAssert getWeight(cat2) == 0

# Methods use dynamic dispatch.
# cat2, which is of type Animal, uses the method for Cat.
method getW(animal: Animal): int {.base.} =
  animal.weight

method getW(cat: Cat): int =
  42

doAssert getW(cat1) == 42
doAssert getW(cat2) == 42