diff --git a/app/artifact-cas/README.md b/app/artifact-cas/README.md new file mode 100644 index 000000000..b18ae5801 --- /dev/null +++ b/app/artifact-cas/README.md @@ -0,0 +1,110 @@ +# Artifact Content Addressable Storage (CAS) Proxy + +The artifact proxy is a **Content Addressable Storage (CAS) Proxy** that sits in front of different storage backends. + +Clients such as the Chainloop Control Plane or the CLI can use this proxy to make sure that **immutable uploads are performed**. Files that later, on **can only be referenced by their content digest (sha256)**. + +NOTE: At the moment of this writing, only an [OCI](https://opencontainers.org/) storage backend is supported. In the future you should potentially expect Object Storage Support (i.e AWS s3) as well. + +![cas](../../docs/img/cas-overview.png) + +The project is a [Go](https://go.dev/) service that leverages [protocol buffers](https://github.com/protocolbuffers/protobuf) and [gRPC](https://grpc.io/) for its streaming API, [wire](https://github.com/google/wire/) for dependency injection and the [Kratos framework](https://github.com/go-kratos/kratos) for additional utilities such middlewares, configuration management or error handling. + +The proxy API implements [a bytestream](https://pkg.go.dev/google.golang.org/api/transport/bytestream) gRPC service. This enables an efficient, and modern, streaming API for chunk based operations on top of HTTP/2. + +Its structure contains the following top to down layers. + +- API definition layer `./api/`. proto definitions and generated code for the external gRPC API +- Server layer `./internal/server`. Definition and registration of the HTTP and gRPC servers and middlewares. +- Service layer `./internal/service`. Implementation of the protocol buffer services. + +## System Dependencies + +The CAS proxy **has only one running dependency**. A secret storage backend to retrieve the OCI repository credentials. Currently we support both [Hashicorp Vault](https://www.vaultproject.io/) and [AWS Secret Manager](https://aws.amazon.com/secrets-manager/). + +This secret backend is used to download OCI repository credentials (repository path + key pair) during upload/downloads. This makes the Artifact CAS multi-tenant by default since the destination OCI backend gets selected at runtime. + +## AuthN/AuthZ + +The Artifact CAS API expects each request to contain a [JSON Web Token](https://auth0.com/docs/secure/tokens/json-web-tokens) with references to a) what operation is this token allowed to do (Download, Upload) and b) a reference to where the CAS can find the OCI credentials. + +Currently, this token is generated by the Control Plane and used on demand. You can find the generator we use to craft those tokens [here](../../internal/robotaccount/cas/robotaccount.go). + +The token gets signed by the control plane with a private key and verified by the CAS with the previously configured public key. + +Note: there are plans to support [JWKS endpoints](https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-key-sets) to enable easy rotation of credentials. + +## Runbook + +We leverage `make` for most development tasks. Run `make -C app/artifact-cas` to see a list of the available tasks. + +### Run the project in development + +Refer to [development guide](../../devel/README.md) for more information but in a nutshell. + +``` +# Run external dependency (Vault for secret management) + +docker compose -f devel/docker-compose.yml up + +# Run the service +make -C app/artifact-cas run +``` + +Next, follow the steps that can be found [here](../../devel/README.md#4---using-the-cli-pointing-to-the-local-environment) to configure the CLI + +Once configured you can try to perform a file upload and download. + +```sh +cd app/cli +go run main.go --insecure artifact upload -f [FILE] # will return a sha256 digest + +# download +go run main.go --insecure artifact download -d [sha256:...] +``` + +### Run tests + +``` +make test +``` + +### Build binary + +``` +make build +``` + +### Generate API code from protocol buffer defintions (\*.proto) + +We leverage buf.io to lint and generate proto files. Make sure you [install buf](https://docs.buf.build/installation) first. Once done, generating the API code is as easy as executing + +``` +make api +``` + +### Update configuration schema + +The service runtime configuration is implemented by using [kratos built-in config module](https://go-kratos.dev/en/docs/component/config/). + +Meaning that the configuration schema is defined at `internal/conf/conf.proto`. + +To regenerate it run: + +``` +make config +``` + +### Update dependency injection + +In order to enforce inversion of control and prevent import cycles we use [wire](https://github.com/google/wire/) for dependency injection. + +Wire has a fairly steep learning curve, so we recommend taking a look [at their tutorial](https://github.com/google/wire/blob/main/_tutorial/README.md). In practice, in this project you will find a couple of wire_gen.go files (i.e `cmd/wire_gen.go`) and different provider defined. + +If you need to re-generate the injection code after a change just run `make generate` + +## Contribution guidelines + +Please make sure to review the [Contribution guidelines](../../CONTRIBUTING.md) and feel free to reach out if you have any questions! + +Welcome! diff --git a/app/cli/README.md b/app/cli/README.md new file mode 100644 index 000000000..550751ec9 --- /dev/null +++ b/app/cli/README.md @@ -0,0 +1,62 @@ +# Command Line Interface (CLI) + +This Command Line Interface (CLI) is a local client that's used for two purposes + +a) Operator [Management Tasks](https://docs.chainloop.dev/getting-started/workflow-definition) + +- operate on the [control plane](../controlplane/) +- upload/download artifacts to the [artifact proxy](../artifact-cas/). + +b) [Attestation Crafting Process](https://docs.chainloop.dev/getting-started/attestation-crafting) + +- This CLI will be used in different CI/CD systems to perform the [attestation process](https://docs.chainloop.dev/getting-started/attestation-crafting) + +![cas](../../docs/img/cli-overview.png) + +The project is a [Go](https://go.dev/) CLI that leverages [Cobra](https://github.com/spf13/cobra) for CLI scaffolding, [Viper](https://github.com/spf13/viper) for configuration handling, [gRPC](https://grpc.io/) to communicate with both the control plane and the Artifact CAS APIs, and the [cosign](https://github.com/sigstore/cosign), [in-toto](https://github.com/in-toto/in-toto), [DSEE](https://github.com/secure-systems-lab/dsse/) and [SLSA](https://github.com/slsa-framework/slsa) projects to implement the attestation process. + +## Runbook + +We leverage `make` for most development tasks. Run `make -C app/cli` to see a list of the available tasks. + +### Run the project in development + +Refer to [development guide](../../devel/README.md) for more information but in a nutshell. + +``` +go run app/cli/main.go --insecure +``` + +### Configure the CLI to point to the local control plane and CAS services. + +If you want to use this CLI pointing to a local or custom instance of Chainloop, you need to perform a config override this way. + +``` +go run app/cli/main.go config save --insecure --control-plane localhost:9000 --artifact-cas localhost:9001 +``` + +### Run tests + +``` +make test +``` + +### Build binary + +``` +make build +``` + +### Generate API code from protocol buffer defintions (\*.proto) + +We leverage buf.io to lint and generate proto files. Make sure you [install buf](https://docs.buf.build/installation) first. Once done, generating the API code is as easy as executing + +``` +make api +``` + +## Contribution guidelines + +Please make sure to review the [Contribution guidelines](../../CONTRIBUTING.md) and feel free to reach out if you have any questions! + +Welcome! diff --git a/app/controlplane/README.md b/app/controlplane/README.md index 6b58f80c4..368986a61 100644 --- a/app/controlplane/README.md +++ b/app/controlplane/README.md @@ -50,6 +50,12 @@ We've implemented two kinds of tests, unit tests (`make run test-unit`) and inte To run all the tests `make test` +### Build binary + +``` +make build +``` + ### Generate API code from protocol buffer defintions (\*.proto) We leverage buf.io to lint and generate proto files. Make sure you [install buf](https://docs.buf.build/installation) first. diff --git a/docs/img/cas-overview.png b/docs/img/cas-overview.png new file mode 100644 index 000000000..07242e0cd Binary files /dev/null and b/docs/img/cas-overview.png differ diff --git a/docs/img/cli-overview.png b/docs/img/cli-overview.png new file mode 100644 index 000000000..1e14bac19 Binary files /dev/null and b/docs/img/cli-overview.png differ