The artifact proxy is a Content-Addressable Storage (CAS) Proxy that sits in front of different storage backends.
Clients such as the Chainloop Control Plane or the CLI use this proxy to make sure that uploaded artifacts are immutable and uniquely identifiable by their content digest (sha256sum).
NOTE: At the moment of this writing, only an OCI storage backend is supported. In the future you should potentially expect Object Storage Support (i.e AWS s3) as well.
The project is a Go service that leverages protocol buffers and gRPC for its streaming API, wire for dependency injection and the Kratos framework for additional utilities such middlewares, configuration management or error handling.
The proxy API implements a bytestream gRPC service. This enables an efficient, and modern, streaming API on top of HTTP/2.
Its structure contains the following top to down layers.
- API definition layer
./api/. proto definitions and generated code for the external gRPC API - Server layer
./internal/server. Definition and registration of the HTTP and gRPC servers and middlewares. - Service layer
./internal/service. Implementation of the protocol buffer services.
The CAS proxy has only one running dependency. A secret storage backend to retrieve the OCI repository credentials. Currently, we support Hashicorp Vault, AWS Secrets Manager AND GCP Secret Manager.
This secret backend is used to download OCI repository credentials (repository path + key pair) during upload/downloads. This makes the Artifact CAS multi-tenant by default since the destination OCI backend gets selected at runtime.
The Artifact CAS API expects each request to contain a JSON Web Token with references to a) what operation is this token allowed to do (download or upload) and b) a reference to where the CAS can find the target OCI credentials.
Currently, this token is generated by the Control Plane and used on demand. You can find the generator we use to craft those tokens here.
The token gets signed by the control plane with a private key and verified by the CAS with the previously configured public key.
Note: there are plans to support JWKS endpoints to enable easy rotation of credentials.
The client code can be found here
We use make for most development tasks. Run make -C app/artifact-cas to see a list of the available tasks.
Refer to development guide for more information but in a nutshell.
# Run external dependency (Vault for secret management)
docker compose -f devel/compose.yml up
# Run the service
make -C app/artifact-cas run
Next, follow the steps that can be found here to configure the CLI
Once configured you can try to perform a file upload and download.
cd app/cli
go run main.go --insecure artifact upload -f [FILE] # will return a sha256 digest
# download
go run main.go --insecure artifact download -d [sha256:...]make test
make build
We use buf.io to lint and generate proto files. Make sure you install buf first. Once done, generating the API code is as easy as executing
make api
The service runtime configuration is implemented by using kratos built-in config module.
Meaning that the configuration schema is defined at internal/conf/conf.proto.
To regenerate it run:
make config
In order to enforce inversion of control and prevent import cycles we use wire for dependency injection.
Wire has a fairly steep learning curve, so we recommend taking a look at their tutorial. In practice, in this project you will find a couple of wire_gen.go files (i.e cmd/wire_gen.go) and different provider defined.
If you need to re-generate the injection code after a change just run make generate
Please make sure to review the Contribution guidelines and feel free to reach out if you have any questions!
Welcome!