Skip to content

Commit 9b088f1

Browse files
authored
docs: proxy and CLI readme files (#20)
1 parent d80cbd9 commit 9b088f1

File tree

5 files changed

+178
-0
lines changed

5 files changed

+178
-0
lines changed

app/artifact-cas/README.md

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Artifact Content Addressable Storage (CAS) Proxy
2+
3+
The artifact proxy is a **Content Addressable Storage (CAS) Proxy** that sits in front of different storage backends.
4+
5+
Clients such as the Chainloop Control Plane or the CLI can use this proxy to make sure that **immutable uploads are performed**. Files that later, on **can only be referenced by their content digest (sha256)**.
6+
7+
NOTE: At the moment of this writing, only an [OCI](https://opencontainers.org/) storage backend is supported. In the future you should potentially expect Object Storage Support (i.e AWS s3) as well.
8+
9+
![cas](../../docs/img/cas-overview.png)
10+
11+
The project is a [Go](https://go.dev/) service that leverages [protocol buffers](https://github.com/protocolbuffers/protobuf) and [gRPC](https://grpc.io/) for its streaming API, [wire](https://github.com/google/wire/) for dependency injection and the [Kratos framework](https://github.com/go-kratos/kratos) for additional utilities such middlewares, configuration management or error handling.
12+
13+
The proxy API implements [a bytestream](https://pkg.go.dev/google.golang.org/api/transport/bytestream) gRPC service. This enables an efficient, and modern, streaming API for chunk based operations on top of HTTP/2.
14+
15+
Its structure contains the following top to down layers.
16+
17+
- API definition layer `./api/`. proto definitions and generated code for the external gRPC API
18+
- Server layer `./internal/server`. Definition and registration of the HTTP and gRPC servers and middlewares.
19+
- Service layer `./internal/service`. Implementation of the protocol buffer services.
20+
21+
## System Dependencies
22+
23+
The CAS proxy **has only one running dependency**. A secret storage backend to retrieve the OCI repository credentials. Currently we support both [Hashicorp Vault](https://www.vaultproject.io/) and [AWS Secret Manager](https://aws.amazon.com/secrets-manager/).
24+
25+
This secret backend is used to download OCI repository credentials (repository path + key pair) during upload/downloads. This makes the Artifact CAS multi-tenant by default since the destination OCI backend gets selected at runtime.
26+
27+
## AuthN/AuthZ
28+
29+
The Artifact CAS API expects each request to contain a [JSON Web Token](https://auth0.com/docs/secure/tokens/json-web-tokens) with references to a) what operation is this token allowed to do (Download, Upload) and b) a reference to where the CAS can find the OCI credentials.
30+
31+
Currently, this token is generated by the Control Plane and used on demand. You can find the generator we use to craft those tokens [here](../../internal/robotaccount/cas/robotaccount.go).
32+
33+
The token gets signed by the control plane with a private key and verified by the CAS with the previously configured public key.
34+
35+
Note: there are plans to support [JWKS endpoints](https://auth0.com/docs/secure/tokens/json-web-tokens/json-web-key-sets) to enable easy rotation of credentials.
36+
37+
## Runbook
38+
39+
We leverage `make` for most development tasks. Run `make -C app/artifact-cas` to see a list of the available tasks.
40+
41+
### Run the project in development
42+
43+
Refer to [development guide](../../devel/README.md) for more information but in a nutshell.
44+
45+
```
46+
# Run external dependency (Vault for secret management)
47+
48+
docker compose -f devel/docker-compose.yml up
49+
50+
# Run the service
51+
make -C app/artifact-cas run
52+
```
53+
54+
Next, follow the steps that can be found [here](../../devel/README.md#4---using-the-cli-pointing-to-the-local-environment) to configure the CLI
55+
56+
Once configured you can try to perform a file upload and download.
57+
58+
```sh
59+
cd app/cli
60+
go run main.go --insecure artifact upload -f [FILE] # will return a sha256 digest
61+
62+
# download
63+
go run main.go --insecure artifact download -d [sha256:...]
64+
```
65+
66+
### Run tests
67+
68+
```
69+
make test
70+
```
71+
72+
### Build binary
73+
74+
```
75+
make build
76+
```
77+
78+
### Generate API code from protocol buffer defintions (\*.proto)
79+
80+
We leverage buf.io to lint and generate proto files. Make sure you [install buf](https://docs.buf.build/installation) first. Once done, generating the API code is as easy as executing
81+
82+
```
83+
make api
84+
```
85+
86+
### Update configuration schema
87+
88+
The service runtime configuration is implemented by using [kratos built-in config module](https://go-kratos.dev/en/docs/component/config/).
89+
90+
Meaning that the configuration schema is defined at `internal/conf/conf.proto`.
91+
92+
To regenerate it run:
93+
94+
```
95+
make config
96+
```
97+
98+
### Update dependency injection
99+
100+
In order to enforce inversion of control and prevent import cycles we use [wire](https://github.com/google/wire/) for dependency injection.
101+
102+
Wire has a fairly steep learning curve, so we recommend taking a look [at their tutorial](https://github.com/google/wire/blob/main/_tutorial/README.md). In practice, in this project you will find a couple of wire_gen.go files (i.e `cmd/wire_gen.go`) and different provider defined.
103+
104+
If you need to re-generate the injection code after a change just run `make generate`
105+
106+
## Contribution guidelines
107+
108+
Please make sure to review the [Contribution guidelines](../../CONTRIBUTING.md) and feel free to reach out if you have any questions!
109+
110+
Welcome!

app/cli/README.md

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Command Line Interface (CLI)
2+
3+
This Command Line Interface (CLI) is a local client that's used for two purposes
4+
5+
a) Operator [Management Tasks](https://docs.chainloop.dev/getting-started/workflow-definition)
6+
7+
- operate on the [control plane](../controlplane/)
8+
- upload/download artifacts to the [artifact proxy](../artifact-cas/).
9+
10+
b) [Attestation Crafting Process](https://docs.chainloop.dev/getting-started/attestation-crafting)
11+
12+
- This CLI will be used in different CI/CD systems to perform the [attestation process](https://docs.chainloop.dev/getting-started/attestation-crafting)
13+
14+
![cas](../../docs/img/cli-overview.png)
15+
16+
The project is a [Go](https://go.dev/) CLI that leverages [Cobra](https://github.com/spf13/cobra) for CLI scaffolding, [Viper](https://github.com/spf13/viper) for configuration handling, [gRPC](https://grpc.io/) to communicate with both the control plane and the Artifact CAS APIs, and the [cosign](https://github.com/sigstore/cosign), [in-toto](https://github.com/in-toto/in-toto), [DSEE](https://github.com/secure-systems-lab/dsse/) and [SLSA](https://github.com/slsa-framework/slsa) projects to implement the attestation process.
17+
18+
## Runbook
19+
20+
We leverage `make` for most development tasks. Run `make -C app/cli` to see a list of the available tasks.
21+
22+
### Run the project in development
23+
24+
Refer to [development guide](../../devel/README.md) for more information but in a nutshell.
25+
26+
```
27+
go run app/cli/main.go --insecure
28+
```
29+
30+
### Configure the CLI to point to the local control plane and CAS services.
31+
32+
If you want to use this CLI pointing to a local or custom instance of Chainloop, you need to perform a config override this way.
33+
34+
```
35+
go run app/cli/main.go config save --insecure --control-plane localhost:9000 --artifact-cas localhost:9001
36+
```
37+
38+
### Run tests
39+
40+
```
41+
make test
42+
```
43+
44+
### Build binary
45+
46+
```
47+
make build
48+
```
49+
50+
### Generate API code from protocol buffer defintions (\*.proto)
51+
52+
We leverage buf.io to lint and generate proto files. Make sure you [install buf](https://docs.buf.build/installation) first. Once done, generating the API code is as easy as executing
53+
54+
```
55+
make api
56+
```
57+
58+
## Contribution guidelines
59+
60+
Please make sure to review the [Contribution guidelines](../../CONTRIBUTING.md) and feel free to reach out if you have any questions!
61+
62+
Welcome!

app/controlplane/README.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,12 @@ We've implemented two kinds of tests, unit tests (`make run test-unit`) and inte
5050

5151
To run all the tests `make test`
5252

53+
### Build binary
54+
55+
```
56+
make build
57+
```
58+
5359
### Generate API code from protocol buffer defintions (\*.proto)
5460

5561
We leverage buf.io to lint and generate proto files. Make sure you [install buf](https://docs.buf.build/installation) first.

docs/img/cas-overview.png

223 KB
Loading

docs/img/cli-overview.png

60.6 KB
Loading

0 commit comments

Comments
 (0)