Skip to content

feat: inline CAS backend support in client #246

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 18, 2023
Merged

feat: inline CAS backend support in client #246

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ee6fef3
feat: inline CAS backend support in client
migmartri Jul 17, 2023
3e65e37
feat: add tests
migmartri Jul 17, 2023
4336385
feat: add tests
migmartri Jul 17, 2023
ffacad6
feat: update tests
migmartri Jul 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 6 additions & 5 deletions app/cli/api/attestation/v1/crafting_state.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,21 +20,22 @@ import (
)

type NormalizedMaterialOutput struct {
Value, Digest string
IsOutput bool
Name, Digest string
IsOutput bool
Content []byte
}

func (m *Attestation_Material) NormalizedOutput() *NormalizedMaterialOutput {
switch m.MaterialType {
case schemaapi.CraftingSchema_Material_ARTIFACT, schemaapi.CraftingSchema_Material_SBOM_CYCLONEDX_JSON, schemaapi.CraftingSchema_Material_SBOM_SPDX_JSON:
a := m.GetArtifact()
return &NormalizedMaterialOutput{a.Name, a.Digest, a.IsSubject}
return &NormalizedMaterialOutput{a.Name, a.Digest, a.IsSubject, a.Content}
case schemaapi.CraftingSchema_Material_CONTAINER_IMAGE:
a := m.GetContainerImage()
return &NormalizedMaterialOutput{a.Name, a.Digest, a.IsSubject}
return &NormalizedMaterialOutput{a.Name, a.Digest, a.IsSubject, nil}
case schemaapi.CraftingSchema_Material_STRING:
a := m.GetString_()
return &NormalizedMaterialOutput{Value: a.Value}
return &NormalizedMaterialOutput{Content: []byte(a.Value)}
}

return nil
Expand Down
139 changes: 81 additions & 58 deletions app/cli/api/attestation/v1/crafting_state.pb.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions app/cli/api/attestation/v1/crafting_state.pb.validate.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 6 additions & 0 deletions app/cli/api/attestation/v1/crafting_state.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,9 @@ message Attestation {

// Whether the material has been uploaded to the CAS
bool uploaded_to_cas = 7;
// If the material content has been injected inline in the attestation
// leveraging a form of inline CAS
bool inline_cas = 8;

message KeyVal {
string id = 1 [(validate.rules).string.min_len = 1];
Expand All @@ -65,6 +68,9 @@ message Attestation {
// which also has annotated the fileName
string digest = 3 [(validate.rules).string.min_len = 1];
bool is_subject = 4;
// Inline content of the artifact.
// This is optional and is used for small artifacts that can be stored inline in the attestation
bytes content = 5;
}
}

Expand Down
16 changes: 15 additions & 1 deletion app/cli/internal/action/attestation_add.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,21 @@ func (action *AttestationAdd) Run(k, v string) error {

cc := casclient.New(artifactCASConn, casclient.WithLogger(action.Logger))

if err := action.c.AddMaterial(k, v, cc); err != nil {
// // TODO: retrieve this information from backend
backend := &casclient.CASBackend{
Uploader: cc,
Name: "OCI",
// 100MB max size
MaxSize: 100 * 1024 * 1024,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would keep a separate limit for inline and a default one like this for CAS. I think I would lower this value

Copy link
Member Author

@migmartri migmartri Jul 18, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, note that this is changing in #247

$ go run main.go --insecure cas-backend ls                                                                                    
┌──────────────────────────────────────┬─────────────────────────────────────┬──────────┬─────────────────────────────────────┬───────────────┬─────────┐     │ ID                                   │ LOCATION                            │ PROVIDER │ DESCRIPTION                         │ LIMITS        │ DEFAULT │
├──────────────────────────────────────┼─────────────────────────────────────┼──────────┼─────────────────────────────────────┼───────────────┼─────────┤     │ 36c9b3c2-6946-4972-a832-f0137d7646d7 │ us-central1-docker.pkg.dev/axiomati │ OCI      │ fopoobar                            │ MaxSize: 100M │ false   │
│                                      │ c-grove-366622/chainloop-demo-org   │          │                                     │               │         │     ├──────────────────────────────────────┼─────────────────────────────────────┼──────────┼─────────────────────────────────────┼───────────────┼─────────┤
│ f99fbbae-2a7a-4be3-94bf-1850bfa4b8c2 │                                     │ INLINE   │ Embed artifacts content in the atte │ MaxSize: 500K │ true    │     │                                      │                                     │          │ station                             │               │         │
└──────────────────────────────────────┴─────────────────────────────────────┴──────────┴─────────────────────────────────────┴───────────────┴─────────┘

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: reducing max size. Until we support changing it I'd not go lower than what we have today. We'd need to check the size of all our binaries and such to make sure they fit.

}

// backend := &casclient.CASBackend{
// Name: "INLINE",
// // 500KB max size
// MaxSize: 500 * 1024,
// }

if err := action.c.AddMaterial(k, v, backend); err != nil {
action.Logger.Err(err).Msg("adding material")
return err
}
Expand Down
18 changes: 14 additions & 4 deletions app/controlplane/internal/service/attestation.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -298,10 +298,15 @@ func bizAttestationToPb(att *biz.Attestation) (*cpAPI.AttestationItem, error) {
return nil, fmt.Errorf("error extracting predicate from attestation: %w", err)
}

materials, err := extractMaterials(predicate.GetMaterials())
if err != nil {
return nil, fmt.Errorf("error extracting materials from attestation: %w", err)
}

return &cpAPI.AttestationItem{
Envelope: encodedAttestation,
EnvVars: extractEnvVariables(predicate.GetEnvVars()),
Materials: extractMaterials(predicate.GetMaterials()),
Materials: materials,
}, nil
}

Expand All @@ -319,18 +324,23 @@ func extractEnvVariables(in map[string]string) []*cpAPI.AttestationItem_EnvVaria
return res
}

func extractMaterials(in []*chainloop.NormalizedMaterial) []*cpAPI.AttestationItem_Material {
func extractMaterials(in []*chainloop.NormalizedMaterial) ([]*cpAPI.AttestationItem_Material, error) {
res := make([]*cpAPI.AttestationItem_Material, 0, len(in))
for _, m := range in {
// Initialize simply with the value
displayValue := m.Value
// Override if there is a hash attached
if m.Hash != nil {
displayValue = fmt.Sprintf("%s@%s", m.Value, m.Hash)
name := m.Value
if m.EmbeddedInline || m.UploadedToCAS {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this deserve a separate helper function like is_downloadable.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, the problem is that embedded doesn't mean that can be downloaded.

In fact artifact download doesn't work with embedded (inline) artifacts. We'd need to have an artifact-attestation mapping to make it work.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I mean, we can call it differently but we already use this "if m.EmbeddedInline || m.UploadedToCAS" in two places.

name = m.Filename
}

displayValue = fmt.Sprintf("%s@%s", name, m.Hash)
}

res = append(res, &cpAPI.AttestationItem_Material{Name: m.Name, Value: displayValue, Type: m.Type})
}

return res
return res, nil
}
Loading