merge upstream grafeas changes while preserving commit log (grafeas#7)

* updating json examples and extended the grpc client example * Deal with Windows line-endings that git introduces * Allow for Windows protoc download * Deal with explicitly shelling out via bash * Unzip on Windows slow to release file handles * fix marshalling error when deleting projects * backfill tests * Add tests on permission checking Add some tests to validate the permission checks on UpdateOccurrence operations. To support the tests, add a allowListAuth test implementation of Auth that just stores a list of allowed project/permission pairs. Two of the tests are documenting incorrect behavior that will be fixed in a later commit. Namely: (1) Updating the NoteName should require NotesAttachOccurrence on the old NoteName (as this is effectively a detach operation). (2) When excluding the NoteName from the update using an UpdateMask, the permission check should be performed against the existing NoteName, not whatever was passed in (and which will be ignored). * Fix permission checks in occurrence update function This commit fixes the permission checks in UpdateOccurrence and updates the tests from the last commit to demonstrate the change. In particular, we now check the NotesAttachOccurrence on the occurrence's existing NoteName before allowing updates. We continue to check the NotesAttachOccurrence on the updated NoteName (if it is different from the existing NoteName whose permission has already been checked). This means that updating an occurrence from one note to another requires NotesAttachOccurrence on both notes. * Changes to v1 proto only * fix the field number of inclusive. * also add changes to v1beta1 * fix typo * Update running_grafeas.md The instructions for using the curl command with the certificate didn't work for me. I got the error message: `curl: (58) unable to set private key file: 'server.pem' type PEM` When I added these additional flags, it worked fine. * add grafeas-elasticsearch to list of backends * Add the vendor field to Vulnerability.Detail * Increase max page size in v1 API: 1k -> 10k. * fixes ARM build * initial commit of grafeas proto * add files generated by running make * Added roadmap and clarifications for architecture * addressing comments on the PR * further comments addressed * Add the non_compliance_reason field to ComplianceOccurrence The field is necessary to support the cases where the benchmark is non-compliant but there are no associated files, like the upcoming SQL checks. * remove unused deps from compliance proto * add COMPLIANCE to note kind * Added SECURITY guidelines * fix: broken link * Add the vendor info to the V1 api. * Initial commit of DSSE Attestation and IntotoProvenance support. * Remove prefix from intotoprovenance * Add generation code. * Add dsse references in grafeas.proto * Fixes in reference to the comments. * Fix common.proto field format * Change names of fields to work around naming conflicts. * Renamed Builder to BuilderConfig, to not conflict with the Java generated Builder class. * Make casing of dsse consistent * Make casing of dsse consistent * Move the package type and effective severity to the PackageIssue level When we are creating vulnerabilities, we may find vulnerabilities for multiple different package types (e.g. OS, Maven, NPM, Go, etc.) on a single Note. Since type is currently on the Vulnerability occurrence level, we need to move it down into the PackageIssue level. Similarly, vulnerabilities issued for different package types (potentially by different vulnerability sources) can have different effective severities. Therefore, this field also needs to be moved down into the PackageIssue level. * Change intoto provenance arguments to be of Any type. * Change intoto provenance arguments to be of Any type. * Fix small typo * Add comment about "Any" * SBOM protos * move spdx proto to v1beta1 and split into seperate files * Add TODO to encapsulate spdx license fields * Change arguments and recipe from []Any to just Any, to more closely follow the spec. * Add SPDX Relationships protos and consolidate SPDX protos into one file. * Remove obsolete SPDX files. * Added fields to PackageInfoOccurrence and RelationshipNote * Created License message * Use License message instead of the string field type * Use annotations for output only fields * Clean up comments for output only fields * Top-level enums should appear ahead of messages * Correction: top-level enums should appear after messages * 1. Add a new type, of type slsa_provenance, which exactly follows the slsa spec (as opposed to the intoto_provenance, which has a few differences). 2. Add slsa_provenance as a predicate type to the intoto_statement 3. Add the intoto_statement to the build type. 4. Change the json serialization of the intoto_statement type to "_type" to follow the spec. * Correct import paths * Keep intoto_provenance as-is. * Cmment out the slsa provenance to see if compilation works. * Fix spelling of deployable * 1. Move intoto_statement to its own file, replacing dsse_attestation and references to it. 2. Add slsa_provenaecn and intoto_statement to generate file. * Add build verification and tests * Keep dsse attestation, just move around intoto_statement to its own file. * Keep dsse attestation with the original statement * Added links for clarification. * Rename Builder message to BuilderConfig * Prepend messages with "Slsa" to avoid naming conflicts in the "grafeas.v1" package. * changes for cvss v3 * Add grafeas-rds to list of backends. * use backticks for strings * regen with protoc 3.11.4 and grpc-gateway 1.19.6 * add v1 proto generated code * preserve original v1 generate.go * regen with v2 and sync with master * add git validation that was done in https://git.vzbuilders.com/maditya/grafeas-orig/pull/3 * add pg filter based on mysql filter * Remove pgsql implementation (#1) All the pgsql specific code will be moved out to tetradio repo, as mentioned upstream in grafeas#341 When we are ready, we can submit this and pgsql code to be merged upstream. * add custom uploader_id and content_hash field for package type (grafeas#2) Co-authored-by: jwang04 <jwang04@verizonmedia.com> * regen with protoc 3.11.4 and grpc-gateway 1.19.6 * add v1 proto generated code * preserve original v1 generate.go * regen with v2 and sync with master * add pg filter based on mysql filter * initial working version * fix kind sql parsing * Remove pgsql implementation (#1) All the pgsql specific code will be moved out to tetradio repo, as mentioned upstream in grafeas#341 When we are ready, we can submit this and pgsql code to be merged upstream. Co-authored-by: greendinosaur <greendinosaur@users.noreply.github.com> Co-authored-by: Sean Dukehart <tomcruise81@users.noreply.github.com> Co-authored-by: Michael Parker <michael@parker.gg> Co-authored-by: wkozlik <61755482+wkozlik@users.noreply.github.com> Co-authored-by: Eric Zimanyi <ezimanyi@google.com> Co-authored-by: Aysylu Greenberg <aysylu@users.noreply.github.com> Co-authored-by: Neetha Sebastian <neethas@google.com> Co-authored-by: Dina Graves Portman <dinagraves@google.com> Co-authored-by: Yinpeng Li <yinpengli@google.com> Co-authored-by: Catherine Jones <31020910+catherinejones@users.noreply.github.com> Co-authored-by: Pietro Ferretti <pferretti@google.com> Co-authored-by: Nolan Emirot <emirot.nolan@gmail.com> Co-authored-by: vyinpengli <mousquetairesyl@gmail.com> Co-authored-by: sherzberg <sherzberg@google.com> Co-authored-by: Nicholas Cho <nickcho2@gmail.com> Co-authored-by: Yousef Alowayed <yousef.alowayed@gmail.com> Co-authored-by: Wiktor Kozlik <wkozlik@google.com> Co-authored-by: Ethan Anderson <eanderson@atlassian.com> Co-authored-by: yzhao02 <yonghe.zhao@verizonmedia.com> Co-authored-by: Aditya Mahendrakar <maditya@verizonmedia.com> Co-authored-by: Junyi Wang <junyi.wang@oath.com> Co-authored-by: jwang04 <jwang04@verizonmedia.com>
hkadakia · Nov 19, 2021 · 519ca5e · 519ca5e
1 parent 7253f41
commit 519ca5e
Show file tree

Hide file tree

Showing 76 changed files with 5,049 additions and 12,993 deletions.
diff --git a/README.md b/README.md
@@ -29,6 +29,16 @@ files](https://github.com/Grafeas/Grafeas/tree/master/proto/v1beta1).
 
 ## Grafeas Architecture
 
+Grafeas project consists of
+
+* the Grafeas API,
+* a reference server implementation,
+* [3 community contributed storage backends](https://github.com/grafeas/grafeas/tree/master/go/v1beta1/storage):
+PostgreSQL, BoltDB, and in-memory storage.
+
+Longer-term, these are to be extracted into separate projects (see
+[#341](https://github.com/grafeas/grafeas/issues/341)).
+
 The diagram below shows the boundaries between Grafeas API, server, its storage
 backends and the clients:
 
@@ -40,6 +50,13 @@ The following projects provide bindings for Grafeas API to different storage bac
 
 * [grafeas-pgsql](https://github.com/grafeas/grafeas-pgsql)
 * [grafeas-oracle](https://github.com/judavi/grafeas-oracle)
+* [grafeas-elasticsearch](https://github.com/rode/grafeas-elasticsearch)
+* [grafeas-rds](https://github.com/theparanoids/grafeas-rds)
+
+## Roadmap
+
+Please see the [Grafeas roadmap](https://www.slideshare.net/aysylu/binary-authorization-in-kubernetes/65)
+for the future of the project development.
 
 ## Support
 

diff --git a/SECURITY.md b/SECURITY.md
@@ -0,0 +1,5 @@
+To report a security issue, please email [grafeas-security-report](mailto:grafeas-security-report@google.com)
+with a description of the issue, the steps you took to create the issue,
+affected versions, and if known, mitigations for the issue. Our vulnerability
+management team will acknowledge receiving your email within 3 working days.
+This project follows a 90 day disclosure timeline.
diff --git a/docs/json_examples/note_build.json b/docs/json_examples/note_build.json
@@ -1,4 +1,5 @@
 {
+	"name": "projects/notes_example/notes/exampleBuildNote",
     "shortDescription": "Gives an example build note",
     "longDescription": "Shows the fields that need to be set in order to create a build note inside a project",
     "kind": "BUILD",

diff --git a/docs/json_examples/note_deployment.json b/docs/json_examples/note_deployment.json
@@ -1,4 +1,5 @@
 {
+    "name": "projects/notes_example/notes/exampleDeploymentNote",
     "shortDescription": "Gives an example deployment note",
     "longDescription": "Shows the fields that need to be set in order to create a deployment note inside a project",
     "kind": "DEPLOYMENT",

diff --git a/docs/json_examples/note_vulnerability.json b/docs/json_examples/note_vulnerability.json
@@ -1,4 +1,5 @@
 {    
+    "name": "projects/notes_example/notes/exampleVulnerabilityNote",
     "shortDescription": "Gives an example vulnerability note",
     "longDescription": "Shows the fields that need to be set in order to create a vulnerability note inside a project",
     "kind": "VULNERABILITY",

diff --git a/docs/running_grafeas.md b/docs/running_grafeas.md
@@ -138,12 +138,12 @@ openssl pkcs12 -in server.p12 -out server.pem -clcerts
 Now, `curl` the endpoint:
 
 ```bash
-curl -k --cert server.pem https://localhost:8080/v1beta1/projects
+curl -k --key server.key --cacert ca.pem --cert server.pem https://localhost:8080/v1beta1/projects
 ```
 
 ### gRPC with a go client
 
-[client.go](../go/v1beta1/example/client.go) contains a small example of a go
+[client.go](../go/v1beta1/example/client/client.go) contains a small example of a go
 client that connects to Grafeas and outputs the notes in `myproject`:
 
 ```bash

diff --git a/go.mod b/go.mod
@@ -1,6 +1,6 @@
 module github.com/grafeas/grafeas
 
-go 1.12
+go 1.14
 
 require (
 	github.com/antlr/antlr4 v0.0.0-20190819145818-b43a4c3a8015
@@ -12,11 +12,12 @@ require (
 	github.com/google/uuid v1.1.2
 	github.com/grpc-ecosystem/grpc-gateway/v2 v2.0.1
 	github.com/rs/cors v1.7.0
-	github.com/spf13/viper v1.4.0
-	golang.org/x/net v0.0.0-20200822124328-c89045814202
-	google.golang.org/genproto v0.0.0-20201019141844-1ed22bb0c154
-	google.golang.org/grpc v1.33.1
-	google.golang.org/grpc/examples v0.0.0-20201222210138-66c1393796be // indirect
+	github.com/spf13/viper v1.7.1
+	golang.org/x/net v0.0.0-20201110031124-69a78807bb2b
+	golang.org/x/sys v0.0.0-20210426230700-d19ff857e887 // indirect
+	google.golang.org/genproto v0.0.0-20201113130914-ce600e9a6f9e
+	google.golang.org/grpc v1.33.2
+	google.golang.org/grpc/examples v0.0.0-20201112215255-90f1b3ee835b // indirect
 	google.golang.org/protobuf v1.25.0
-	gopkg.in/yaml.v2 v2.2.8 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
 )
diff --git a/go.sum b/go.sum
diff --git a/go/filtering/parser/parser_test.go b/go/filtering/parser/parser_test.go
@@ -18,6 +18,7 @@ package parser
 import (
 	"fmt"
 	"io/ioutil"
+	"runtime"
 	"strings"
 	"testing"
 
@@ -93,21 +94,26 @@ func verifyBaseline(t *testing.T, baseline baseline) {
 }
 
 func newTestBaselines(filename string) ([]baseline, error) {
+	lineSeparator := "\n"
+	if runtime.GOOS == "windows" {
+		lineSeparator = "\r\n"
+	}
+
 	bytes, err := ioutil.ReadFile(fmt.Sprintf("testdata/%s.baseline", filename))
 	if err != nil {
 		panic(fmt.Sprintf("Could not read provided file: %s", filename))
 	}
-	testCases := strings.Split(string(bytes), TestCaseDelimiter)
+	testCases := strings.Split(string(bytes), strings.ReplaceAll(TestCaseDelimiter, "\n", lineSeparator))
 	baselines := make([]baseline, len(testCases))
 	for i, testCase := range testCases {
 		testCaseName := fmt.Sprintf("%s[%d]", filename, i)
-		inputOutput := strings.Split(testCase, InputOutputDelimiter)
+		inputOutput := strings.Split(testCase, strings.ReplaceAll(InputOutputDelimiter, "\n", lineSeparator))
 		input, output := inputOutput[0], inputOutput[1]
 		baselines[i] = baseline{
 			source:   common.NewStringSource(input, testCaseName),
 			expected: &expr.ParsedExpr{},
 		}
-		resultOrError := strings.Split(output, DiagnosticsDelimiter)
+		resultOrError := strings.Split(output, strings.ReplaceAll(DiagnosticsDelimiter, "\n", lineSeparator))
 		result := resultOrError[0]
 		if result != NoResult {
 			if err := pb.UnmarshalText(output, baselines[i].expected); err != nil {

diff --git a/go/v1/api/grafeas.go b/go/v1/api/grafeas.go
@@ -26,7 +26,7 @@ import (
 
 const (
 	defaultPageSize = 20
-	maxPageSize     = 1000
+	maxPageSize     = 10000
 	maxBatchSize    = 1000
 
 	// NotesGet is the permission to get a note.

diff --git a/go/v1/api/grafeas_test.go b/go/v1/api/grafeas_test.go
@@ -375,6 +375,35 @@ func (s *fakeStorage) ListNoteOccurrences(ctx context.Context, pID, nID, filter,
 	return foundOccs, "", nil
 }
 
+// projectPermission binds a permission to a project.
+type projectPermission struct {
+	permission iam.Permission
+	projectID  string
+}
+
+// allowListAuth implements grafeas.Auth, denying all access checks except those
+// in the specified allowList.
+type allowListAuth struct {
+	allowList []projectPermission
+}
+
+func (a *allowListAuth) CheckAccessAndProject(ctx context.Context, projectID string, entityID string, p iam.Permission) error {
+	for _, allowed := range a.allowList {
+		if allowed.permission == p && allowed.projectID == projectID {
+			return nil
+		}
+	}
+	return status.Errorf(codes.PermissionDenied, "permission %q denied for %q or %q", p, projectID, entityID)
+}
+
+func (a *allowListAuth) EndUserID(ctx context.Context) (string, error) {
+	return "42", nil
+}
+
+func (a *allowListAuth) PurgePolicy(ctx context.Context, projectID string, entityID string, r iam.Resource) error {
+	return nil
+}
+
 type fakeAuth struct {
 	// Whether auth calls return an error to exercise err code paths.
 	authErr, endUserIDErr, purgeErr bool
@@ -427,9 +456,14 @@ func TestValidatePageSize(t *testing.T) {
 			wantPS: 500,
 		},
 		{
-			desc:   "page size of 1000 (the max allowed), valid",
-			ps:     1000,
-			wantPS: 1000,
+			desc:   "page size of 5000, valid",
+			ps:     5000,
+			wantPS: 5000,
+		},
+		{
+			desc:   "page size of 10000 (the max allowed), valid",
+			ps:     10000,
+			wantPS: 10000,
 		},
 	}
 
@@ -461,13 +495,13 @@ func TestValidatePageSizeErrors(t *testing.T) {
 			wantErrCode: codes.InvalidArgument,
 		},
 		{
-			desc:        "page size of 1001 (larger than max allowed), want error",
-			ps:          1001,
+			desc:        "page size of 10001 (larger than max allowed), want error",
+			ps:          10001,
 			wantErrCode: codes.InvalidArgument,
 		},
 		{
-			desc:        "page size of 5000 (larger than max allowed), want error",
-			ps:          5000,
+			desc:        "page size of 50000 (larger than max allowed), want error",
+			ps:          50000,
 			wantErrCode: codes.InvalidArgument,
 		},
 	}

diff --git a/go/v1/api/validators/build/build.go b/go/v1/api/validators/build/build.go
@@ -39,9 +39,13 @@ func ValidateNote(b *gpb.BuildNote) []error {
 func ValidateOccurrence(d *gpb.BuildOccurrence) []error {
 	errs := []error{}
 
-	if p := d.GetProvenance(); p == nil {
-		errs = append(errs, errors.New("provenance is required"))
-	} else {
+	p := d.GetProvenance()
+	i := d.GetIntotoProvenance()
+	s := d.GetIntotoStatement()
+	if p == nil && i == nil && s == nil {
+		errs = append(errs, errors.New("either Provenance or intotoProvenance or intotoStatement is required"))
+	}
+	if p != nil {
 		for _, err := range provenance.ValidateBuildProvenance(p) {
 			errs = append(errs, fmt.Errorf("provenance.%s", err))
 		}

diff --git a/go/v1/api/validators/build/build_test.go b/go/v1/api/validators/build/build_test.go
@@ -64,14 +64,30 @@ func TestValidateOccurrence(t *testing.T) {
 			wantErrs: true,
 		},
 		{
-			desc: "valid details, want success",
+			desc: "valid details with provenance, want success",
 			d: &gpb.BuildOccurrence{
 				Provenance: &gpb.BuildProvenance{
 					Id: "8c0b1847-f78b-4bf7-8b2e-38e1bb48b125",
 				},
 			},
 			wantErrs: false,
 		},
+		{
+			desc: "valid details with intotoprovenance, want success",
+			d: &gpb.BuildOccurrence{
+				IntotoProvenance: &gpb.InTotoProvenance{},
+			},
+			wantErrs: false,
+		},
+		{
+			desc: "valid details with intotostatement, want success",
+			d: &gpb.BuildOccurrence{
+				IntotoStatement: &gpb.InTotoStatement{
+					Type: "my_type",
+				},
+			},
+			wantErrs: false,
+		},
 	}
 
 	for _, tt := range tests {

diff --git a/go/v1/api/validators/grafeas/grafeas.go b/go/v1/api/validators/grafeas/grafeas.go
@@ -66,7 +66,7 @@ func ValidateNote(n *gpb.Note) error {
 
 	if d := n.GetDeployment(); d != nil {
 		for _, err := range deployment.ValidateNote(d) {
-			errs = append(errs, fmt.Errorf("deplyable.%s", err))
+			errs = append(errs, fmt.Errorf("deployable.%s", err))
 		}
 	}
 

diff --git a/go/v1beta1/api/grafeas_test.go b/go/v1beta1/api/grafeas_test.go
@@ -427,6 +427,35 @@ func (s *fakeStorage) GetVulnerabilityOccurrencesSummary(ctx context.Context, pr
 	}, nil
 }
 
+// projectPermission binds a permission to a project.
+type projectPermission struct {
+	permission iam.Permission
+	projectID  string
+}
+
+// allowListAuth implements grafeas.Auth, denying all access checks except those
+// in the specified allowList.
+type allowListAuth struct {
+	allowList []projectPermission
+}
+
+func (a *allowListAuth) CheckAccessAndProject(ctx context.Context, projectID string, entityID string, p iam.Permission) error {
+	for _, allowed := range a.allowList {
+		if allowed.permission == p && allowed.projectID == projectID {
+			return nil
+		}
+	}
+	return status.Errorf(codes.PermissionDenied, "permission %q denied for %q or %q", p, projectID, entityID)
+}
+
+func (a *allowListAuth) EndUserID(ctx context.Context) (string, error) {
+	return "42", nil
+}
+
+func (a *allowListAuth) PurgePolicy(ctx context.Context, projectID string, entityID string, r iam.Resource) error {
+	return nil
+}
+
 type fakeAuth struct {
 	// Whether auth calls return an error to exercise err code paths.
 	authErr, endUserIDErr, purgeErr bool

diff --git a/go/v1beta1/api/note_test.go b/go/v1beta1/api/note_test.go
@@ -671,8 +671,10 @@ func TestDeleteNote(t *testing.T) {
 	req := &gpb.DeleteNoteRequest{
 		Name: "projects/goog-vulnz/notes/CVE-UH-OH",
 	}
-	if _, err := g.DeleteNote(ctx, req); err != nil {
+	if n, err := g.DeleteNote(ctx, req); err != nil {
 		t.Errorf("Got err %v, want success", err)
+	} else if n == nil {
+		t.Error("expected response to not be nil")
 	}
 }
 

diff --git a/go/v1beta1/api/occurrence_test.go b/go/v1beta1/api/occurrence_test.go
@@ -652,8 +652,10 @@ func TestDeleteOccurrence(t *testing.T) {
 			req := &gpb.DeleteOccurrenceRequest{
 				Name: createdOcc.Name,
 			}
-			if _, err := g.DeleteOccurrence(ctx, req); err != nil {
+			if oc, err := g.DeleteOccurrence(ctx, req); err != nil {
 				t.Errorf("Got err %v, want success", err)
+			} else if oc == nil {
+				t.Error("expected response to not be nil")
 			}
 		})
 	}

diff --git a/go/v1beta1/api/validators/grafeas/grafeas.go b/go/v1beta1/api/validators/grafeas/grafeas.go
@@ -67,7 +67,7 @@ func ValidateNote(n *gpb.Note) error {
 
 	if d := n.GetDeployable(); d != nil {
 		for _, err := range deployment.ValidateDeployable(d) {
-			errs = append(errs, fmt.Errorf("deplyable.%s", err))
+			errs = append(errs, fmt.Errorf("deployable.%s", err))
 		}
 	}