feat(referrers): persistence layer

[migmartri]

Persistence layer for #415. once we merge this code, newly received attestations will get denormalized and their materials and subjects will be stored in this new index. On a following patch, we'll add an API endpoint to read the stored data.

Note for reviewers: Most of the content in data/ent can be ignored, except data/ent/schema/referrer.go and the new migration file (data/ent/migrate/migration/20231107121730.sql).

Data model

It extends the data model by storing referrers, which are entities represented by their content digest + artifact type.

In addition to that table, two more are added. One to link to the organizations where these items have been present (the attestation that contained them was for) + dependencies (which are other referrers). The dependencies table is meant to map material/subject <-> attestation relationships.

Entrypoint

When an attestation is received, it gets unpacked and persisted.

Retrieve information

A method GetFromRoot(ctx context.Context, digest string) (*StoredReferrer, error) has been added and allows retrieving referrers + their dependencies by providing a digest.

Note: If more than one item with the same digest and different artifact type is found, we fail for now. Eventually, we'll ask the user to provide the artifact type to filter out.

See biz/referrer_[integration]_test.go to know more about how the creation and retrieval of the data happens.

[migmartri]

@jotadrilo, since you mentioned the OCI's referrer API. I am curious to get your take on this, especially on the broader picture #415

[buccarel]

The code looks good to me, but keep in mind that my high-level picture is not fully developed yet

[danlishka]

What instead of bool for downloadable we add CASMapping.ID? Would that bring any value?

[danlishka]

I think it would be good to add more info here about why we add the artifact type and

CREATE UNIQUE INDEX "referrer_digest_artifact_type" ON "referrers" ("digest", "artifact_type");

sha256 for a container image and for a container image manifest file can be the same.

[migmartri]

I think it would be good to add more info here about why we add the artifact type and

CREATE UNIQUE INDEX "referrer_digest_artifact_type" ON "referrers" ("digest", "artifact_type");

sha256 for a container image and for a container image manifest file can be the same.

makes sense, will do!

[migmartri]

What instead of bool for downloadable we add CASMapping.ID? Would that bring any value?

I do not think so, some reasons

• The relationship between referrer -> casMapping will need to be 1-N. We store in referrer while the cas-mapping can be many. So we'll need a joint table.
• Even if we do the joint table, the result of asking how to download a referrer will return a list of mappings, which can be easily retrieved by searching for digest in the mapping as we do today.
• In fact, searching for a cas-backend to download something (by digest), does more than just find the cas-backends but also if they are public. We will not have access to that functionality if we build the referrer <-> mapping relationship ourselves.

But I am curious on what you think, I might be missing something.

[danlishka]

LGTM! Thanks for the online review together