feat(referrers): shared index API

[migmartri]

Exposes a discovery API that returns the referrers from public workflows of allowed organizations.

In practice, an organization that has been added to the allowList can mark a workflow as public, and its referrers will be exposed through a shared public, index.

Configuration

The list of organizations that can opt to publish referrers in the shared index is defined via a static configuration sntry. This acts as a trusted publisher mechanism.

# example of enabling the index and an org
referrer_shared_index:
  enabled: true
  allowed_orgs:
    - 7ae5a35a-9f01-4711-b293-bd1d161c01a9 # practical_mayer

API changes

This public index is available both via gRPC and HTTP and DOES NOT require authentication

Behavior

For example, let's say we have an artifact associated with five attestations.

By calling the discover endpoint, you'll get all the attestations regardless of whether they belong to a private or a public workflow. They belong to one of your organizations.

$ chainloop discover -d sha256:ebe2e6ffab93f4b2e2743d3557bd094427ea059adc78cfe2cd330a72f61731e8 -k ARTIFACT   

{
   "digest": "sha256:ebe2e6ffab93f4b2e2743d3557bd094427ea059adc78cfe2cd330a72f61731e8",
   "kind": "ARTIFACT",
   "downloadable": true,
   "public": true,
   "createdAt": "2023-11-09T21:44:31.372575Z",
   "references": [
      {
         "digest": "sha256:119bdefae108a32a81e599e5cf6872c4e88c902cf5a55c58e82598a4c7fcef53",
         "kind": "ATTESTATION",
         "downloadable": true,
         "public": false,
         "createdAt": "2023-11-14T22:20:40.567136Z",
         "references": []
      },
      {
         "digest": "sha256:4b83ebba199499a1cbbd58517f5818d8e3fd56cb5cc244c26f4dd0729e2fccb2",
         "kind": "ATTESTATION",
         "downloadable": true,
         "public": true,
         "createdAt": "2023-11-14T22:45:57.238998Z",
         "references": []
      },
      {
         "digest": "sha256:d1b54d55787308f9ac3facde069afc7c00bf6897fb07d6109253546bec44f2b7",
         "kind": "ATTESTATION",
         "downloadable": true,
         "public": true,
         "createdAt": "2023-11-14T23:00:08.998459Z",
         "references": []
      },
      {
         "digest": "sha256:e4b6b708d089ed6a0061b6981cfaaaeff3fc2a95f2b370bc8e1450d95913c667",
         "kind": "ATTESTATION",
         "downloadable": true,
         "public": false,
         "createdAt": "2023-11-14T22:23:19.606271Z",
         "references": []
      },
      {
         "digest": "sha256:e6ea5db82aa1084f8dfa96799e0a401d996e214fd75921663d8a2f4c6a2687e3",
         "kind": "ATTESTATION",
         "downloadable": true,
         "public": false,
         "createdAt": "2023-11-14T22:22:45.154694Z",
         "references": []
      }
   ]
}

But there is a way to only get public referrers added to the shared index, by passing the --public flag.

chainloop discover -d sha256:ebe2e6ffab93f4b2e2743d3557bd094427ea059adc78cfe2cd330a72f61731e8 -k ARTIFACT  --public

{
   "digest": "sha256:ebe2e6ffab93f4b2e2743d3557bd094427ea059adc78cfe2cd330a72f61731e8",
   "kind": "ARTIFACT",
   "downloadable": true,
   "public": true,
   "createdAt": "2023-11-09T21:44:31.372575Z",
   "references": [
      {
         "digest": "sha256:4b83ebba199499a1cbbd58517f5818d8e3fd56cb5cc244c26f4dd0729e2fccb2",
         "kind": "ATTESTATION",
         "downloadable": true,
         "public": true,
         "createdAt": "2023-11-14T22:45:57.238998Z",
         "references": []
      },
      {
         "digest": "sha256:d1b54d55787308f9ac3facde069afc7c00bf6897fb07d6109253546bec44f2b7",
         "kind": "ATTESTATION",
         "downloadable": true,
         "public": true,
         "createdAt": "2023-11-14T23:00:08.998459Z",
         "references": []
      }
   ]

This one includes only the public attestations for this org.

HTTP endpoint

The HTTP endpoint is /discover/sha256:...

$ curl -s "http://localhost:8000/discover/sha256:4b83ebba199499a1cbbd58517f5818d8e3fd56cb5cc244c26f4dd0729e2fccb2"

{
  "result": {
    "digest": "sha256:4b83ebba199499a1cbbd58517f5818d8e3fd56cb5cc244c26f4dd0729e2fccb2",
    "kind": "ATTESTATION",
    "downloadable": true,
    "public": true,
    "references": [
      {
        "digest": "sha1:586b1ce3fd6d0f64a09aa0cd7a58470517d558c7",
        "kind": "GIT_HEAD_COMMIT",
        "downloadable": false,
        "public": true,
        "references": [],
        "createdAt": "2023-11-14T22:20:40.570963Z"
      },
      {
        "digest": "sha256:ebe2e6ffab93f4b2e2743d3557bd094427ea059adc78cfe2cd330a72f61731e8",
        "kind": "SBOM_CYCLONEDX_JSON",
        "downloadable": false,
        "public": true,
        "references": [],
        "createdAt": "2023-11-09T10:45:14.353328Z"
      },
      {
        "digest": "sha256:ebe2e6ffab93f4b2e2743d3557bd094427ea059adc78cfe2cd330a72f61731e8",
        "kind": "ARTIFACT",
        "downloadable": true,
        "public": true,
        "references": [],
        "createdAt": "2023-11-09T21:44:31.372575Z"
      },
      {
        "digest": "sha256:f1456d1cc4c943d3376716d3290ac4588f3f350f515543490b6930e4735ac043",
        "kind": "CONTAINER_IMAGE",
        "downloadable": false,
        "public": true,
        "references": [],
        "createdAt": "2023-11-09T21:44:31.377839Z"
      }
    ],
    "createdAt": "2023-11-14T22:45:57.238998Z"
  }
}

Other implementation changes

• I've dropped the need of having a referrer <-> organization table since now we are connected to workflows

Refs #435

[danlishka]

LGTM. Looking forward to it!

[buccarel]

it looks good to me.
just 1 comment about nomenclature but it's not a blocker, I can approve it

[buccarel]

Maybe here we have some nomenclature inconsistency?
xyzPrivate vs. xyzPublic everywhere, except here: the public has the "Public" suffix, the private it simply doesn't have suffix

[buccarel]

[very nit]

Maybe when the decl is this long, it could be done vertical?

func NewReferrerUseCase(
        repo ReferrerRepo,
        wfRepo WorkflowRepo,
        mRepo MembershipRepo,
        indexCfg *conf.ReferrerSharedIndex,
        l log.Logger,
) (
        *ReferrerUseCase,
        error,
) {
...
}

Not a big deal thogh.

[buccarel]

Never mind, it's merged already