[fan-out integration] add resiliency to firing mechanism

[migmartri]

Chainloop, currently has a mechanism to send attestation and artifact metadata to third party-integrations such an OCI registry (attestation) and Dependency-Track (CycloneDX SBOM).

It's implementation today happens in the service layer and has some drawbacks due to it's naive initial implementation using a go-routine

chainloop/app/controlplane/internal/service/attestation.go

Line 223 in e770fae

go func() {

• It does not have delivery guarantees.
• It happens in memory so container restarts or new rollouts could potentially drop the task.
• There is no receipt, ack of this fan-out execution of happening. Only looking at the logs (or sentry alerts) can give us some indication if a third-party integration fire has happened correctly. [fan-out integration] Record triggered integrations and results #123
• There is no way to handle an attestation that came in the past, a.k.a replay
• The firing logic is coupled to the service layer.

Lucky for us those shortcomings can be easily overcame by using an event bus / queue mechanism such as nats.io streams.

We should design a new architecture that fixes the aforementioned shortcomings and sets the stage to a generic third-party integration framework #38