Reports aggregation in a separate process

[eddycharly]

This PR proposes to change the way we generate reports by generating per resource reports, let Kubernetes take care of reports lifecycle, and aggregate reports into higher level reports in a separate process/controller.

It is aligned with other KDPs meant to decompose Kyverno in multiple processes.

[chipzoller]

You've got some relics from another KDP you used as a template in here :)

[eddycharly]

You've got some relics from another KDP you used as a template in here :)

Fixed the one in the Metas section

[eddycharly]

Removed Prior art that leaked from the other KDP too.

[realshuting]

Index:
1.
2.
3.

[eddycharly]

This will appear correctly in md viewer.

[realshuting]

Is report here a report change request or a policy report?

[eddycharly]

I tested with RCR but could be a policy report as well.

[Boojapho]

@eddycharly If you align the reports and TTL with each resource, then it seems like we could just populate the report in the status of the resource itself. I would think that would simplify the architecture. Aggregation would occur by reading the resource status.

[eddycharly]

@Boojapho The status of analysed resources ?

[Boojapho]

@Boojapho The status of analysed resources ?

Just realized when you asked this that Kyverno doesn't own the status fields of the analyzed resources. So, this wouldn't work.

You could achieve a similar effect using annotations on the analyzed resource. But, that would require you to have write access to all resources, so that's not going to work either.

[eddycharly]

Yeah, basically we want a report, makes sense that it lives in it's own type.
I prefer not to bake that in the resource itself TBH.

[chipzoller]

It seems that creating a 1:1 mapping of resource:report could really put a squeeze on Kubernetes' data store (etcd). To me, it still seems like splitting reports per policy (and overflowing to more if the need arises) may be more efficient.

[eddycharly]

Splitting per policy has some advantages too but we won't be able to use k8s native garbage collection.
Let's try to list pro/cons of both approaches.

[realshuting]

Splitting per policy has some advantages too but we won't be able to use k8s native garbage collection. Let's try to list pro/cons of both approaches.

Initially I thought we were going to set ownerReference for RCRs (to solve the current RCR cleanup issue), and generate a report per namespace when we discussed offline.

[eddycharly]

Initially I thought we were going to set ownerReference for RCRs (to solve the current RCR cleanup issue)

That's what i have now. This doesn't make a real difference for @chipzoller point though, it's still a one:one mapping.

[realshuting]

Initially I thought we were going to set ownerReference for RCRs (to solve the current RCR cleanup issue)

That's what i have now. This doesn't make a real difference for @chipzoller point though, it's still a one:one mapping.

I meant to generate an RCR per resource, and merge RCRs to a single policy report for one namespace.

[eddycharly]

It seems that creating a 1:1 mapping of resource:report could really put a squeeze on Kubernetes' data store

True but I don't know if it's an issue, when managing a large cluster you usually size the control plane adequately.

[eddycharly]

I meant to generate an RCR per resource, and merge RCRs to a single policy report for one namespace.

Do you expect that we remove the RCR once merged into the report ?
My initial idea was to keep it for as long as the resource exists, hence creating a 1:1 mapping.

[realshuting]

I meant to generate an RCR per resource, and merge RCRs to a single policy report for one namespace.

Do you expect that we remove the RCR once merged into the report ? My initial idea was to keep it for as long as the resource exists, hence creating a 1:1 mapping.

Do you expect that we remove the RCR once merged into the report ?

No, per this proposal, the RCR shares the same lifecycle with the resource. Just that in addition to creating the RCR per resource, we will merge RCRs into one report per namespace. But this could increase the size by x2 as we duplicate the data.

[eddycharly]

But this could increase the size by x2 as we duplicate the data

Are you suggesting that we don't need to aggregate at the namespace level ? ;-)
In the end if we have the reports for all resources, why do we need to aggregate ?
We could just list reports instead of get, maybe just aggregate summaries ?

[Boojapho]

You should consider minimizing the Kubernetes API call load in the solution. The more you can aggregate in memory and then write to the report once, the less probability of hitting throttling issues.

[eddycharly]

You should consider minimizing the Kubernetes API call load in the solution

Definitely.

On the other hand, The solution we have now is not perfect, we are constantly creating/deleting RCRs instead of keeping them around and updating them when necessary.

[chipzoller]

This has already been completed, yes?