etcdserver: request is too large

[chen-keinan]

It has been observed that in some cases the report produced by trivy-operator is hitting the default etcd request limit and fails.

The reason for report getting too big is due to amount of vulnerabilities (found in image) and it associated data stored in the report.

Workaround for this issue is to tune etcd request limit

There are three potential solution for this issue:

• Reduce the amount of vulnerability data ,fields , stored in the report. the report refer to Aqua AVD so any data that can be found in reference can be omitted from report Reduce the amount of vulnerability fields stored in the vulnerabilities report #442
• The report stored a lot of repetitive data , maybe we need to restructure the report to reduce the amount of repetitive data which will end out with smaller report Remove vulnerability report repetitive data #443
• use EndpointSlice , sharding, and split the report to smaller reports use EndpointSlice for splitting the vulnerability report to smaller reports #445

[chary1112004]

@chen-keinan we also see the issue in the log. Do you have any update to resolve this issue?

{"level":"error","ts":"2024-01-17T00:43:52Z","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-7c4d69b694","namespace":"trivy-system"},"namespace":"trivy-system","name":"scan-vulnerabilityreport-7c4d69b694","reconcileID":"7a07aa93-7c7e-4530-9c54-8d3f0d10f24b","error":"etcdserver: request is too large","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}

[chen-keinan]

not really , its not an easy one, the only workaround I can think of is increasing the the request limit in etcd

[chary1112004]

not really , its not an easy one, the only workaround I can think of is increasing the the request limit in etcd

For increasing the request limit in etcd, I think it only works for kubernetes installed self nodes, not kubernetes provided by cloud provider since master nodes are managed by cloud provider.

[AndriiBurenko]

@chen-keinan Hi! I have the same problem. We use AWS EKS and can't change request limit in etcd. Do you have any update to resolve this issue?

We use:
Trivy-Operator version (helm-chart): 0.19.1
Kubernetes version (use kubectl version): 1.28.0

[lindsaygrace]

@chen-keinan Its a breaking change, but perhaps there can be a Vulnerability resource that trivy-operator maintains for each unique discovered vulnerability. That VulnerabilityReport references and includes some key details. This way fields that take up a lot of space like the extra links are kept to their own resource.

[chen-keinan]

@lindsaygrace could be , another option is to compress the reported data and encode it (save it to crd body), but it will not be human readable.

[Orrimp]

not really , its not an easy one, the only workaround I can think of is increasing the the request limit in etcd

We have our own Kubernetes Cluster but increasing the request limit could create another bunch of problems. The increased lag would compromise the stability of the cluster. Therefore we need to decrease the amount of information in the the report. I like the idea of @lindsaygrace to store each found vulnerability seperataly. We have multiple identical CVEs in os and library (java)

[alanrichman]

@chen-keinan We have the Trivy Operator deployed on a handful of GKE clusters and are encountering this issue as well. Ideally we would love to see a way to persist the data outside of the cluster to avoid etcd size limits entirely, but we are open to any other suggestions for a workaround.

[cthtrifork]

@lindsaygrace could be , another option is to compress the reported data and encode it (save it to crd body), but it will not be human readable.

If there is a way to hook into the serializer, we could enable a config flag for this and just encode the data in base64 (encryption does not seem necessary). If you could point me in the right direction, I could take a look at it.