No more VulnerabilityReport + "illegal base64 data at input byte 4" or "cannot unmarshal number (...)" #942
Labels
kind/bug
Categorizes issue or PR as related to a bug.
target/kubernetes
Issues relating to kubernetes cluster scanning
Comments
chen-keinan
added
the
target/kubernetes
|
@ledroide thank you for sharing this issue (this line could cause the issue
Related #757 |
|
@chen-keinan Thanks for your answer. parametersTrying :
trivy-operator-7ddfd74ccd-fr8jr trivy-operator {"level":"error","ts":"2023-02-10T14:14:28Z","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-599465f897","namespace":"trivy-system"},"namespace":"trivy-system","name":"scan-vulnerabilityreport-599465f897","reconcileID":"392b6496-8757-47ce-92d9-37f8a7516d9d","error":"json: cannot unmarshal number into Go value of type trivy.ScanReport","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/internal/controller/controller.go:235"}more logsscan podsAs requested, here below, logs from various clusters for scan pods. Only the first one is set with Others are set with default values, including controller podFrom cluster where From another one where |
|
@ledroide thanks you for response with many examples , I need to see the output as operator query it from if you could run the following when |
|
@chen-keinan : Here is what you have asked for : |
|
@ledroide have you changed the there was a Breaking / Deprecation changes in API with trivy 0.37.x |
Yes, it was an attempt to try fixing this issue, but with no effect. |
thanks , it will be great to get (with trivy v0.36.0) kubectl logs -n trivy-system where compressLogs=true note: need to make sure that scan jobs from prev check are deleted before trying again with imageRef of trivy v0.36.0 |
|
@chen-keinan is the 0.37.x breaking issue you mentioned this one: aquasecurity/trivy#3467 If i grep -v WARN on that and pipe through base64 --decode it results in a bzip2 json file. |
@jrhunger true , that's why we didn't move to v0.37.0 yet in trivy-operator , it require code change, it need to replace |
Yes, Trivy-operator 0.11 with trivy 0.36.0, fix this error "error": "illegal base64 data at input byte 4". |
|
Related #948 |
3 tasks
|
@chen-keinan : You have found the root cause. Thanks a lot.
I have seen that #949 is merged, and will wait for the next trivy-operator tag and image build to retry with trivy 0.37.x. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Symptoms
After months running fine, with many VulnerabilityReports in all namespaces, we discovered that there was no more VulnerabilityReport in any namespace, for any of our k8s clusters.
$ kubectl get pod,job -l app.kubernetes.io/managed-by=trivy-operator,vulnerabilityReport.scanner=Trivy -n trivy-system NAME READY STATUS RESTARTS AGE pod/scan-vulnerabilityreport-67679699c8-ps88m 0/1 Completed 0 39h NAME COMPLETIONS DURATION AGE job.batch/scan-vulnerabilityreport-67679699c8 1/1 16s 39h $ $ kubectl logs -l app.kubernetes.io/managed-by=trivy-operator,vulnerabilityReport.scanner=Trivy -n trivy-system Defaulted container "node-cache" out of: node-cache, e0073c41-3c2f-49ef-a8b3-f84bb4cd143e (init) y48MKDTFL5nsozxSuwMguemrO+wcbBebJtsrYg2PgsN5CeH++zoinFT/H4EiOsiIx8uT3/xHtyN7 +SUWq1Lfg3nLBiiSRvmtI4jrUtUWuLC6z9/hh+9vpH0avTipY634PJQn0EBWyonPfZ8yc0ierBQj mRSEKqqkz5nYunMUqVD8RwsmionV59Ob/mbA1DvRef0pWyDvFsXxYRAu1eARqas+Lh2bnNzHXJI4 Yhg3XAuudMYTuhkWi0sui7B9tmWCmn7FtMLaLFlyEOwU7OJp05Yj5DyxosbRAhNgYSIdDs7jPEiE IdiaE5ooV6Z8+T0Qzllu5MuaZ2hhDNLwPDUXXvMbGbhApMgfCkwbNy42t9Vw23pyGqIWsNZ0YIOS QVQ8d49h9o65aDo4qJHUY7WpUEOouVn0k9SAW4GAX/DHArnccBG4OUxLQihayABAL6tg68gd2FUM mIKGEVDhptG4SDtH+NHgk4DpZPoOSCL3kT6al1iMfZP76h/Eb/3vBHpSJVcP9JTxSRR4I2VwtHod vS6HRSlSEbpO2+520/1PY8vK8XbMpH/B52fKPrGIE+VfN80L0Op7D5wVHbH8MSexaGbTgeofghvM z/Ygf7qAkRQB/8w+4/sf4h/xkH/r/0qhs2plkCMv/aQiTOQRv+wPWcA8GH/+LuSKcKEh4Z6WzA==illegal base64 data at input byte 4errors :{"level":"error","ts":"2023-02-10T08:12:40Z","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-67679699c8","namespace":"trivy-system"},"namespace":"trivy-system","name":"scan-vulnerabilityreport-67679699c8","reconcileID":"ee597f18-0f67-4c0e-bcf0-c322762002d1","error":"illegal base64 data at input byte 4","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/internal/controller/controller.go:235"}What we have tried
scanJob.compressLogs: "true"to"false"and restart deploy/trivy-operator -> no effect on VulnerabilityReports and jobs, however the error reported by trivy-operator is nowjson: cannot unmarshal number into Go value of type trivy.ScanReport- which had been reported by old trivy-operator - error: json: cannot unmarshal number into Go value of type trivy.ScanReport #206 fixed by fix: client server mode is not compatible with latest trivy #190 for trivy-operator 0.1.0 :{"level":"error","ts":"2023-02-10T08:55:22Z","msg":"Reconciler error","controller":"job","controllerGroup":"batch","controllerKind":"Job","Job":{"name":"scan-vulnerabilityreport-7c884d4d56","namespace":"trivy-system"},"namespace":"trivy-system","name":"scan-vulnerabilityreport-7c884d4d56","reconcileID":"d4cc7a3f-8d98-4f1e-b250-b4867574cb59","error":"json: cannot unmarshal number into Go value of type trivy.ScanReport","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.2/pkg/internal/controller/controller.go:235"}Environment
The text was updated successfully, but these errors were encountered: