Intermittent download issues

[tim-sendible]

error: failed to download file, status code: 403 Forbidden
2021/10/25 13:13:31 failed to download file, status code: 403 Forbidden
2021/10/25 13:13:31 failed to download file, status code: 403 Forbidden
ARMO security scanner starting
[progress] Downloading/Loading policy definitions
kind: Framework, name: nsa, error: control 'nsa' not found

This is the v1.0.126 container running in eks 1.21. Running the command kubescape scan framework nsa

If I wait an hour or two, one run will be successful, and then subsequent runs will fail again.

[tim-sendible]

This 403 error occurs at the top of every command I perform within the container eg:

/usr/bin # kubescape download -h
error: failed to download file, status code: 403 Forbidden

[tim-sendible]

When it does work, the output is... odd:

This is running quay.io/armosec/kubescape:v1.0.126

$ kubescape scan framework nsa --submit false
Warning: You are not updated to the latest release: v1.0.126
ARMO security scanner starting
[progress] Downloading/Loading policy definitions
[success] Downloaded/Loaded policy
[Error] [lstat /proc/35/fd/9: no such file or directory]
empty list of workloads - no workloads found

[dwertent]

Hi @tim-sendible, this is the GitHub behavior since you reached the rate limit.
@slashben, @rotemamsa I think we should find an immediate solution for this.

Since you managed to scan successfully, the Kubescape caches the policies in ~/.kubescape/<policy name>.json.
Try running -

kubescape scan framework nsa --use-default

The use-default will flag Kubescape to load the policy from the cached file.

Regarding the docker image, you can scan using docker but you need to make sure the docker container has access to your cluster. Otherwise, you can scan with the docker container yaml files/GitHub repositories e.g.

docker run quay.io/armosec/kubescape scan framework nsa https://github.com/GoogleCloudPlatform/microservices-demo

You can also scan from within the cluster using the cronJob, we will publish an official documentation ASAP, but for now please download the CronJob yaml, and remove the ConfigMap and the --submit from the command.

[tim-sendible]

Thanks @dwertent. That cache won't work, because it doesn't exist. Once a Kubernetes pod has completed its run, it destroys itself. I think the only workaround I have at this time is to somehow save the cache policies in persistent storage for future pods to use, which is really not a very kubernetes-friendly approach.

Alternatively, is there a user-friendly option allowing me to change the policy download location? I could then try and mirror the policies locally to prevent github ratelimiting me.

I am already basing my work on your cronjob (without the configmap or the volume), and the above error is what I'm getting as a result.

[dwertent]

@tim-sendible I agree with you, I experience the same behavior and it's very important to find a good and solid workaround.

Alternatively, is there a user-friendly option allowing me to change the policy download location? I could then try and mirror the policies locally to prevent github ratelimiting me.

Yes, you can download the policies using the download command ->

kubescape download framework nsa --output /tmp/nsa.json

And run with use-from flag ->

kubescape scan framework nsa --use-from /tmp/nsa.json

[tim-sendible]

@dwertent ah of course. I had tried that, but was already being rate limited so didn't have any success.

So it seems a half-sensible workaround for me is to create my own container that has these files pre-downloaded. This can tide me over while you guys come up with a more solid solution that doesn't hit github's rate limiting. Thanks.

[dwertent]

@tim-sendible In the Dockerfile you can add the download comment so the policies will be stored in the docker image, just don't forget to run with the use-default flag.

For now, since you have issues downloading the policies, I published a new tag with the policies already downloaded :)
quay.io/armosec/kubescape:local.1.0.126.

Good luck.

[tim-sendible]

Thanks @dwertent. I have something that at least runs.

Struggling to get this to work with individually-defined controlls (eg kubescape scan control c-0037 --use-from /policies/nsa.json), and a global scan seems to be failing my ingress nginx for CVE-2021-25742 even though they are patched.. but these are not related to this issue, so I will raise others after some debugging, thanks.

[dwertent]

Struggling to get this to work with individually-defined controlls (eg kubescape scan control c-0037 --use-from /policies/nsa.json)

@tim-sendible

This is because you are running a single control policy but the input you are providing is a framework policy.
You can ether run kubescape scan framework nsa --use-from /policies/nsa.json or kubescape scan control c-0037 --use-from /policies/c-0037.json.

I see there is some confusion and I apologize. We constantly improve the command and documentation based on users feedback, so thank you for your feedback and please feel free to share any other feedback or ideas you have.

CVE-2021-25742 even though they are patched

Indeed this is a different issue, we are working to improve this control since it's a very tricky test.

[tim-sendible]

When I get time, I don't mind helping with the docs or the examples. I'm not a good developer, but I can write things down :)

[dwertent]

Well that sounds great since I'm the apposite 😉

[dwertent]

Kubescape downloads the release and does no longer use api.github so this is fixed :)