Add workaround to increase available memory for e2e test

[yuntanghsu]

Recently e2e tests are failed due to the error: "Error: No space left on device".
Adding workaround to increase available memory at the beginning of e2e test.

Previously, we have 31G available memory when using sudo apt-get clean .

Adding sudo rm -rf /usr/share/dotnet, sudo rm -rf /opt/ghc, sudo rm -rf "/usr/local/share/boost", sudo rm -rf "$AGENT_TOOLSDIRECTORY" will increase the available memory to 41G

Signed-off-by: Yun-Tang Hsu hsuy@vmware.com

[yuntanghsu]

/theia-test-e2e

[yuntanghsu]

/theia-test-e2e

[wsquan171]

Thanks Yun-Tang for taking a look of this issue. This workaround should suffice for short term, but we probably need to revisit our e2e on GitHub workflow for the next release.

In our test-e2e-kind.sh, we have

FLOW_VISIBILITY_CMD=$(dirname $0)"/../../hack/generate-manifest.sh --ch-size 100Mi --ch-monitor-threshold 0.1"
FLOW_VISIBILITY_WITH_SPARK_CMD=$(dirname $0)"/../../hack/generate-manifest.sh --no-grafana --spark-operator"
FLOW_VISIBILITY_CH_ONLY_CMD=$(dirname $0)"/../../hack/generate-manifest.sh --no-grafana"

which means we only limited CH disk to 100M for the TestFlowVisibility case, but left it with 8GiB of disk claim for TestPolicyRecommendation, TestTheiaClickHouseStatusCommand and TestUpgrade. I assume disk allocation of 8 GiB for these cases contributed to disk exhuastion? Maybe we can also take a deeper look and see if we can also limit CH disk size for these cases. per linked issue, these runners only guarentee 10GiB of free disk space, which we have exceeded way too much (if it's still the case).

[yuntanghsu]

2022/10/07 00:28:24 Num nodes: 3
2022/10/07 00:28:24 Applying Antrea YAML
2022/10/07 00:29:55 Error when creating logs directory 'beforeTeardown.Oct07-00-29-55': mkdir beforeTeardown.Oct07-00-29-55: no space left on device
2022/10/07 00:29:55 Error when deploying Antrea: error when waiting for antrea-controller rollout to complete - rc: 1 - stdout: Waiting for deployment "antrea-controller" rollout to finish: 0 of 1 updated replicas are available... stderr: error: timed out waiting for the condition

The e2e test failed right after applying antrea YAML. So I think clickhouse might not be related to current issue? Not sure if it's a bug on action side? actions/runner-images#709 (similar error)

The other concern is, for the TestTheiaClickHouseStatusCommand e2e test, since it needs to test the insert-rate information, 100M is not enough for this test (not sure if the other two tests need 8GiB storage?)

[yuntanghsu]

/theia-test-e2e

[yanjunz97]

Thanks Yun-Tang for working on this. Regarding the ClickHouse question, I would like to mention that in TestPolicyRecommendation and TestTheiaClickHouseStatusCommand, CH is deployed in RAM instead of in disk. Thus the storage size should not have effect on the disk space. In TestUpgrade, CH is deployed with local PV, but the size allocated does not mean the size it actually uses. The size 8G is just an annotation, and the disk space it uses depends on the testing contents.

[yanjunz97]

LGTM as a workaround. I agree we may need to revisit our e2e test storage consumption in the future to see if we can reduce the space usage.