[Ported][CI/Build] Share HuggingFace downloads between test runs

[DarkLight1337]

The models tests keep getting interrupted (presumably due to running too long). This PR attempts to reduce the running time via:

• Share the HuggingFace cache between Kubernetes containers during CI by storing it in a hostPath volume.
• Disabling graph construction (considering that the vLLM model is only run once per test, not including the profile run).
• Reuse the HuggingFace cache between Docker containers via a shared volume.

[DarkLight1337]

Using eager mode doesn't seem to lead to significant improvement. It seems that the bottleneck is in downloading the models, so we should parallelize this process.

[DarkLight1337]

Tbh it is probably better if we have a way to avoid re-downloading the models each time. Any thoughts?

[DarkLight1337]

I'm not that experienced in Kubernetes but from my understanding, placing the HuggingFace cache inside a Volume should avoid the need to redownload the models when tests are run again in the same Pod.

@rkooo567 is it possible on your end to force the CI to run on the same Pod so we can test whether the cache actually works in this way?

[rkooo567]

Hmm I am not super familiar with how CI works actually (idk if we even use k8s under the hood). cc @simon-mo for thoughts..

[DarkLight1337]

@khluu since you're involved with CI, can you help out with this? Particularly the part concerning Kubernetes.

[simon-mo]

I believe we should download the model each time. @robertgshaw2-neuralmagic mentioned that putting them on NFS is a bit tricky because it might reaches rate limit.

[simon-mo]

hostPath a possible workaround.

[rkooo567]

Disabling graph construction (considering that the vLLM model is only run once per test, not including the profile run).

Hmm I am against this. Imo we should test the default config for those tests (especially the test_model)

[DarkLight1337]

I have updated this PR to work on AWS pipeline.

Looks like this shaved around 10 minutes off the duration of model tests. Going to rerun the test just to be sure.

[DarkLight1337]

Looks like this shaved around 10 minutes off the duration of model tests. Going to rerun the test just to be sure.

This doesn't seem to be the case anymore. It's hard to determine the real effect since the test runs aren't necessarily performed on the same machine (from my understanding).

[DarkLight1337]

Due to #5757, I have moved this PR to vllm-project/ci-infra#8.