[ci][llm] Deflake llm_batch_vllm:test_vllm_llama_lora model downloading behavior

[nrghosh]

Description

• Fix llm_batch_vllm:test_vllm_llama_lora model downloading
• Improve confusing log message re: model downloading behavior

Problem Model downloading locking issue was causing vLLM / engine init to fail with "Cannot find any model weights".

Cause: all download types (tokenizer, full model, etc) were sharing the
same file lock. When a tokenizer-only download acq the lock first and
ocmpleted, subsequent full model downloads were getting TimeoutError and
would wait for lock release -> then return without downloading (assuming
that the weights were cached).

This was observed inconsistently due to the model files being cached
(and test passing) in some cases.

Reproduction:

1. Clear cache with rm -rf ~/.cache/huggingface/hub/models--llama-3.2-216M-lora-dummy/
2. Run test with pytest -vs release/llm_tests/batch/test_batch_vllm.py::test_vllm_llama_lora
3. Observe failure to get waits + engine init failure: RuntimeError: Cannot find any model weights

Fix: use separate lock paths for different download types:

This way, tokenizer-only downloads completing can't take full model
downloads to skip directly fetching the model weights.

Testing (on Anyscale and Ray OSS)

1. Clear cache
2. Add fix
3. Run test - success

Also ran all llm_batch_vllm release tests on Anyscale and Ray OSS to confirm no regressions in model downloading behavior

Related issues

n/a - internal Ray CI

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[gemini-code-assist]

Code Review

This pull request effectively resolves a race condition in model downloading by introducing separate file locks for different download types. The use of a lock_suffix based on the download type is a clean and direct solution to the problem described. Additionally, the clarification in the log message for missing hash files is a welcome improvement, reducing potential confusion for users. The changes are well-implemented and address the issue correctly. I have one minor suggestion to improve maintainability.

[gemini-code-assist]

The logic to use different lock suffixes based on the download type is a great fix for the race condition. To improve maintainability and avoid magic strings, you could consider defining "-tokenizer", "-exclude-safetensors", and "-full" as module-level or class-level constants. This would make the code more robust and easier to read, especially if these suffixes are ever used elsewhere.