feat(benchmark): Add vllm_performance benchmark for prithvi

[mgazz]

No description provided.

[DRL-NextGen]

Checks Summary

Last run: 2026-07-08T08:39:33.985Z

Mend Unified Agent vulnerability scan found 14 vulnerabilities:

Severity	Identifier	Package	Details	Fix
❗ Critical	CVE-2025-69872	diskcache-5.6.3-py3-none-any.whl	DiskCache (python-diskcache) through 5.6.3 uses Python pickle for serialization by default. An attac... DiskCache (python-diskcache) through 5.6.3 uses Python pickle for serialization by default. An attacker with write access to the cache directory can achieve arbitrary code execution when a victim application reads from the cache.	Not Available
❗ Critical	CVE-2026-48746	vllm-0.21.0-1-cp38-abi3-manylinux_2_24_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). From 0.3.0 until 0.22.0, a... vLLM is an inference and serving engine for large language models (LLMs). From 0.3.0 until 0.22.0, a vulnerability in ASGI web servers and starlette's trust on those web servers enables an authentication bypass of the OpenAI API AuthenticationMiddleware. It allows to use the API without providing the configured VLLM_API_KEY or --api-key. This vulnerability is fixed in 0.22.0.	Upgrade to version https://github.com/vllm-project/vllm.git - v0.22.0
🔺 High	CVE-2026-41523	vllm-0.21.0-1-cp38-abi3-manylinux_2_24_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). Prior to 0.22.0, an assert... vLLM is an inference and serving engine for large language models (LLMs). Prior to 0.22.0, an assert-based security check in vLLM's activation function loading allows any unauthenticated attacker to achieve arbitrary code execution on the server by publishing a malicious HuggingFace model, when vLLM runs in Python optimized mode (python -O or PYTHONOPTIMIZE=1). This vulnerability is fixed in 0.22.0.	Upgrade to version https://github.com/vllm-project/vllm.git - v0.22.0
🔺 High	CVE-2026-55574	vllm-0.21.0-1-cp38-abi3-manylinux_2_24_x86_64.whl	vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. Prior to 0.24.... vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. Prior to 0.24.0, the structured_outputs.regex API parameter passes a user-supplied regular expression string directly to the grammar compiler backends with no compilation timeout; in the xgrammar backend the string reaches the regex compiler with no guard, and in the outlines backend the validation step blocks structural issues such as lookarounds and backreferences but performs no complexity analysis, so a pattern with nested quantifiers passes all checks and causes exponential state-space expansion, allowing a single request containing an adversarial regex to hang an inference worker indefinitely and deny service. This issue is fixed in version 0.24.0.	Upgrade to version vllm - 0.24.0,vllm - 0.24.0,https://github.com/vllm-project/vllm.git - v0.24.0
🔺 High	CVE-2026-54234	vllm-0.21.0-1-cp38-abi3-manylinux_2_24_x86_64.whl	vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. Prior to 0.24.... vLLM is a high-throughput and memory-efficient inference and serving engine for LLMs. Prior to 0.24.0, a frontend-legal multi-request speculative decoding workload can cause the rejection sampler to produce a recovered token equal to the model vocabulary size boundary value, which is then converted to negative one when the engine selects the next live token for a request and is written back into the drafter's input ids; that out-of-vocabulary value is later consumed by the model's embedding and attention path and crashes the engine worker with a GPU device-side assertion. The same triggering request sequence is reachable through the public gRPC Generate and Abort endpoints, so a remote client that can send generation requests can crash the shared engine worker, aborting concurrent requests and causing a service-wide denial of service for other clients of the deployment until the worker is restarted. This issue is fixed in version 0.24.0.	Upgrade to version https://github.com/vllm-project/vllm.git - v0.24.0,vllm - 0.24.0,vllm - 0.24.0
🔷 Medium	CVE-2026-55514	vllm-0.21.0-1-cp38-abi3-manylinux_2_24_x86_64.whl	vLLM is a library for LLM inference and serving. From 0.12.0 to before 0.24.0, sending a pure prompt... vLLM is a library for LLM inference and serving. From 0.12.0 to before 0.24.0, sending a pure prompt embeds payload in a /v1/completions request with a model using M-RoPE causes EngineCore to fail an assertion and fatally crash, shutting down the entire server application. Any remote user who is authorized to make a /v1/completions request can make such a request and induce a crash. This issue is fixed in version 0.24.0.	Upgrade to version vllm - 0.24.0,https://github.com/vllm-project/vllm.git - v0.24.0,vllm - 0.24.0
🔷 Medium	CVE-2026-47155	vllm-0.21.0-1-cp38-abi3-manylinux_2_24_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). Prior to 0.22.0, vLLM's re... vLLM is an inference and serving engine for large language models (LLMs). Prior to 0.22.0, vLLM's revision pinning controls do not consistently apply to all artifacts loaded for a model. A deployment that supplies --revision or --code-revision can still load dynamic code, GGUF files, image processors, retrieval side weights, or same-repository subfolder weights/config from an unpinned/default revision. This is a supply-chain integrity issue for pinned vLLM deployments. Operators can believe they are serving a reviewed model revision while vLLM resolves behavior-affecting nested or sibling artifacts outside that reviewed revision. This vulnerability is fixed in 0.22.0.	Upgrade to version https://github.com/vllm-project/vllm.git - v0.22.0
🔷 Medium	CVE-2026-53923	vllm-0.21.0-1-cp38-abi3-manylinux_2_24_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). From 0.5.5 until 0.23.1rc0... vLLM is an inference and serving engine for large language models (LLMs). From 0.5.5 until 0.23.1rc0, integer truncation of tensor dimensions in vLLM's GGUF dequantize kernels (csrc/quantization/gguf/gguf_kernel.cu) causes partial tensor processing. The output tensor is allocated at full size via torch::empty (uninitialized memory), but the dequantize CUDA kernel processes only a truncated number of elements. The unfilled portion of the output tensor retains whatever was previously in GPU memory. In multi-tenant inference deployments, this residual GPU memory may contain tensor data from other users' inference requests, constituting information disclosure. This vulnerability is fixed in 0.23.1rc0.	Not Available
🔷 Medium	CVE-2026-54233	vllm-0.21.0-1-cp38-abi3-manylinux_2_24_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). Prior to 0.23.1rc0, vLLM's... vLLM is an inference and serving engine for large language models (LLMs). Prior to 0.23.1rc0, vLLM's /v1/audio/transcriptions endpoint limits compressed upload size but not decoded PCM output. A 25MB OPUS file expands to ~14.9GB of float32 PCM at decode time. This vulnerability is fixed in 0.23.1rc0.	Not Available
🔷 Medium	CVE-2026-12491	vllm-0.21.0-1-cp38-abi3-manylinux_2_24_x86_64.whl	A flaw was found in vLLM, an open-source library for large language model inference. This vulnerabil... A flaw was found in vLLM, an open-source library for large language model inference. This vulnerability arises from improper handling of image metadata, specifically EXIF orientation and PNG transparency (tRNS) data, during image processing. When images are converted to RGB, transparency information may be implicitly discarded or remapped, leading to unexpected rendering of transparent pixels and distortion of input content. This can result in the model misinterpreting image content, potentially affecting the integrity of processed data.	Not Available
🔷 Medium	CVE-2026-54236	vllm-0.21.0-1-cp38-abi3-manylinux_2_24_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). Prior to 0.23.1rc0, the fi... vLLM is an inference and serving engine for large language models (LLMs). Prior to 0.23.1rc0, the fix for CVE-2026-22778, which introduced a sanitize_message helper that strips object-repr memory addresses from error messages before they reach the client, is incomplete: several response paths echo str(exc) directly to clients without calling sanitize_message. The unsanitized sites include the Anthropic API router in vllm/entrypoints/anthropic/api_router.py (the POST /v1/messages and POST /v1/messages/count_tokens handlers), the Server-Sent Events streaming converter in vllm/entrypoints/anthropic/serving.py, and the realtime speech-to-text WebSocket in vllm/entrypoints/speech_to_text/realtime/connection.py. These paths catch the exception inside the route coroutine and construct the JSONResponse themselves, bypassing the sanitizing global FastAPI exception handler, and WebSocket frames do not traverse that handler chain at all. Using the same primitive as the parent issue, an unauthenticated attacker can send malformed image bytes through the Anthropic Messages API image content parts so that PIL.Image.open raises an UnidentifiedImageError whose message contains the BytesIO object repr, leaking the heap memory address verbatim in the error.message field of the response body. This vulnerability is fixed in 0.23.1rc0.	Not Available
🔷 Medium	CVE-2026-54235	vllm-0.21.0-1-cp38-abi3-manylinux_2_24_x86_64.whl	vLLM is an inference and serving engine for large language models (LLMs). Prior to 0.23.1rc0, ll tem... vLLM is an inference and serving engine for large language models (LLMs). Prior to 0.23.1rc0, ll temperature validation gates use comparison operators (<, >), which silently evaluate to False for NaN and for positive Infinity in Python's IEEE 754 float semantics. Both values pass every guard and propagate to GPU sampling kernels, where they produce undefined behavior or CUDA errors that can crash the inference worker. This vulnerability is fixed in 0.23.1rc0.	Not Available
🔷 Medium	CVE-2025-3000	torch-2.11.0-cp312-cp312-manylinux_2_28_x86_64.whl	A vulnerability classified as critical has been found in PyTorch 2.6.0. This affects the function to... A vulnerability classified as critical has been found in PyTorch 2.6.0. This affects the function torch.jit.script. The manipulation leads to memory corruption. It is possible to launch the attack on the local host. The exploit has been disclosed to the public and may be used.	Not Available
🔸 Low	CVE-2025-63396	torch-2.11.0-cp312-cp312-manylinux_2_28_x86_64.whl	An issue was discovered in PyTorch v2.5 and v2.7.1. Omission of profiler.stop() can cause torch.prof... An issue was discovered in PyTorch v2.5 and v2.7.1. Omission of profiler.stop() can cause torch.profiler.profile (PythonTracer) to crash or hang during finalization, leading to a Denial of Service (DoS).	Not Available

[christian-pinto]

@mgazz We have an issue currently blocking our CI. I will have tests and benchmarks test runs done as soon as we get the CI going again.

[christian-pinto]

@mgazz please merge with the latest master and then run

git restore --source origin/main -- uv.lock
uv lock

Then add the new uv.lock and commit. Make sure the pre-commit hooks are installed becuase they will re-generate the requirement files.

[christian-pinto]

/run_benchmarks

[DRL-NextGen]

✅ Benchmark Run (2026-07-03 13:01 UTC)

Status	Duration	Ray Job ID	Instance
✅	45s	raysubmit_eBX6vxZcWXP7rCCq	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

[DRL-NextGen]

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

[christian-pinto]

/run_benchmarks

[DRL-NextGen]

✅ Benchmark Run (2026-07-08 08:33 UTC)

Status	Duration	Ray Job ID	Instance
✅	-	raysubmit_kCSfr6uEANHdn6qW	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

[DRL-NextGen]

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here

[christian-pinto]

/run_benchmarks

[DRL-NextGen]

🔁 Benchmark Run (2026-07-08 08:47 UTC)

Status	Ray Job ID	Instance
🔄	raysubmit_7U4393QH8DGxgrWh	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

[christian-pinto]

/run_benchmarks

[DRL-NextGen]

🔁 Benchmark Run (2026-07-08 09:07 UTC)

Status	Ray Job ID	Instance
❌	-	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

[christian-pinto]

/run_benchmarks

[DRL-NextGen]

✅ Benchmark Run (2026-07-08 09:48 UTC)

Status	Duration	Ray Job ID	Instance
✅	175s	raysubmit_h8hjk6QypreGPMdE	packages/terratorch/models/prithvi/benchmark_instances/vllm_performance

[DRL-NextGen]

@mgazz @IBM/algorithm-nexus-maintainers All benchmark runs have completed, see summary here