Skip to content

feat(vllm_performance): promote geospatial experiments to stable and add endpoint experiment with custom dataset#744

Merged
christian-pinto merged 10 commits into
mainfrom
cp_vllm_perf_geo_updates
Apr 9, 2026
Merged

feat(vllm_performance): promote geospatial experiments to stable and add endpoint experiment with custom dataset#744
christian-pinto merged 10 commits into
mainfrom
cp_vllm_perf_geo_updates

Conversation

@christian-pinto
Copy link
Copy Markdown
Member

@christian-pinto christian-pinto commented Mar 25, 2026

No description provided.

christian-pinto
christian-pinto commented Mar 25, 2026
View reviewed changes
Comment thread ...s/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/execute_benchmark.py Outdated
Comment thread ...s/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/execute_benchmark.py Outdated
@DRL-NextGen
Copy link
Copy Markdown
Member

DRL-NextGen commented Mar 25, 2026 •
edited
Loading

No vulnerabilities found.

@AlessandroPomponio AlessandroPomponio changed the title Update the vllm-performance actuator to run experiments with geospatial models feat(vllm_performance): add experiment to test geospatial endpoint with a custom dataset Mar 27, 2026
@michael-johnston
Copy link
Copy Markdown
Member

michael-johnston commented Mar 30, 2026

@christian-pinto FYI plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiment_executor.py need black run on it.

@christian-pinto
Copy link
Copy Markdown
Member Author

christian-pinto commented Mar 31, 2026

@christian-pinto FYI plugins/actuators/vllm_performance/ado_actuators/vllm_performance/experiment_executor.py need black run on it.

Weird, the pre-commit hooks had all passed when I pushed those changes. I will give it another pass as there are more changes coming.

@christian-pinto christian-pinto changed the title feat(vllm_performance): add experiment to test geospatial endpoint with a custom dataset feat(vllm_performance): Add experiment to test geospatial endpoint with a custom dataset and remove geospatial experiments from BETA features Mar 31, 2026
@christian-pinto christian-pinto force-pushed the cp_vllm_perf_geo_updates branch from 4a14ce3 to 9705998 Compare March 31, 2026 09:53
@michael-johnston
Copy link
Copy Markdown
Member

michael-johnston commented Apr 2, 2026

Ready for review?

@christian-pinto
Copy link
Copy Markdown
Member Author

christian-pinto commented Apr 2, 2026

Ready for review?

Waiting for @mgazz to confirm he could replicate with guidellm on his side

@mgazz
Copy link
Copy Markdown
Member

mgazz commented Apr 3, 2026

Works for me. For context, the only thing I had to do to switch between vllm bench and guidellm was to swap the experiment section in the discovery space

from

experiments:
  - actuatorIdentifier: "vllm_performance"
    experimentIdentifier: "test-geospatial-deployment-custom-dataset-v1"

to

experiments:
  - actuatorIdentifier: "vllm_performance"
    experimentIdentifier: "test-geospatial-deployment-guidellm-custom-dataset-v1"

Full discovery space definition: discoveryspace_geospatial.yaml

@christian-pinto
Copy link
Copy Markdown
Member Author

christian-pinto commented Apr 7, 2026

@michael-johnston ready for review.

@DRL-NextGen
Copy link
Copy Markdown
Member

DRL-NextGen commented Apr 7, 2026 •
edited
Loading

No vulnerabilities found.

@AlessandroPomponio AlessandroPomponio modified the milestone: 1.8.0 Apr 8, 2026
@AlessandroPomponio AlessandroPomponio added enhancement New feature or request ci Enables CI integration labels Apr 8, 2026
AlessandroPomponio
AlessandroPomponio requested changes Apr 8, 2026
View reviewed changes
Comment thread ...rformance/ado_actuators/vllm_performance/vllm_performance_test/execute_guidellm_benchmark.py Outdated
Comment thread ...s/vllm_performance/ado_actuators/vllm_performance/vllm_performance_test/execute_benchmark.py Outdated
Comment thread website/docs/examples/vllm-performance-geospatial.md
@AlessandroPomponio AlessandroPomponio changed the title feat(vllm_performance): Add experiment to test geospatial endpoint with a custom dataset and remove geospatial experiments from BETA features feat(vllm_performance): promote geospatial experiments to stable and add endpoint experiment with custom dataset Apr 8, 2026
michael-johnston
michael-johnston reviewed Apr 8, 2026
View reviewed changes
Comment thread website/docs/examples/vllm-performance-geospatial.md Outdated
Comment thread website/docs/examples/vllm-performance-geospatial.md Outdated
@DRL-NextGen
Copy link
Copy Markdown
Member

DRL-NextGen commented Apr 9, 2026 •
edited
Loading

No vulnerabilities found.

AlessandroPomponio
AlessandroPomponio reviewed Apr 9, 2026
View reviewed changes
Comment thread website/docs/examples/vllm-performance-geospatial.md
Comment thread website/docs/examples/vllm-performance-geospatial.md Outdated
Comment thread website/docs/examples/vllm-performance-geospatial.md
Comment thread website/docs/examples/vllm-performance-geospatial.md
michael-johnston
michael-johnston reviewed Apr 9, 2026
View reviewed changes
Comment thread website/docs/actuators/vllm_performance.md
christian-pinto and others added 4 commits April 9, 2026 14:15
@christian-pinto
feat(vllm_performance): Added performance_testing-geospatial-endpoint…
1ddd1da 
…-custom-dataset experiment

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@mgazz @christian-pinto
fix(vllm_performance): set correct experiment identifiers and propert…
72e11f8 
…y configuration to test existing vllm geospatial endpoint

Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>
@mgazz @christian-pinto
fix(vllm_performance): update default values to benchmark geospatial …
457a2a9 
…models

Signed-off-by: Michele Gazzetti <michele.gazzetti1@ibm.com>
@christian-pinto
feat(vllm_performance): Upgraded custom arguments in execute_benchmark
0991e67 
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@christian-pinto
feat(vllm_performance): Removed BETA status from geospatial experimen…
12ab058 
…ts and added geospatial guidellm experiments

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@christian-pinto
fix(vllm_performance): Fixed ruff errors and re-added datasets as dep…
158a0c2 
…endency

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@christian-pinto
fix(vllm_performance): Fixed typo in terramind dataset JSONL file
14f7f3d 
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@christian-pinto
fix(vllm-perf): Addressed review comments
4133cdc 
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@christian-pinto
fix(vllm-perf): Addressed second review comments
60653b9 
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@christian-pinto
chore(vllm-perf): Added actuator configuration image_secret to legacy…
04c67b9 
… migrators

Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
@christian-pinto christian-pinto force-pushed the cp_vllm_perf_geo_updates branch from a17c42e to 04c67b9 Compare April 9, 2026 13:56
AlessandroPomponio
AlessandroPomponio approved these changes Apr 9, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@AlessandroPomponio AlessandroPomponio left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM thanks

@christian-pinto christian-pinto added this pull request to the merge queue Apr 9, 2026
Merged via the queue into main with commit 73c9d31 Apr 9, 2026
19 checks passed
@christian-pinto christian-pinto deleted the cp_vllm_perf_geo_updates branch April 9, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@michael-johnston michael-johnston michael-johnston left review comments

@AlessandroPomponio AlessandroPomponio AlessandroPomponio approved these changes

Assignees

No one assigned

Labels

ci Enables CI integration enhancement New feature or request

Projects

None yet

Milestone

1.8.0

Development

Successfully merging this pull request may close these issues.

5 participants

@christian-pinto @DRL-NextGen @michael-johnston @mgazz @AlessandroPomponio