[SGLang Workflow] Upload benchmark results to AWS S3 #69
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
changed the title
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
temporarily deployed
to
pytorch-x-vllm
GitHub Actions
Inactive
namanlalitnyu
temporarily deployed
to
pytorch-x-vllm
GitHub Actions
Inactive
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
huydhn
reviewed
Aug 28, 2025
huydhn
approved these changes
Aug 28, 2025
huydhn
reviewed
Aug 28, 2025
| echo "⚠️ No benchmark results found in ${BENCHMARK_RESULTS}" >> $GITHUB_STEP_SUMMARY | ||
| fi | ||
| python3 .github/scripts/upload_benchmark_results.py \ |
There was a problem hiding this comment.
I notice that this workflow doesn't have a schedule yet. Are you planning to add one? I'm thinking daily, biweekly, or weekly depending on how frequent we need to look at sglang results.
There was a problem hiding this comment.
Thanks for pointing it out, I have added it for weekly as of now, as we discussed initially. SGLang releases the new stable versions every monthly, but their pre-release versions mostly happen every week, so I think it would be fine to have it weekly for now. But, in future, we can update it to lesser/more as well based on the feedback.
There was a problem hiding this comment.
I think weekly is a good starting point
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
commented
Aug 28, 2025
namanlalitnyu
commented
Aug 28, 2025
| "dataset_path": "./ShareGPT_V3_unfiltered_cleaned_split.json", | ||
| "num_prompts": 200 | ||
| } | ||
| }, |
There was a problem hiding this comment.
Added a couple of tests for having more metrics on the SGLang dashboard.
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Error
namanlalitnyu
had a problem deploying
to
pytorch-x-vllm
GitHub Actions
Failure
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These changes upload the SGLang benchmark results to the AWS S3 bucket, which internally triggers the AWS Lambda function to upload them to the clickhouse database. Eventually, being rendered in the HUD Dashboard.
For SGLang benchmarking, currently, the
vllm bench servecommand was generating atest.pytorch.jsonfile containing the Pytorch formatted benchmark results, but it was hardcoding the benchmark name tovllm benchmark. Due to that, whenever the SGLang benchmarking was being run, it was also allocating the same benchmark name. This was leading to issues in the HUD dashboard where theclickhouse sql querieswere trying to find thebenchmark nameasSGLang benchmark, but due to this issue, the results were returning an empty array.So, added a check in the bash script to replace the benchmark name with correct one, as the original implementation is being done in the vllm code repository.
Testing:
Verified the changes from the Github workflow that the results are getting uploaded correctly to AWS S3Verified the changes from the flambeau dashboard that the results are getting uploaded to Clickhouse database.