Skip to content

Conversation

@jeremiaswerner
Copy link
Collaborator

@jeremiaswerner jeremiaswerner commented Nov 14, 2025

Following changes have been performed:

  • rework the application code to use vLLM
  • use the Granite-4.0-Micro model from Huggingface
  • store the model on the input store as a model cache to speed up subsequent runs
  • support for concurrency on H100 to support larger models

As a result, this tutorial can now support high-throughput batch inferencing use cases with any LLM.

Rendered version:
https://github.com/IBM/CodeEngine/blob/4a58342856c3828b7477f67af9aa2c8338d41a47/serverless-fleets/tutorials/inferencing/README.md

Since I added 8000 recipes pls review at a per commit basis

@jeremiaswerner jeremiaswerner self-assigned this Nov 14, 2025
Copy link
Collaborator

@reggeenr reggeenr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jeremiaswerner jeremiaswerner merged commit b680aa8 into IBM:main Nov 19, 2025
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants