lkevinzc / inference-benchmark Public

forked from tensorchord/inference-benchmark

Notifications You must be signed in to change notification settings
Fork 0
Star 0

WIP Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)

0 stars 3 forks Branches Tags Activity

Notifications

Name		Name	Last commit message	Last commit date
Latest commit History 16 Commits
.github/workflows		.github/workflows
benchmark		benchmark
src		src
.gitignore		.gitignore
Dockerfile		Dockerfile
Makefile		Makefile
README.md		README.md
pyproject.toml		pyproject.toml

Repository files navigation

Inference Benchmark

Benchmark for machine learning model online serving.

Collected information

throughput
latency
- P99
- P90
- P50
CPU usage
memory usage
GPU utilization
GPU memory usage

Tasks

Frameworks

About

WIP Benchmark for machine learning model online serving (LLM, embedding, Stable-Diffusion, Whisper)

Report repository

Releases

No releases published

Packages

No packages published

Languages

Python 84.0%
Dockerfile 14.6%
Makefile 1.4%