Mortred AI Model Server is a toy web server for deep learning models. Server tries its best to make the most usage of your cpu and gpu resources. All dl models are trained by tensorflow/pytorch and deployed via MNN toolkit and supply web service through workflow framework finally.

Do not hesitate to let me know if you find bugs here cause I'm a c-with-struct noob 🙃

The three major components are illustrated on the architecture picture below.

A quick overview and examples for both serving and model benchmarking are provided below. Detailed documentation and examples will be provided in the docs folder.

You're welcomed to ask questions and help me to make it better!

All models and detectors can be downloaded from my Hugging Face Page.

• Quick Start
• Benchmark
• Tutorials
• How To
• Web Server Configuration
• HTTP API Contract

Linux is the only supported platform. Two deployment profiles exist and one switch drives everything (build, dependencies, model catalog, weight subset):

	gpu (default)	cpu
backends	MNN-CUDA / ORT-CUDA / TensorRT	MNN-CPU / ORT-CPU
hardware	NVIDIA GPU + CUDA 11/12	any x64 machine
models	full zoo	curated set (mobilenetv2, resnet50, yolov8, hrnet)

Three entries, one core (mortredctl): pick whichever fits; they all end at the same mortredctl doctor acceptance gate.

curl -fsSL https://raw.githubusercontent.com/MaybeSheewill-CV/mortred_model_server/main/scripts/bootstrap.sh | bash

Detects your hardware (NVIDIA GPU → gpu, otherwise cpu), then delegates to the docker track (if docker is present) or downloads the latest release tarball and runs its installer.

git clone https://github.com/MaybeSheewill-CV/mortred_model_server.git
cd mortred_model_server
python3 scripts/fetch_weights.py --profile cpu        # or: gpu
MORTRED_API_TOKEN=<mgmt-token> MORTRED_GATEWAY_AUTH_TOKEN=<infer-token> \
    docker compose --profile cpu up -d                # or: --profile gpu
curl -fs http://localhost:8787/api/v1/health

Download mortred_model_server-<version>-<profile>-linux-x64.tar.gz from Releases, verify its .sha256, then:

tar -xzf mortred_model_server-*-linux-x64.tar.gz && cd mortred_model_server-*-linux-x64
sudo ./install.sh                                  # runtime deps + /opt/mortred + systemd
sudoedit /etc/mortred/supervisor.env               # set both tokens
cd /opt/mortred && python3 scripts/fetch_weights.py --profile cpu
sudo systemctl start mortred-supervisor

mortredctl init [--profile cpu|gpu]   # detect hw, fetch weight subset, verify
mortredctl doctor                     # live acceptance (healthz + auth probes)
mortredctl status | catalog           # runtime introspection

GPU note: TensorRT engines are per-machine artifacts; convert missing ones with scripts/convert_trt_engines.sh, or start the container with -e MORTRED_AUTO_BUILD_ENGINES=true to convert before autostart.

# dependencies (version matrix + sha256 pinned + idempotent stamps)
./scripts/install_deps.sh --all              # gpu line (CUDA 11 default)
./scripts/install_deps.sh --cpu --all        # cpu line (no NVIDIA/TRT at all)

# configure + build (presets carry the profile)
cmake --preset full && cmake --build --preset full            # gpu
cmake --preset full-cpu && cmake --build --preset full-cpu    # cpu

# verify
./scripts/verify_deployment.sh --basic

Unit tests only (no engines needed; system packages or vcpkg):

cmake --preset tests-only && cmake --build --preset tests-only && ctest --preset tests-only

The complete operations manual - architecture diagrams, per-track walkthroughs, security checklist, upgrades, troubleshooting - lives in docs/deployment.md / 中文版.

The benchmark test environment is as follows：

OS: Ubuntu 20.04.5 LTS / 5.15.0-87-generic

MEMORY: 32G DIMM DDR4 Synchronous 2666 MHz

CPU: Intel(R) Core(TM) i5-10400 CPU @ 2.90GHz

GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0

GPU: GeForce RTX 3080

CUDA: CUDA Version: 11.5

GPU Driver: Driver Version: 495.29.05

All models loop several times to avoid the influence of gpu's warmup and only model's inference time has been counted.

Benchmark Code Snappit

• Details Of Model Inference Benchmark
• About Model Configuration

• Image Classification Model Server Tutorials
• Image Segmentation Model Server Tutorials
• Image Object Detection Model Server Tutorials
• Image Enhancement Model Server Tutorials
• Image Feature Point Model Server Tutorials

• How To Add New Model 🔥🔥
• How To Add New Server 🔥🔥

• Description About Model Server
• HTTP API Contract (topology, auth, status mapping, overload behaviour)

Build and install all third-party dependencies (MNN / WORKFLOW / ONNXRUNTIME / TensorRT / CUDA / fmt / header-only libs) into 3rd_party/{include,libs} with a single script — no manual compilation or copying:

./scripts/install_deps.sh --all     # build/install everything (CUDA 11 baseline)
./scripts/install_deps.sh --check   # verify integrity and print versions
./scripts/install_deps.sh --cuda-version 12   # switch to the CUDA 12 / TRT 10 line

docker build -t mortred_model_server .
docker run --gpus all -p 8080:8080 -p 8787:8787 \
  -v $PWD/weights:/opt/mortred/weights \
  -e MORTRED_GATEWAY_AUTH_TOKEN=your-inference-token \
  -e MORTRED_API_TOKEN=your-management-token \
  mortred_model_server
# or: docker compose up -d   (see docker-compose.yml)

The image builds all deps + the full project, runs the unit/e2e tests, and ships the control plane. In-container topology: mortred-supervisor (management :8787, embedded web UI + REST API) supervises mortred-gateway (data plane :8080, the single inference entry) and all model servers; model processes bind loopback only and are no longer exposed port by port. External exposure must terminate TLS at a reverse proxy.

Prebuilt engines may mismatch your GPU architecture / TRT version. Regenerate them from the ONNX sources for this machine. Conversion uses the external trtexec CLI (TensorRT official tool): sudo ./scripts/install_deps.sh --nvidia installs it into 3rd_party/bin/, or point to your system TensorRT copy with --trtexec /path/to/trtexec:

./scripts/convert_trt_engines.sh --list    # show the manifest (19 engines)
./scripts/convert_trt_engines.sh           # convert missing engines (FP16 + dynamic profiles)
./scripts/convert_trt_engines.sh --force   # rebuild everything

The script detects the local TensorRT major version and emits the matching workspace flag. Use --trtexec when multiple TensorRT installations coexist.

• Add more model into model zoo

mortred_model_server refers to the following projects:

• https://github.com/sogou/workflow
• https://github.com/alibaba/MNN
• https://github.com/PaddlePaddle/PaddleSeg
• https://github.com/Tencent/rapidjson
• https://github.com/ToruNiina/toml11