Repository navigation
OpenJet 0.5.3: parallel coding sessions on one GPU
OpenJet 0.5.3: parallel coding sessions on one GPU
OpenJet's CUDA engine can now serve several coding agents at once from a single GPU. The Qwen3.8-27B Q4_K_M path supports up to eight concurrent conversations, with a shared KV cache and reuse of common system and tool prefixes.
On one RTX 3090, four real coding agents achieved 150–200 output tokens/sec in aggregate over elapsed run time, including time spent on tool calls in the maintainer's runs. Short-context generation measurements reach approximately 290 tokens/sec in aggregate. These are different workloads: the real-agent figure includes tool execution time, while the short-context figure measures generation. Results vary with context, prompts and how much time each agent spends using tools.
Parallel serving
Start the Q4_K_M engine with four conversation slots:
python3 megakernel/server/mk_server.py \
-m /path/to/Qwen3.8-27B-Q4_K_M.gguf --agents 4Connect your agents to http://127.0.0.1:18080/v1 using the OpenAI-compatible chat API. See the source installation and four-agent setup.
Active sessions decode together. Incoming prompts can be processed while other sessions continue decoding. Sessions share one paged q8_0 KV pool; the documented 24 GB RTX 3090 configuration holds about 132K context positions in total. A common system prompt and tool list can be shared between sessions.
Live session monitor
mk_top shows each session's task, output, context use and generation rate alongside GPU memory, power and utilisation. It reads the server's new GET /slots endpoint.
From a current source checkout, with the server and agents running:
python3 megakernel/tools/mk_top.py --url http://127.0.0.1:18080Watch the video in the README.
Memory handling
The engine releases the mapped GGUF after loading the weights. Under WSL, the server also drops the file from the page cache, reducing memory that would otherwise remain counted against Windows.
Compatibility
- Parallel serving currently requires Qwen3.8-27B Q4_K_M and a compatible GPU. The documented RTX 30/40/50-series path requires Linux, at least 24 GB VRAM and the CUDA toolkit.
- The mixed-quant engine remains single-conversation and rejects parallel mode before loading.
- Context memory is shared across sessions; 132K is the total for the documented 3090 configuration.
- Performance figures above are maintainer-reported RTX 3090 results. They do not establish the same speed on other GPUs or a hosted-service performance guarantee.
Install or upgrade the package:
pipx install open-jet
# Existing installation:
pipx upgrade open-jetFollow the linked source instructions for the parallel setup. The engine README contains the API, benchmark methods and full limitations.