Repository navigation
A full rewrite for UE 5.8. TensorFlow runs out of process in an embedded Python with a per-project venv, through PythonCLI 0.4.0 on CLISystem 1.4.0. Both plugins are bundled.
The refactor
1.x was last updated for UE 5.3. Its component was a Blueprint on top of MachineLearningRemote, which sent everything over socket.io to a separate Python server.
- Messages were capped at about 1 MB.
- Floats travelled as JSON text.
- Latent nodes hung when Python returned None or raised.
- The embedded Python 3.7 path was broken.
callEventand training-complete were never wired.
2.0 replaces the transport and keeps the API. It's now three layers, each its own plugin:
| Layer | Role |
|---|---|
| CLISystem 1.4 | child processes with framed stdin/stdout |
| PythonCLI 0.4 | the generic layer: an embedded CPython 3.12 and uv, a per-project venv, a shared warm Python process, and the components, streams and ML API |
| TensorFlow 2.0 | thin: environment profiles, the TensorFlow component, texture helpers, examples |
- Control messages go as framed stdio.
- Bulk data (render targets, audio, tensors) goes through shared memory.
Upgrading from 1.x
- Remove MachineLearningRemote-Unreal from your project; it's no longer used.
upymodule.jsonis gone. Pick an environment profile, or list packages in Python CLI > Requirements orContent/Scripts/requirements.txt.- Blueprint references to the old component load through a CoreRedirect, but its variables (Tensor Flow Module, Should Train On Begin Play) need setting again.
- 1.x's On Event is now On Custom Event.
Changes
- Python out of process: a crash in TensorFlow never takes the editor down, and the GIL never touches the game thread. MachineLearningRemote, socket.io and UnrealEnginePython are gone.
- Environment profiles (Project Settings > Plugins > TensorFlow):
- TensorFlow CPU 2.21 is the default.
- TensorFlow with CUDA, for Linux.
- Keras 3 on PyTorch, for GPU on Windows.
- None, to bring your own packages.
- Setup: the first editor start installs the profile into
Intermediate/PythonCLI/venvin the background. - Same Python API:
TFPluginAPIscripts with camelCase hooks,getApi(),callEvent,shouldStopandimport unreal_engine as uekeep working. snake_case hooks work too, andue.keras.StopOnRequest/ProgressEventsstop and report Keras training. - TensorFlow Component (now C++): Begin Training, Stop Training, Send Json Input, Send Json Input Texture, Call Custom Function. A CoreRedirect loads 1.x Blueprint references.
- Streams: render targets, audio and gameplay observations go to Python every frame, through shared memory without stalls.
- TF Audio Capture is rebuilt on Unreal's AudioCapture.
- Example map
/TensorFlow/Examples/MnistExample: trains MNIST, then classifies the digit a camera sees, live. - Packaging: Win64 and Linux builds bundle the interpreter and, by default, the installed packages. They're installed for the target platform, so a Linux build packaged on Windows works. Install On First Run and None are the other modes.
- Tests:
- 30 automation tests (
TensorFlow.*,PythonCLI.*) pass in the Win64 editor and in packaged Win64 builds, with Bundle and with Install On First Run. - In packaged Linux x86_64 builds (WSL2), 25 of them run and pass. The rest are skipped: the render-target tests need a GPU, the submix test needs an audio device, the requirements test doesn't apply to bundled packages, and the microphone test is opt-in.
- 30 automation tests (
Notes
- TF1 graph and session scripts need porting to TF2 / Keras 3.
- The last 1.x code is tagged
v1.6.0beta-legacy.