[RFC]: FlowMesh OSS Pre-Release Roadmap (collecting ideas...)

[timzsu]

Motivation.

FlowMesh OSS is the foundation of LumidOS, and therefore, downstream systems like Lumilake OSS depend on it. This RFC outlines the development roadmap for FlowMesh between the OSS migration and the first release.

Proposed Change.

Required. (Essential for downstream applications like Lumilake.)

Architecture

• Fold lumilake.governance into FlowMesh — move lumilake.governance (MinIO + DuckDB lineage tracking) under src/server/governance/. feat: workflow trace via OTel spans + analyzer #3.
• Complete hook infrastructure — finalize the interfaces that Runmesh can augment. [RFC]: Split shared hook contracts from product-specific extension points #27.
• Template cleanup — some configurations like replicas are not used anywhere in the codebase. We need to clean them up so users won't be confused. [RFC]: Cleanup workflow templates.  #28.

CI/CD & Quality

• security check — [Feature]: Add security CI scanners (zizmor, gitleaks, bandit, pip-audit) #6.

Hardware Support

• aarch64 worker image — Add a worker image that supports ARM-based CPU (e.g., GB10). feat: support aarch64 stack and worker images #30

Pre-release

• Prepare packages (flowmesh[sdk|cli|hook]) for releasing on pip. refactor: prepare lightweight PyPI package publishing #25
• Prepare versioning and release cycles. chore: add PyPI release workflow #26

Optional.

Architecture

• Pydantic schema simplification — Use Pydantic schema across the repo so dict resolving is guarded by schema (e.g., the result schema for executors).
• Refactor SSH session and connection naming.
• Refactor omni executors to use the mixins.
• Refactor all executors to avoid silently capping data to a valid range. Discussed in fix: unblock PPO multi-GPU training #18.
• Data interface & graph template refactor — refactor src/worker/executors/mixins/data.py and src/worker/executors/utils/graph_templates.py to simplify the convoluted logic.
• rename GpuPlatformInfo.gpus -> GpuPlatformInfo.devices. Discussed in feat: support aarch64 stack and worker images #30. [BREAKING] refactor: rename GpuPlatformInfo.gpus to .devices #33

Operations

• Worker scratch auto-remove policy — Auto remove old data on RESULTS_DIR and scratch so long-lived worker don't fill disks.

Bugfix

• Fix SSH resource access — Currently, SSH containers can access all CPU and memory resources of the machine on which the target worker is running. We should limit the accessible resources based on the requested task spec/max accessible resources set by the worker’s config.

CC List.

@kaiitunnz @J1shen @tjluyao

[kaiitunnz]

We also have the following tasks to consider.

Pre-release

• Prepare packages (flowmesh[core|sdk|cli|hook]) for releasing on pip.

TBD

• Fix SSH resource access: Currently, SSH containers can access all CPU and memory resources of the machine on which the target worker is running. We should limit the accessible resources based on the requested task spec/max accessible resources set by the worker’s config.
• Refactor SSH session and connection naming.
• Implement executors’ result schemas.

The TBD list may be deferred to a future release.

[kaiitunnz]

Pre-release Bugs

The list below tracks bugs that must be fixed before release. Please update if you find any other issues.

• Workflow execution stuck when destination is local. The execution of a workflow whose destination is set to "local" (e.g., templates/dag_inference_example.yaml) is stuck on the downstream task with the "PENDING" status. The cause may be in the local path resolution in the dispatcher. Need further investigation. (Resolved in fix: resolve local destination hang and unreachable pigz in worker images #17)
• FlowMesh CLI has logout without login. FlowMesh CLI commands depend on the FlowMesh config file, which is previously only written by the login command, which has been removed before the initial commit. We need to totally remove the user management command group and reconsider config file initialization and modification flow. (Resolved in refactor: remove legacy CLI runtime helpers and fix GPU worker count #16)
• pigz is never reachable in shipped worker images. archive_model_dir falls back to single-threaded python tarfile because neither Dockerfile.cpu nor Dockerfile.cuda installs pigz, so the optional parallel-gzip branch (shutil.which("pigz") → None) is dead in default containers and every trainer pays the slow archive cost on multi-GB checkpoints. (Resolved in fix: resolve local destination hang and unreachable pigz in worker images #17)
• Multi-GPU workers not reachable via CLI. flowmesh stack worker up gpu --targets 0,1 creates two single-GPU workers (one per id), so the multi-GPU spawn paths in SFT/DPO/PPO and the cuda_devices: list[int] field already wired in the docker adapter only fire when callers go through --config / --config-raw. (Resolved in refactor: remove legacy CLI runtime helpers and fix GPU worker count #16)
• PPO multi-gpu training is broken. PPO training hangs halfway, as described in refactor: in-process torchrun + DeepSpeed launchers; fix multi-GPU SFT/DPO/PPO spawn #12. (Resolved in fix: unblock PPO multi-GPU training #18)
• Make the use of quotation marks consistent in all templates. They are now inconsistent as discussed in fix: unblock PPO multi-GPU training #18. refactor: clean up legacy template fields and surface remaining knobs #34
• Misleading CLI error message. flowmesh stack up worker gpu -t 1 raises "No such option: -t" while flowmesh stack up worker gpu leads to "Got unexpected extra arguments (worker gpu)". We should consider swapping the priority of the error reporting logic.

[timzsu]

This RFC is concluded as the first release is published.