Skip to content

feat: add ImageClassificationExecutor for vision fine-tuning#62

Merged
kaiitunnz merged 14 commits into
mainfrom
feat/image-classification-executor
Jun 10, 2026
Merged

feat: add ImageClassificationExecutor for vision fine-tuning#62
kaiitunnz merged 14 commits into
mainfrom
feat/image-classification-executor

Conversation

@J1shen

@J1shen J1shen commented May 31, 2026

Copy link
Copy Markdown
Contributor

Purpose

FlowMesh's training executors (SFT/LoRA/DPO/PPO) are all LLM fine-tuning. This PR adds a first-class image-classification training task type so vision models can be fine-tuned through the same workflow/submit path, instead of users wrapping their own trainer in an ssh container job.

Changes

  • src/shared/tasks/task_type.py — add IMAGE_CLASSIFICATION_TRAINING = "image_classification_training".
  • src/shared/tasks/specs/training.py — add ImageClassificationTrainingSpec{Strict,Template}; the strict spec requires model.source.identifier (validated, no silent default).
  • src/shared/tasks/specs/__init__.py, src/shared/tasks/envelope.py — export the spec and wire it into the taskType discriminator union (strict + template).
  • sdk/src/flowmesh/models/common.py — mirror the new enum member (kept in lockstep with the server enum, enforced by tests/sdk/test_schema_compat.py).
  • src/worker/executors/image_classification_executor.py — new ImageClassificationExecutor: AutoImageProcessor + AutoModelForImageClassification + HF Trainer. Loads the dataset via datasets, infers num_labels/id2label, trains, evaluates, saves final_model/. Honors training.optimizer (→ TrainingArguments.optim) and training.augmentation (h-flip + 4px-pad random crop on the train split), and returns the per-step train_losses. Reuses the existing checkpoint/artifact helpers (determine_resume_path, archive_model_dir, maybe_upload_artifacts, TrainingMixin._cleanup_local_artifacts).
  • src/worker/executors/__init__.py — register image_classification_training in EXECUTOR_REGISTRY / EXECUTOR_CLASS_NAMES / __all__.
  • src/worker/main.py — instantiate the new executor on worker startup: add image_classification_training to the executor build list and to _EXECUTORS_TO_WRAP (training executors run in an MPExecutor subprocess for GPU cleanup). Without this the worker registered the type but fell back to the transformers executor and rejected the spec.
  • src/server/task/models.py — add image_classification_training to TRAINING_TASK_TYPES so the server classifies/routes it as a training task (per review feedback).
  • docs/EXECUTORS.md — add the taskType row; relabel the existing training row as LLM fine-tuning.
  • examples/templates/image_classification_vit.yaml — runnable ViT-on-CIFAR-10 demo.
  • tests/worker/test_executor_registry.py, tests/shared/test_image_classification_spec.py — registry key, _EXECUTORS_TO_WRAP guard, envelope-resolution and required-model coverage.

Dependencies

  • pyproject.toml, uv.lock, src/server/requirements.txt, src/worker/requirements/requirements.txt — raise the aiohttp floor >=3.13.4>=3.14.0 (re-locked to 3.14.1) and regenerate the worker requirements. This is not a feature change: the strict server pip-audit job started failing on aiohttp 3.13.5 for GHSA-jg22-mg44-37j8 and GHSA-hg6j-4rv6-33pg (both fixed in 3.14.0), which blocked this branch's CI. Bundled here only to keep the pipeline green; happy to split it into its own PR if preferred.

Design

The executor mirrors SFTExecutor's structure (spec → dataset prep → Trainer → save → result) so it slots into the existing training conventions, and deliberately adds no new feature dependencies — it reuses the runtime-inference group (transformers/torch/datasets/Pillow). It is single-GPU for now: allow_multi_gpu is downgraded with a warning (same as LoRASFTExecutor); multi-GPU via a torchrun/DeepSpeed dist-entry mirroring sft_dist_entry.py is a follow-up. The model_arch-free, HF-identifier-driven design means any AutoModelForImageClassification checkpoint (ViT/ResNet/ConvNeXt/…) works without code changes.

Test Plan

  • uv run pre-commit run --all-files
  • uv run pytest tests/ --ignore=tests/worker/test_mp_executor_cleanup_gpu.py
  • uvx bandit==1.9.4 -c pyproject.toml -r src/
  • End-to-end through a real stack (no Docker worker image): brought up a local server + an in-venv GPU worker and drove 4 optimizer-selected variants as image_classification_training workflows on Fashion-MNIST (H100). Confirmed server validation → best_fit dispatch → executor run → result via the control plane; also ran the executor in-process for an AdamW-vs-SGD optimizer contrast.

Test Result

  • pre-commit: all hooks pass (gitleaks/isort/black/ruff/codespell/mypy).
  • pytest: 990 passed.
  • bandit: no issues.
  • pip-audit: passes with the aiohttp floor raised.
  • Stack run: all 4 variants dispatched to the worker and trained through FlowMesh (eval_accuracy/train_losses returned in the result payload); the in-process AdamW run converged (loss 3.16→1.16) while SGD at the same budget stayed flat (~3.0–3.4), confirming training.optimizer is honored.

Pre-submission Checklist
  • I have read the contribution guidelines.
  • I have run pre-commit run --all-files and fixed any issues.
  • I have added or updated tests covering my changes (if applicable).
  • I have verified that uv run pytest tests/ passes locally.
  • If I changed shared schemas or proto definitions, I have checked downstream compatibility across Server and Worker.
  • If I changed the SDK or CLI, I have verified the affected packages work (uv sync --all-packages --group ci --frozen).
  • If this is a breaking change, I have prefixed the PR title with [BREAKING] and described migration steps above.
  • I have updated documentation or config examples if user-facing behavior changed.

Add a fifth training executor alongside SFT/LoRA/DPO/PPO that fine-tunes
image classification models via AutoModelForImageClassification and the
Transformers Trainer. Register the image_classification taskType, add the
ImageClassificationSpec, wire it into the envelope discriminator and the
worker executor registry, and ship a ViT-on-CIFAR-10 demo template.

Single-GPU only for now; multi-GPU via torchrun/DeepSpeed is a follow-up
mirroring the SFT distributed entrypoint.

Signed-off-by: Junyi Shen <jyshen44@gmail.com>
@J1shen J1shen requested review from kaiitunnz and timzsu as code owners May 31, 2026 17:17
The SDK TaskType enum must stay in lockstep with the server enum
(enforced by tests/sdk/test_schema_compat.py); add the new member.

Signed-off-by: Junyi Shen <jyshen44@gmail.com>
@J1shen J1shen changed the title feat(worker): add ImageClassificationExecutor for vision fine-tuning feat: add ImageClassificationExecutor for vision fine-tuning May 31, 2026
…classification

Wire two previously-ignored training knobs into ImageClassificationExecutor
and expose the per-step training loss curve:

- training.optimizer maps onto TrainingArguments.optim (adam->adamw_torch,
  sgd->sgd, else passthrough)
- training.augmentation applies horizontal flip + 4px-pad random crop to the
  train split only
- ImageClassificationResult.train_losses carries the Trainer loss history

Document the new knobs in the ViT demo template.

Signed-off-by: Junyi Shen <jyshen44@gmail.com>

@timzsu timzsu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FlowMesh is intended to be a "service fabric for composable LLM workflows". Could you include some justification on whether you introduce the ViT training path instead of Omni LLM training (e.g. FlowGRPO training for Qwen-Image, etc.)?

In addition, the PR description looks like LLM generated and do not follow the template. Please fix :)

Comment thread docs/EXECUTORS.md Outdated
Comment thread src/worker/executors/image_classification_executor.py Outdated
J1shen added 3 commits June 1, 2026 12:27
…e model id

Address review feedback on the vision-training executor:

- Rename the taskType / enum / spec classes from image_classification to
  image_classification_training so it reads unambiguously as a training task
  alongside sft/lora_sft/dpo/ppo (server + SDK enums kept in sync).
- Require model.source.identifier: ImageClassificationTrainingSpecStrict
  rejects a missing model at validation, and the executor no longer falls
  back to a hard-coded ViT default.
- Add spec tests covering envelope resolution and the required-model rule.

Signed-off-by: Junyi Shen <jyshen44@gmail.com>
Refresh the supported-workloads summary to include image_classification_training
and add a task-types table covering training/inference plus the misc data and
utility tasks (data_profiling, data_retrieval, embedding, api, echo).

Signed-off-by: Junyi Shen <jyshen44@gmail.com>
…te from LLM tasks

Signed-off-by: Junyi Shen <jyshen44@gmail.com>
@J1shen J1shen requested a review from timzsu June 3, 2026 03:48

@timzsu timzsu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that there are a few places not following the code style guidelines. Could you take a look at the doc? I have left comments in places that I spotted.

By the way, in ./src/server/task/models.py, please extend TRAINING_TASK_TYPES to include the new executor class.

Comment thread README.md Outdated
Comment thread src/worker/executors/image_classification_executor.py Outdated
Comment thread src/worker/executors/image_classification_executor.py Outdated
Comment thread src/worker/executors/image_classification_executor.py Outdated
Comment thread src/worker/executors/image_classification_executor.py Outdated
Comment thread src/worker/executors/image_classification_executor.py Outdated
Comment thread src/worker/executors/image_classification_executor.py Outdated
J1shen added 2 commits June 8, 2026 14:11
- Register image_classification_training in server TRAINING_TASK_TYPES so
  the task is categorized as training.
- Resolve labels via an isinstance(ClassLabel) guard instead of getattr.
- Simplify max_samples / max_eval_samples handling with walrus assignment.
- Drop the hasattr check in accuracy metric (len works regardless).
- Remove dead local reassignments and the accompanying type: ignore.

Signed-off-by: Junyi Shen <jyshen44@gmail.com>
…g6j-4rv6-33pg

pip-audit (no-ignore server scan) flagged aiohttp 3.13.5 for two advisories
published upstream, both fixed in 3.14.0. Raise the floor, re-lock (resolves
3.14.1), and regenerate the worker requirements files.

Signed-off-by: Junyi Shen <jyshen44@gmail.com>
@J1shen J1shen requested a review from timzsu June 8, 2026 06:32
J1shen and others added 2 commits June 8, 2026 14:45
The worker builds its executor set from an explicit list in worker.main and
wraps training executors for subprocess GPU isolation; the new task type was
registered in EXECUTOR_REGISTRY but missing from both, so the worker fell back
to the transformers executor and rejected the spec. Add it to the build list
and _EXECUTORS_TO_WRAP, and guard the wiring with a test.

Signed-off-by: Junyi Shen <jyshen44@gmail.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
timzsu
timzsu previously approved these changes Jun 8, 2026

@timzsu timzsu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @kaiitunnz Can you take a look at this PR?

@kaiitunnz kaiitunnz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Several comments. PTAL.

Comment thread src/worker/executors/image_classification_executor.py Outdated
Comment thread src/shared/tasks/specs/training.py
Comment thread src/worker/executors/image_classification_executor.py
Comment thread src/worker/executors/image_classification_executor.py Outdated
Comment thread src/worker/executors/image_classification_executor.py Outdated
Comment thread src/worker/executors/image_classification_executor.py
Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>
- Rename ImageClassificationExecutor -> ImageClassificationTrainingExecutor
  (and the result model) so the name is unambiguously training, not inference.
- Add the _require_model validator to the template spec too (shared helper) so
  validate_workflow rejects a missing model.source.identifier early.
- Pass processing_class=processor to Trainer so resuming from an intermediate
  checkpoint restores the image processor.
- Keep eval_accuracy of 0.0 instead of coercing it to None.
- Copy spec.training before mutating allow_multi_gpu.
- Remap raw dataset labels to contiguous ids in the transform (identity for
  ClassLabel columns, label2id lookup otherwise) so _image_collate no longer
  fails on string labels.

Signed-off-by: Junyi Shen <jyshen44@gmail.com>
@J1shen J1shen requested a review from kaiitunnz June 9, 2026 15:53

@kaiitunnz kaiitunnz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@kaiitunnz kaiitunnz merged commit 4e50132 into main Jun 10, 2026
11 checks passed
@kaiitunnz kaiitunnz deleted the feat/image-classification-executor branch June 10, 2026 07:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants