feat: add ImageClassificationExecutor for vision fine-tuning#62
Conversation
Add a fifth training executor alongside SFT/LoRA/DPO/PPO that fine-tunes image classification models via AutoModelForImageClassification and the Transformers Trainer. Register the image_classification taskType, add the ImageClassificationSpec, wire it into the envelope discriminator and the worker executor registry, and ship a ViT-on-CIFAR-10 demo template. Single-GPU only for now; multi-GPU via torchrun/DeepSpeed is a follow-up mirroring the SFT distributed entrypoint. Signed-off-by: Junyi Shen <jyshen44@gmail.com>
The SDK TaskType enum must stay in lockstep with the server enum (enforced by tests/sdk/test_schema_compat.py); add the new member. Signed-off-by: Junyi Shen <jyshen44@gmail.com>
…classification Wire two previously-ignored training knobs into ImageClassificationExecutor and expose the per-step training loss curve: - training.optimizer maps onto TrainingArguments.optim (adam->adamw_torch, sgd->sgd, else passthrough) - training.augmentation applies horizontal flip + 4px-pad random crop to the train split only - ImageClassificationResult.train_losses carries the Trainer loss history Document the new knobs in the ViT demo template. Signed-off-by: Junyi Shen <jyshen44@gmail.com>
timzsu
left a comment
There was a problem hiding this comment.
FlowMesh is intended to be a "service fabric for composable LLM workflows". Could you include some justification on whether you introduce the ViT training path instead of Omni LLM training (e.g. FlowGRPO training for Qwen-Image, etc.)?
In addition, the PR description looks like LLM generated and do not follow the template. Please fix :)
…e model id Address review feedback on the vision-training executor: - Rename the taskType / enum / spec classes from image_classification to image_classification_training so it reads unambiguously as a training task alongside sft/lora_sft/dpo/ppo (server + SDK enums kept in sync). - Require model.source.identifier: ImageClassificationTrainingSpecStrict rejects a missing model at validation, and the executor no longer falls back to a hard-coded ViT default. - Add spec tests covering envelope resolution and the required-model rule. Signed-off-by: Junyi Shen <jyshen44@gmail.com>
Refresh the supported-workloads summary to include image_classification_training and add a task-types table covering training/inference plus the misc data and utility tasks (data_profiling, data_retrieval, embedding, api, echo). Signed-off-by: Junyi Shen <jyshen44@gmail.com>
…te from LLM tasks Signed-off-by: Junyi Shen <jyshen44@gmail.com>
timzsu
left a comment
There was a problem hiding this comment.
It seems that there are a few places not following the code style guidelines. Could you take a look at the doc? I have left comments in places that I spotted.
By the way, in ./src/server/task/models.py, please extend TRAINING_TASK_TYPES to include the new executor class.
- Register image_classification_training in server TRAINING_TASK_TYPES so the task is categorized as training. - Resolve labels via an isinstance(ClassLabel) guard instead of getattr. - Simplify max_samples / max_eval_samples handling with walrus assignment. - Drop the hasattr check in accuracy metric (len works regardless). - Remove dead local reassignments and the accompanying type: ignore. Signed-off-by: Junyi Shen <jyshen44@gmail.com>
…g6j-4rv6-33pg pip-audit (no-ignore server scan) flagged aiohttp 3.13.5 for two advisories published upstream, both fixed in 3.14.0. Raise the floor, re-lock (resolves 3.14.1), and regenerate the worker requirements files. Signed-off-by: Junyi Shen <jyshen44@gmail.com>
The worker builds its executor set from an explicit list in worker.main and wraps training executors for subprocess GPU isolation; the new task type was registered in EXECUTOR_REGISTRY but missing from both, so the worker fell back to the transformers executor and rejected the spec. Add it to the build list and _EXECUTORS_TO_WRAP, and guard the wiring with a test. Signed-off-by: Junyi Shen <jyshen44@gmail.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
timzsu
left a comment
There was a problem hiding this comment.
LGTM. @kaiitunnz Can you take a look at this PR?
kaiitunnz
left a comment
There was a problem hiding this comment.
Several comments. PTAL.
Signed-off-by: Noppanat Wadlom <noppanat.wad@gmail.com>
- Rename ImageClassificationExecutor -> ImageClassificationTrainingExecutor (and the result model) so the name is unambiguously training, not inference. - Add the _require_model validator to the template spec too (shared helper) so validate_workflow rejects a missing model.source.identifier early. - Pass processing_class=processor to Trainer so resuming from an intermediate checkpoint restores the image processor. - Keep eval_accuracy of 0.0 instead of coercing it to None. - Copy spec.training before mutating allow_multi_gpu. - Remap raw dataset labels to contiguous ids in the transform (identity for ClassLabel columns, label2id lookup otherwise) so _image_collate no longer fails on string labels. Signed-off-by: Junyi Shen <jyshen44@gmail.com>
Purpose
FlowMesh's training executors (SFT/LoRA/DPO/PPO) are all LLM fine-tuning. This PR adds a first-class image-classification training task type so vision models can be fine-tuned through the same workflow/submit path, instead of users wrapping their own trainer in an
sshcontainer job.Changes
src/shared/tasks/task_type.py— addIMAGE_CLASSIFICATION_TRAINING = "image_classification_training".src/shared/tasks/specs/training.py— addImageClassificationTrainingSpec{Strict,Template}; the strict spec requiresmodel.source.identifier(validated, no silent default).src/shared/tasks/specs/__init__.py,src/shared/tasks/envelope.py— export the spec and wire it into thetaskTypediscriminator union (strict + template).sdk/src/flowmesh/models/common.py— mirror the new enum member (kept in lockstep with the server enum, enforced bytests/sdk/test_schema_compat.py).src/worker/executors/image_classification_executor.py— newImageClassificationExecutor:AutoImageProcessor+AutoModelForImageClassification+ HFTrainer. Loads the dataset viadatasets, infersnum_labels/id2label, trains, evaluates, savesfinal_model/. Honorstraining.optimizer(→TrainingArguments.optim) andtraining.augmentation(h-flip + 4px-pad random crop on the train split), and returns the per-steptrain_losses. Reuses the existing checkpoint/artifact helpers (determine_resume_path,archive_model_dir,maybe_upload_artifacts,TrainingMixin._cleanup_local_artifacts).src/worker/executors/__init__.py— registerimage_classification_traininginEXECUTOR_REGISTRY/EXECUTOR_CLASS_NAMES/__all__.src/worker/main.py— instantiate the new executor on worker startup: addimage_classification_trainingto the executor build list and to_EXECUTORS_TO_WRAP(training executors run in anMPExecutorsubprocess for GPU cleanup). Without this the worker registered the type but fell back to the transformers executor and rejected the spec.src/server/task/models.py— addimage_classification_trainingtoTRAINING_TASK_TYPESso the server classifies/routes it as a training task (per review feedback).docs/EXECUTORS.md— add the taskType row; relabel the existing training row as LLM fine-tuning.examples/templates/image_classification_vit.yaml— runnable ViT-on-CIFAR-10 demo.tests/worker/test_executor_registry.py,tests/shared/test_image_classification_spec.py— registry key,_EXECUTORS_TO_WRAPguard, envelope-resolution and required-model coverage.Dependencies
pyproject.toml,uv.lock,src/server/requirements.txt,src/worker/requirements/requirements.txt— raise theaiohttpfloor>=3.13.4→>=3.14.0(re-locked to 3.14.1) and regenerate the worker requirements. This is not a feature change: the strict serverpip-auditjob started failing onaiohttp 3.13.5for GHSA-jg22-mg44-37j8 and GHSA-hg6j-4rv6-33pg (both fixed in 3.14.0), which blocked this branch's CI. Bundled here only to keep the pipeline green; happy to split it into its own PR if preferred.Design
The executor mirrors
SFTExecutor's structure (spec → dataset prep → Trainer → save → result) so it slots into the existing training conventions, and deliberately adds no new feature dependencies — it reuses theruntime-inferencegroup (transformers/torch/datasets/Pillow). It is single-GPU for now:allow_multi_gpuis downgraded with a warning (same asLoRASFTExecutor); multi-GPU via atorchrun/DeepSpeed dist-entry mirroringsft_dist_entry.pyis a follow-up. Themodel_arch-free, HF-identifier-driven design means anyAutoModelForImageClassificationcheckpoint (ViT/ResNet/ConvNeXt/…) works without code changes.Test Plan
uv run pre-commit run --all-filesuv run pytest tests/ --ignore=tests/worker/test_mp_executor_cleanup_gpu.pyuvx bandit==1.9.4 -c pyproject.toml -r src/image_classification_trainingworkflows on Fashion-MNIST (H100). Confirmed server validation →best_fitdispatch → executor run → result via the control plane; also ran the executor in-process for an AdamW-vs-SGD optimizer contrast.Test Result
eval_accuracy/train_lossesreturned in the result payload); the in-process AdamW run converged (loss 3.16→1.16) while SGD at the same budget stayed flat (~3.0–3.4), confirmingtraining.optimizeris honored.Pre-submission Checklist
pre-commit run --all-filesand fixed any issues.uv run pytest tests/passes locally.uv sync --all-packages --group ci --frozen).[BREAKING]and described migration steps above.