fix: authenticate cross-worker image fetches to FlowMesh origin only - #115
Merged
Conversation
The bare http(s) image-item branch in DataMixin fetched images with an
unauthenticated requests.get. When a keyframe/image artifact is produced on
one worker and consumed on another that lacks a local copy (multi-node
topology), the ref resolves to the FlowMesh results URL
({base_url}/api/v1/results/{task_id}/files/{rel}); the unauthenticated GET
received a non-image body and failed with PIL.UnidentifiedImageError.
Route only FlowMesh-origin URLs through the auth-aware artifact path
(resolve_artifact + auth_headers), gated by a new is_flowmesh_origin_url()
helper that compares the URL's scheme+netloc against the worker's configured
FLOWMESH_BASE_URL. Arbitrary/public URLs keep the original unauthenticated
fetch with an explicit bounded 15s timeout, so the worker's bearer token is
never sent off-origin and a hung public host cannot stall a worker for the
resolver's 1800s default. S3, local-path, and dict-url branches are unchanged.
Adds regression tests: a FlowMesh-origin URL receives Authorization: Bearer
and decodes; an external URL receives no credential even when FLOWMESH_API_KEY
is set and uses the bounded timeout.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Zhengyuan Su <su.zhengyuan@u.nus.edu>
timzsu
marked this pull request as ready for review
July 27, 2026 13:51
8 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Fix a cross-worker image-fetch failure in VLM/image workflows on multi-node deployments (reported as
PIL.UnidentifiedImageError). When an image artifact is produced on one worker and consumed on a different worker without a local copy, the artifact resolves to a FlowMesh results URL that the image branch inDataMixinfetched without auth — so under enforced auth the endpoint returns a 401 non-image body and the fetch/decode fails. Co-located runs never hit this (the artifact resolves to a local path). The naive fix (auth on every image URL) would leak the worker's bearer token to arbitrary hosts, so this PR authenticates only the worker's own FlowMesh origin.Changes
artifacts.py— addis_flowmesh_origin_url(url): true only when the URL's scheme+host match the worker'sFLOWMESH_BASE_URL.data.py— split thehttp(s)image branch: FlowMesh-origin URLs go through the auth-aware_load_image_from_artifact; all other URLs keep the original unauthenticated fetch with a bounded 15s timeout.test_data_mixin_lineage.py— two regression tests: origin URL getsAuthorization: Bearer; external URL gets no credential (even withFLOWMESH_API_KEYset) and the 15s timeout.Design
Only the cross-worker results-URL fallback in
data.pywas not already auth-aware. It now routes through the same auth-aware helper, gated to the worker's own origin viaFLOWMESH_BASE_URL(the existing trust boundary, already used byssh_executor.pyand stamped on uploaded artifacts). Origin-only (scheme+host+port) keeps external/public URLs unauthenticated and bounds their timeout to 15s, avoiding both token leakage and the 1800s default stall.Test Plan
End-to-end multi-node validation was run on a real two-node deployment (below), reproducing the producer/consumer-on-separate-results-storage condition on a single host.
Test Result
tests/worker/suite not run locally (GPU/model-download heavy, >20 min) — deferred to CI.End-to-end (multi-node, enforced auth)
Validated on a real two-node FlowMesh deployment on a single host, adapted from the
35_multi_node_deploymentenv template: root node (redis_control+redis_telemetry+ server:8020) plus a worker node (server only, Redis → root), with distinct stack suffixes / ports / results volumes so producer and consumer do not share results storage — the condition that forces the HTTP-URL fallback. Auth was genuinely enforced via thesimple_pluginIdentityProvider (confirmed on/api/v1/nodes: 401 no token, 401 bad token, 200 with token).A producer artifact was stored on root via the authenticated
POST /api/v1/results/{task_id}/files; the shipped string-URL image branch ofDataMixin._collect_prompts_for_specwas then exercised inside the real worker container (host-networked, realFLOWMESH_API_KEY/FLOWMESH_BASE_URL) against the resolved results URL:requests.get(url, timeout=15)HTTPError 401 Unauthorizedis_flowmesh_origin_url→_load_image_from_artifact→resolve_artifact(auth)Note: against current shipped code the pre-fix failure surfaces as
requests.HTTPError 401— theresponse.raise_for_status()on the bare-fetch line raises beforeImage.open, i.e. just ahead of theUnidentifiedImageErrorin the original report. The user-visible effect (the visual-embedding task hard-fails on a cross-node image ref) and the fix are unchanged.Pre-submission Checklist
pre-commit run --all-filesand fixed any issues.uv run pytest tests/passes locally.uv sync --all-packages --group ci --frozen).[BREAKING]and described migration steps above.