Skip to content

Add FirmwareJob.source / source_label fields (7a-2a)#556

Merged
bdraco merged 1 commit into
mainfrom
7a-2a-firmware-job-source-field
May 10, 2026
Merged

Add FirmwareJob.source / source_label fields (7a-2a)#556
bdraco merged 1 commit into
mainfrom
7a-2a-firmware-job-source-field

Conversation

@bdraco
Copy link
Copy Markdown
Member

@bdraco bdraco commented May 10, 2026

What does this implement/fix?

First slice of the transparent install flow described in
issue #106 § Transparent install flow.
Makes the local-vs-remote-build distinction a first-class
field
on FirmwareJob: the firmware queue's runner branches
on source to choose its pipeline (local subprocess vs
peer-link submit_job dispatch), and the install dialog reads
source_label to decide whether to render the
"Building on {receiver_label}" sub-line.

No synthetic-job wrapper. No event-stream bridge. One
FirmwareJob per build, the same JOB_* lifecycle events fire
either way, source is the only place LOCAL and REMOTE diverge
from a caller's perspective. The original design doc proposed a
synthetic-FirmwareJob + bridge construction to avoid changing
the install dialog's subscriber; that was extra ceremony for a
problem that doesn't exist (frontend ships lockstep with
backend, so adding a small consumer in the dialog when this PR
lands is one coordinated change). Arch.md catches up in a
separate docs PR.

What lands

New JobSource StrEnum in models/firmware.py:

  • LOCAL — this dashboard's CPU compiled the artifacts
    (today's default; matches every job-row written before this
    PR).
  • REMOTE — a paired receiver compiled the artifacts and
    this dashboard fetched them via download_artifacts.

New fields on FirmwareJob:

  • source: JobSource = JobSource.LOCAL — the discriminator.
  • source_pin_sha256: str = ""machine-readable handle
    on the receiver that compiled this job, when
    source == REMOTE. Matches StoredPairing.pin_sha256.
    Load-bearing for restart recovery: a crash-recovered REMOTE
    job needs to know which receiver to route
    download_artifacts / cancel_job against, and the RAM-only
    _open_peer_links cache doesn't survive the restart. Same
    shape as 5c-2's receiver-side remote_peer (machine id, not
    display label) and for the same reason.
  • source_label: str = "" — display string for the
    receiver-label when source == REMOTE. Mirrors
    StoredPairing.label at job-creation time; doesn't track
    later renames of the pairing label (snapshot semantics). The
    machine-readable handle lookups go through is
    source_pin_sha256; source_label is purely for rendering.

All three default to the LOCAL shape so an offloader upgrading
across this landing sees its prior on-disk job-rows
deserialise correctly.

Distinct from remote_peer / remote_job_id — those are
receiver-side fields filled by 5c-2. The new fields are the
offloader-side for the same delegation seen from the
dispatching dashboard. Both pairs end up set on opposite-side
FirmwareJob rows for a single transparent dispatch.

FirmwareJob.reset()'s "preserves identity" list extended
to include source / source_pin_sha256 / source_label /
remote_peer / remote_job_id. The remote_* omission was
already a latent bug for 5c-2 receivers (a crashed
receiver-side job losing remote_peer would strand the
rebuild from the offloader's correlation cache). Pinned with
a new test.

Test additions

  • test_source_defaults_to_local — default value.
  • test_remote_source_round_trips_through_serialisation
    persistence contract: all three new fields keep their values
    across a dashboard restart.
  • test_older_sidecar_without_source_field_loads_as_local
    upgrade-from-pre-7a-2a path: hand-crafted minimal JSON
    without the new keys deserialises with default values
    rather than crashing.
  • test_malformed_source_value_rejected_on_load — corrupt
    source: "intergalactic" raises on construction; doesn't
    silently land in code that branches on
    is JobSource.REMOTE.
  • test_reset_preserves_dispatch_origin_fields — all five
    dispatch-origin fields survive FirmwareJob.reset().

ruff + mypy clean. 9 tests in tests/models/test_firmware_job.py.

What this PR does NOT do

  • No caller integration. The runner's source-routing
    branch lands in 7a-2b (rename of the previous "event-stream
    bridge" slice; the synthetic framing is gone in the new
    plan) alongside firmware/install integration in 7a-3.
    Today nothing reads the new field; landing it standalone
    keeps the model / persistence layer reviewable ahead of
    the runner's larger blast radius.
  • No frontend consumer. The install dialog's
    "Building on {receiver_label}" sub-line lands when 7a-3
    wires firmware/install through the scheduler; the
    frontend FirmwareJob mirror gets the new fields then.
    Lockstep deployment means this is one coordinated PR
    rather than two waves of compatibility work.

Related issue or feature:

Types of changes

  • Bugfix (non-breaking change which fixes an issue) — bugfix
  • New feature (non-breaking change which adds functionality) — new-feature
  • Enhancement to an existing feature — enhancement
  • Breaking change (fix or feature that would cause existing functionality to not work as expected) — breaking-change
  • Refactor (no behaviour change) — refactor
  • Documentation only — docs
  • Maintenance / chore — maintenance
  • CI / workflow change — ci
  • Dependencies bump — dependencies

Frontend coordination

  • No frontend change needed
  • Companion frontend PR: esphome/device-builder-dashboard-frontend#

Checklist

  • The code change is tested and works locally.
  • Pre-commit hooks pass (ruff, codespell, yaml/json/python checks).
  • Tests have been added or updated under tests/ where applicable.
  • components.json has not been hand-edited.
  • Architecture-level changes are reflected in docs/ARCHITECTURE.md and/or docs/API.md — follow-up docs PR drops the synthetic-job framing this PR replaces.

Copilot AI review requested due to automatic review settings May 10, 2026 23:13
@github-actions github-actions Bot added the new-feature New feature label May 10, 2026
@codspeed-hq
Copy link
Copy Markdown

codspeed-hq Bot commented May 10, 2026

Merging this PR will not alter performance

✅ 12 untouched benchmarks


Comparing 7a-2a-firmware-job-source-field (6751574) with main (2c97a87)

Open in CodSpeed

@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented May 10, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.12%. Comparing base (2c97a87) to head (6751574).

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main     #556   +/-   ##
=======================================
  Coverage   99.12%   99.12%           
=======================================
  Files          77       77           
  Lines       10056    10062    +6     
=======================================
+ Hits         9968     9974    +6     
  Misses         88       88           
Flag Coverage Δ
py3.12 99.09% <100.00%> (+<0.01%) ⬆️
py3.14 99.12% <100.00%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
esphome_device_builder/models/firmware.py 100.00% <100.00%> (ø)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a FirmwareJob “origin” discriminator so the backend can represent locally-built vs remotely-delegated firmware artifacts as the transparent install flow is layered in, while remaining backwards-compatible with pre-existing persisted job sidecars.

Changes:

  • Introduce JobSource (LOCAL / REMOTE) and add FirmwareJob.source + FirmwareJob.source_label with LOCAL-shaped defaults for backward-compatible deserialization.
  • Extend FirmwareJob.reset()’s “preserves identity” contract to explicitly include dispatch-origin fields (source, source_label, remote_peer, remote_job_id).
  • Add unit tests covering defaults, JSON round-trip persistence, upgrade-from-older-sidecar behavior, and reset preservation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
esphome_device_builder/models/firmware.py Adds JobSource enum and new FirmwareJob fields for local vs remote build provenance; updates reset identity contract docs.
tests/models/test_firmware_job.py Adds tests that pin defaults, persistence/deserialization compatibility, and reset preservation for dispatch-origin fields.

Comment thread tests/models/test_firmware_job.py Outdated
Comment thread esphome_device_builder/models/firmware.py Outdated
@bdraco bdraco force-pushed the 7a-2a-firmware-job-source-field branch 2 times, most recently from 0561e5b to 7f7b973 Compare May 10, 2026 23:20
bdraco added a commit that referenced this pull request May 10, 2026
The arch.md description of the transparent install flow's
implementation slices used a "synthetic offloader-side
FirmwareJob + event-stream bridge" framing inherited from the
original design doc. That shape was reverse-engineered from
"what's the minimum change to avoid touching the install
dialog's subscriber?" — a bad starting point. It created two
FirmwareJobs per remote build (one on each side), a bridge
that re-fired OFFLOADER_JOB_* events as JOB_* against a
synthetic id, and a separate RAM-only id mapping. Extra
ceremony for a problem that doesn't exist: lockstep frontend
deployment means the install dialog can learn the new field
(`job.source_label`) in the same coordinated PR that lands the
source-routed runner.

Replaced the 7a-2 / 7a-3 / 7a-4 section with the first-class
shape:

- One `FirmwareJob` per build, the runner branches on `source`
  to pick its pipeline (local subprocess vs peer-link
  `submit_job` dispatch).
- Same `JOB_*` lifecycle events fire either way; the install
  dialog stays on its existing subscriber.
- `OFFLOADER_JOB_*` events stay as the wire-layer fan-out
  (5c-3 keeps them for the explicit Send-builds dialog that
  hangs around as the power-user surface); the runner
  consumes them privately to update its single FirmwareJob.

Renumbered slices to reflect the new shape:

- 7a-2a: `FirmwareJob.source` / `source_pin_sha256` /
  `source_label` field additions (in flight as #556).
- 7a-2b: source-routed runner (renamed from "event-stream
  bridge"; the bridge concept disappears entirely).
- 7a-3: `firmware/install` integration unchanged in spirit
  but now constructs a regular FirmwareJob with source=REMOTE
  rather than a synthetic wrapper.
- 7a-4: cancel translation also simplifies — the runner's
  cancel hook fires remote_build/cancel_job directly, no id
  translation between two FirmwareJobs.

Added explanatory paragraph above the slices about why the
synthetic framing was wrong, so future readers don't have to
reconstruct the reasoning from PR history. Issue #106's
§ Transparent install flow gets the same drop in a separate
edit (the issue body is the source of truth for the full
design; this PR brings arch.md into alignment).

No code change.
bdraco added a commit that referenced this pull request May 10, 2026
The arch.md description of the transparent install flow's
implementation slices used a "synthetic offloader-side
FirmwareJob + event-stream bridge" framing inherited from the
original design doc. That shape was reverse-engineered from
"what's the minimum change to avoid touching the install
dialog's subscriber?" — a bad starting point. It created two
FirmwareJobs per remote build (one on each side), a bridge
that re-fired OFFLOADER_JOB_* events as JOB_* against a
synthetic id, and a separate RAM-only id mapping. Extra
ceremony for a problem that doesn't exist: lockstep frontend
deployment means the install dialog can learn the new field
(`job.source_label`) in the same coordinated PR that lands the
source-routed runner.

Replaced the 7a-2 / 7a-3 / 7a-4 section with the first-class
shape:

- One `FirmwareJob` per build, the runner branches on `source`
  to pick its pipeline (local subprocess vs peer-link
  `submit_job` dispatch).
- Same `JOB_*` lifecycle events fire either way; the install
  dialog stays on its existing subscriber.
- `OFFLOADER_JOB_*` events stay as the wire-layer fan-out
  (5c-3 keeps them for the explicit Send-builds dialog that
  hangs around as the power-user surface); the runner
  consumes them privately to update its single FirmwareJob.

Renumbered slices to reflect the new shape:

- 7a-2a: `FirmwareJob.source` / `source_pin_sha256` /
  `source_label` field additions (in flight as #556).
- 7a-2b: source-routed runner (renamed from "event-stream
  bridge"; the bridge concept disappears entirely).
- 7a-3: `firmware/install` integration unchanged in spirit
  but now constructs a regular FirmwareJob with source=REMOTE
  rather than a synthetic wrapper.
- 7a-4: cancel translation also simplifies — the runner's
  cancel hook fires remote_build/cancel_job directly, no id
  translation between two FirmwareJobs.

Added explanatory paragraph above the slices about why the
synthetic framing was wrong, so future readers don't have to
reconstruct the reasoning from PR history. Issue #106's
§ Transparent install flow gets the same drop in a separate
edit (the issue body is the source of truth for the full
design; this PR brings arch.md into alignment).

No code change.
First slice of the transparent install flow described in issue
#106 § Transparent install flow. Makes the
local-vs-remote-build distinction a first-class field on
FirmwareJob: the firmware queue's runner branches on `source`
to choose its pipeline (local subprocess vs peer-link
`submit_job` dispatch), and the install dialog reads
`source_label` to decide whether to render the
"Building on {receiver_label}" sub-line. No synthetic-job
wrapper, no event-stream bridge — one FirmwareJob per build,
the same lifecycle events fire either way, `source` is the
only place LOCAL and REMOTE diverge from a caller's
perspective.

New `JobSource` StrEnum in `models/firmware.py`:

- `LOCAL` — this dashboard's CPU compiled the artifacts
  (today's default; matches every job-row written before
  this PR).
- `REMOTE` — a paired receiver compiled the artifacts and
  this dashboard fetched them via `download_artifacts`.

New fields on `FirmwareJob`:

- `source: JobSource = JobSource.LOCAL` — the discriminator.
- `source_pin_sha256: str = ""` — machine-readable handle on
  the receiver that compiled this job, when `source == REMOTE`.
  Matches `StoredPairing.pin_sha256`. Load-bearing for restart
  recovery: a crash-recovered REMOTE job needs to know which
  receiver to route `download_artifacts` / `cancel_job` against,
  and the RAM-only `_open_peer_links` cache doesn't survive
  the restart. Same shape as 5c-2's receiver-side `remote_peer`
  (machine id, not display label) and for the same reason.
- `source_label: str = ""` — display string for the
  receiver-label when `source == REMOTE`; empty for LOCAL.
  Mirrors `StoredPairing.label` at job-creation time; doesn't
  track later renames of the pairing label (snapshot semantics).
  The machine-readable handle lookups go through is
  `source_pin_sha256`; `source_label` is purely for rendering.

All three default to the LOCAL shape so an offloader upgrading
across this landing sees its prior on-disk job-rows
deserialise correctly (mashumaro fills defaults for keys
missing from older sidecars).

Distinct from `remote_peer` / `remote_job_id` — those are
*receiver*-side fields filled by 5c-2 when the receiver picks
up an offloader's `submit_job`. `source` / `source_pin_sha256`
/ `source_label` are the *offloader*-side fields for the same
delegation seen from the dispatching dashboard. Both pairs end
up set on opposite-side FirmwareJob rows for a single
transparent dispatch (offloader's job: source=REMOTE +
source_pin_sha256=<receiver_pin> + source_label="<label>";
receiver's job: remote_peer="<offloader dashboard_id>" +
remote_job_id="<offloader job_id>").

`FirmwareJob.reset()`'s "preserves identity" list extended to
include `source` / `source_pin_sha256` / `source_label` /
`remote_peer` / `remote_job_id` — all five describe the job's
dispatch origin, not per-run state, so they have to survive a
crash-recovery rebuild. The previous docstring listed only the
local fields; the omission was load-bearing for 5c-2 receivers
too (a crashed receiver-side job losing `remote_peer` would
strand the rebuild from the offloader's correlation cache).

Test additions:

- test_source_defaults_to_local pins the default.
- test_remote_source_round_trips_through_serialisation pins
  the persistence contract: REMOTE FirmwareJobs keep their
  source / source_pin_sha256 / source_label across a dashboard
  restart, otherwise the firmware queue's runner would lose
  the receiver handle it needs to route download_artifacts /
  cancel_job on the rebuild.
- test_older_sidecar_without_source_field_loads_as_local —
  hand-crafted minimal JSON without the new keys deserialises
  with default values rather than crashing on the
  persistence-load path.
- test_malformed_source_value_rejected_on_load — corrupt
  `source: "intergalactic"` raises on construction; doesn't
  silently land in code that branches on `is JobSource.REMOTE`.
- test_reset_preserves_dispatch_origin_fields pins all five
  dispatch-origin fields surviving FirmwareJob.reset().

ruff + mypy clean. No caller integration in this PR; the
runner's source-routing branch lands in 7a-2b alongside
firmware/install integration in 7a-3.
@bdraco bdraco force-pushed the 7a-2a-firmware-job-source-field branch from 7f7b973 to 6751574 Compare May 10, 2026 23:34
@bdraco bdraco merged commit ed9e994 into main May 10, 2026
13 checks passed
@bdraco bdraco deleted the 7a-2a-firmware-job-source-field branch May 10, 2026 23:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new-feature New feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants