Firmware as source of truth: a reproducible analysis pipeline for every gateway (thanks @gdluck!)

[GreenGrassBlueOcean]

@gdluck @anotherjulien @fedem95 @Interstellar0verdrive @xtimmy86x

Thank you, @gdluck

In OpenWebNet-HA/OWNd#77 @gdluck unpacked the MyHOMEServer1 firmware and found the programs that translate between OpenWebNet text and SCS bus frames (bt_luci, coso, bt_termo, bt_device, bt_multi). They then ran those programs on an emulated SCS bus and used them as an oracle: send a frame in, record which bus frame comes out (or that none does). The result is the best source of truth we have ever had for this protocol:

• 15,529 messages the firmware itself emits, all run through OWNd's parser;
• ten concrete OWNd bugs, each with a patch. Examples: set_central_mode sends codes the gateway drops or misreads, two heating parser crashes on frames the firmware really produces, and brightness 0 builds a level the firmware refuses;
• six frames in OWNd's conformance matrix that the firmware rejects, with replacements it accepts.

Every claim says what kind of evidence backs it (firmware run, read-only gateway probe, or code). That is exactly how protocol facts should be established. Amazing work, thank you!

Proposal: do this for every gateway, as a GitHub pipeline

Today a lot of what we "know" comes from manuals that are incomplete or wrong, and from captures on one or two plants. We keep hitting cases nobody can explain. Example: light 74 on an MH200 answers *1*19*74## plus a WHO 1001 DIM 11 fault mask. WHAT 19 is not in WHO_1.pdf or OPEN.db, and the mask bits are undocumented. That is the reason behind the repairs work in #593 and #611. The firmware's bt_luci / bt_device would tell us which bus frame produces WHAT 19 and what each mask bit means.

I'd like to turn @gdluck's method into a reproducible pipeline, one per gateway model we can get an image for (MH200N, F454, MH202, MyHOMEServer1, ...). Every step from binary to finding should be inspectable, but no vendor binary or decompiled code is ever committed:

Stage	Public on GitHub	Stays local
0. Get the image	vendor URL, version, size, SHA-256	the firmware image
1. Unpack	script + list of extracted programs (path, CPU type, SHA-256)	the extracted files
2. Run (oracle)	emulator harness + simulated SCS bus (our own code), input frame lists	—
3. Record	TSV: input frame → bus frame / no frame, firmware hash in the header	—
4. Static analysis (optional)	Ghidra headless scripts that dump lookup tables; function addresses as references	decompiled C
5. Conclusions	findings/<gateway>/*.md pointing at TSV rows; new golden fixtures and tests in OWNd / MyHOME	—

Anyone holding an image with the same SHA-256 can re-run the scripts and diff the TSVs, so every claim can be checked by someone else. Where the firmware can be downloaded publicly from the vendor, a CI job can fetch it, verify the hash, re-run the pipeline and fail if the committed results change. It would never upload binaries as artifacts.

Ground rules (so this stays on the right side of the law)

Under the EU Software Directive (2009/24/EC), observing, studying and testing a program you lawfully own is allowed (Art. 5(3)), and so is decompiling it to achieve interoperability with an independent program (Art. 6). Neither right can be overridden by a licence. Protocols and data formats are not protected by copyright (CJEU C-406/10, SAS Institute). That leads to these rules:

• only images we lawfully hold (official downloads or our own devices); no circumventing encryption;
• prefer black-box emulation (stage 2) over reading disassembly; use static analysis only when needed and only on the translation code;
• publish facts (frame formats, code tables, behaviour) in our own words; never binaries, disassembly or decompiled code;
• the only goal is interoperability with OWNd / MyHOME.

(Not legal advice, just the framework I think we should stick to.)

Suggested setup

• A separate repo in the org, e.g. OpenWebNet-HA/own-firmware-oracle, so OWNd stays a clean library and the tooling has its own licence and history.
• @gdluck, would you be willing to contribute the SCS bus emulator (oracle2/) and the exact firmware version / SHA-256 you used? That is the piece that makes everything else reproducible.
• First targets: WHAT 19 and the WHO 1001 DIM 11 bits in bt_luci / bt_device, cross-checked live on the MH200. Then the ten OWNd fixes from Information #77, each checked again on other gateway models before we merge them, because MyHOMEServer1 and MH200 behaviour may differ.

@anotherjulien @fedem95 @Interstellar0verdrive, what do you think? In particular:

1. Separate repo or a folder in OWNd?
2. Which gateway models / firmware images do you have access to?
3. Any concerns about the approach or the ground rules?

[anotherjulien]

My point of view these days is that everything should end up in the Encyclopedia and Machine KB 😅
But yes, its place is probably a separate repo in the Org
For my work in the Encyclopedia I have stored a few in an R2 bucket, I can probably make it publicly available if need be.

Product	Firmware version	Archived filename
F450	020010	F450_020010.fwz
F450	020010-PROD	F450_020010-PROD.fwz
F453AV	3.0.14	WS_10DIN_030014.fwz
F454	2.0.51	F454_020051.fwz
F455	1.1.2	F455_1_1_2.fwz
F459	020105	F459_020105.fwz
F460	020012	F460_020012.fwz
F461	020011	F461_020011.fwz
MH202	1.0.24	MH202_010024.fwz

[gdluck]

You know that those fwz files are zip archives and the password is the model. Eg: unzip F461_020011.fwz password: F461
Then you get ext3 or ext4 files that can be mounted and see the OS content

[GreenGrassBlueOcean]

That would be amazing idea if you could place these in a new repo. I can than build a GitHub workflow Then we can place all the firmware there My point of view these days is that everything should end up in the Encyclopedia and Machine KB 😅 But yes, its place is probably a separate repo in the Org For my work in the Encyclopedia I have stored a few in an R2 bucket, I can probably make it publicly available if need be. Product Firmware version Archived filename F450 020010 F450_020010.fwz F450 020010-PROD F450_020010-PROD.fwz F453AV 3.0.14 WS_10DIN_030014.fwz F454 2.0.51 F454_020051.fwz F455 1.1.2 F455_1_1_2.fwz F459 020105 F459_020105.fwz F460 020012 F460_020012.fwz F461 020011 F461_020011.fwz MH202 1.0.24 MH202_010024.fwz — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you authored the thread. Message ID: <OpenWebNet-HA/MyHOME/repo-discussions/613/comments/18717715 @ github . com>

[GreenGrassBlueOcean]

@anotherjulien agreed, and I checked GitHub's limits: they rule out storing the images on GitHub even before redistribution comes up. So: private R2 bucket, read-only key, firmware copied in for the run and deleted afterwards. A concrete plan below.

Why the firmware can't live on GitHub

GitHub limit	Value	Effect on us
File size in git	warning at 50 MiB, blocked at 100 MiB	566 MB / 9 ≈ 63 MB on average; some .fwz files are probably over 100 MiB
Repository size	ideally < 1 GB, strongly < 5 GB	566 MB already, and every new version adds more. Git history never shrinks
Git LFS (free plan)	10 GiB storage, 10 GiB bandwidth/month	About 17 full CI runs × 566 MB use up the monthly bandwidth; after that LFS is off until the next month
Release assets	< 2 GiB per file, no limit on total size or bandwidth	Fits on size, but a public release is redistribution
Actions cache	10 GB per repo	Fits, but PRs from forks can restore caches from the base branch, so a fork PR could leak the firmware. Never cache firmware
Actions artifacts	500 MB–1 GB (free plan)	Only for TSV and manifest files, never binaries

R2 doesn't charge for egress and its free tier is 10 GB, so re-fetching 566 MB on every run costs nothing. Where the vendor still offers a .fwz on homesystems-legrandgroup.com, that can be a second source, with R2 as the archive for versions the vendor has removed.

Design (scales from 2 images to N)

Repo OpenWebNet-HA/own-firmware-oracle (public):

catalog/<product>/<version>.yaml   # product, version, filename, size, sha256, sources[]
tools/fwfetch                      # copies a firmware image to local disk, looked up by its SHA-256
tools/unpack                       # one extractor per format → manifest.tsv
oracle/                            # emulator + simulated SCS bus (@gdluck's oracle2/, if contributed)
results/<product>/<version>/       # manifest.tsv, oracle TSVs (firmware hash in the header)
findings/<product>/*.md            # conclusions pointing at TSV rows

• One catalog entry per image, listing its sources in order (vendor: <url>, then r2: firmware/sha256/<hash>.fwz). Adding firmware = one YAML PR + one bucket upload, no workflow changes.
• fwfetch tries each source, checks size and SHA-256, and fails on any mismatch. It writes to a cache outside the repo (~/.cache/own-fw/sha256/<hash>). Locally it also accepts --from-file, so anyone with an image from their own device can produce results without uploading the binary.
• Results are keyed on (firmware SHA-256, tool version). CI only re-runs images whose key changed, so the cost grows with what changed, not with the size of the catalog.
• Output is deterministic: sorted TSV, one row per frame, no timestamps. A re-run with no changes must give a zero diff; that's the reproducibility check.

Workflows:

1. pr.yml runs on every PR, forks included: lint and unit tests against tiny synthetic fixtures, with no secrets and no firmware.

2. oracle.yml runs on push to main, workflow_dispatch and a weekly schedule:

   • a plan job reads the catalog and outputs a matrix of the images that are out of date;
   • each matrix job (max-parallel: 4) does fetch → verify → unpack → manifest → oracle, then deletes the firmware;
   • a final job opens a PR with the changed TSV files.

   The R2 token lives only in a protected firmware environment limited to main. Fork PRs never see it. No pull_request_target.

3. A guard step fails if any file in results/ is over 5 MB or looks like a binary (ELF/zip/squashfs magic). The same check runs as a pre-commit hook, plus *.fwz in .gitignore.

4. Heavy work (Ghidra, long emulation runs) can move to a self-hosted runner later. The standard runner (14 GB SSD, 6 h per job) is enough for fetch and unpack.

Phase 1 pilot: two images

I suggest F454 2.0.51 and MH202 1.0.24. Both are gateways MyHOME users actually run, and the MH202 is the closest relative of the MH200 for the WHAT 19 / DIM 11 question. (The other choice, F450 020010 vs 020010-PROD, would test diffing two versions of the same product, but helps less with protocol questions.)

1. @anotherjulien, if you're willing:
   • move these two images to content-addressed keys;
   • create a read-only R2 API token limited to that bucket;
   • post the SHA-256 and size of each image. Where the vendor still offers the same version, we check that its download has the same hash.
2. Org admin: create the repo and a firmware environment holding R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY and R2_ENDPOINT.
3. Me: the skeleton, fwfetch (S3 API), a first unpack with unblob or binwalk, both workflows, the guard, and the two catalog entries.
4. Done when:
   • manifest.tsv is committed for both images (path, size, SHA-256, CPU type of each extracted program);
   • a second run gives a zero diff;
   • a fork PR passes CI without the firmware or the secret.
5. Phase 2: the emulator, and the first oracle question: which bus frames make bt_luci / bt_device emit WHAT 19, and what each WHO 1001 DIM 11 mask bit means. Cross-checked live on the MH200.

We don't know yet whether F454 and MH202 run Linux programs like MyHOMEServer1 or bare-metal code. The unpack manifest will tell us, and that decides how much of the emulator approach carries over.

Questions for @anotherjulien:

• Can the bucket's object keys (not the files) be public, so contributors can see what exists?
• Who uploads new images?
• Would you make the key read-only per repo or per prefix?

[anotherjulien]

There is an artifact manifest on the Encyclopedia
From there you can search for kind = firmware and you'll find all the info you need.
The new image upload goes through me and an R2 upload workflow on a machine of mine for the moment, along with the manifest update.
I would make the key per prefix unless you have an objection, but I'll need to reorganize the bucket slightly if so.

[GreenGrassBlueOcean]

Quick data point on the MH200N, since it isn't in @anotherjulien's list.

Its firmware is a public download from BTicino, so I grabbed it to test the fetch side of the pipeline:

• Wrapper zip FW_MH200N_vers_010108.zip — official source bticino.be/.../software-mei-2013/timh200n/, 10,065,770 bytes, SHA-256 61f8196d792fa44e9e4cdb67300fca82cebdb8e55f5d27d695694e9827bc7c50. Not encrypted; contains exactly one file:
• scheduler_010108.fwz (MH200N fw 1.1.8) — 10,064,107 bytes, SHA-256 e32d3404302f3619bbbb564ae50554ab81ac6ef818b6e5a2290142a5e2ab5fb3. A zip built 2012-09-19 with Info.txt + scheduler_rel_1_1_8.zip.

Good news for the pipeline: no R2 needed for this one — CI can fetch it from the vendor and verify both hashes.

The catch: the inner .fwz is encrypted (ZipCrypto), so I can't open it. Per the ground rules in the first post (no circumventing encryption), I've stopped at the listing rather than going further — this is the art. 7 "technical protection measure" line, which sits apart from the decompilation right. So the pipeline can fetch + hash + catalog this image, but status: encrypted and unpack is blocked here.

Which makes the real unblocking question for @gdluck: how did you obtain the MyHOMEServer1 image you ran in #77 — a device-side dump, or an image that wasn't encrypted to begin with? Whatever that route was is probably what stage 1 of the pipeline should standardise on.

And @anotherjulien — are the 9 .fwz files in your bucket encrypted the same way, or do some unpack directly? That decides whether stage 1 is "unzip" or something gateway-specific.

[gdluck]

I explained in a previous message how cand you find the password.

[anotherjulien]

OK, to step back a little.

We are NOT distributing the content of the firmware that is protected.
We are NOT distributing a tool to crack open firmware files.
We ARE using a know password scheme to open them in the privacy of our workflow.
We ARE doing this for the purpose of studying software with the goal of documenting how it works.
We ARE doing all of this with the ultimate goal of providing compatibility with new and entirely separate implementations.

We are 100% in the clear for almost all of the above.
The ONLY point of contention/discussion is "using a known password to open the firmware", and that is very defensible on its own, but in the context of studying a piece of software for interoperability, it is VERY defensible.

[gdluck]

You can find my work here

I managed to control devices by sending electronic hex data directly to SCS bus

[GreenGrassBlueOcean]

that is actually amazing. I saw that another julien already has firmware for most of the gatewasy except for mh200 and mh200n do you maybe have this?

[gdluck]

Only mh201 and mh202:
https://www.homesystems-legrandgroup.com/home/-/productsheets/2482873
https://www.homesystems-legrandgroup.com/home/-/productsheets/2482873

You can find them here

[xtimmy86x]

Thanks @gdluck for sharing this work, and @GreenGrassBlueOcean for proposing a reproducible approach. I support having a separate repository for the tooling, with the findings feeding back into the Encyclopedia / Machine KB and regression tests in OWNd.

I would start by reproducing the existing MyHOMEServer1 experiment, validating a concrete finding against a real gateway, and then extending it to a second model. That would give us a solid basis for the wider pipeline.

Each finding should remain tied to the gateway model and firmware version, clearly distinguishing emulation results from observations on real hardware. Differences between gateways are exactly what we need to understand.

I also wouldn't make testing every gateway a prerequisite for every OWNd fix. Parser crashes on valid messages can be addressed independently, while changes to command generation deserve broader compatibility checks.

On my side, I have an F454, a MyHOMEServer1 and a MH201 available and can help validate specific cases on my installation. A small set of reproducible tests, with the expected responses clearly documented, would make that straightforward.

[Interstellar0verdrive]

Impressive work, thank you @gdluck. I can help on the validation side: I run a MyHOMEServer1 (firmware 2.87.13) on a live plant, so findings from the emulated run can be checked against the real gateway here, with read-only probes or bus captures like the cover trace already in the corpus (#466). The WHO 2 side is the one I care about most for the calibration work: which bus frames the gateway produces for cover commands and the *2*1000#x* status replies, and how timer-driven F411 actuators appear. Happy to run a documented set of checks when the first batch of findings is ready.

[GreenGrassBlueOcean]

Status update: phase 1 is live → https://github.com/OpenWebNet-HA/own-firmware-oracle (Apache-2.0)

What works today (MH200N 1.1.8)

Stage	State
Catalog	catalog/MH200N/010108.yaml: vendor URL, size and SHA-256 of the wrapper zip and the .fwz
Fetch	tools/fwfetch.py downloads from the vendor or takes --from-file, and stops on any size or hash mismatch
Unpack	tools/unpack.py identifies each layer by its magic bytes: zip → .fwz → inner zip → U-Boot uImage → gzip → ext2 (read-only debugfs: no mount, no root) → btweb_app.zip
Manifest	results/MH200N/010108/manifest.tsv: 818 rows of path / type / size / SHA-256. A second run gives a zero diff.
Guard + CI	tools/guard.py fails if any tracked file has ELF/zip/gzip/uImage/ext magic or is over 5 MB. Every PR, forks included, runs lint, the guard and synthetic tests, with no secrets and no firmware.

@gdluck was right about the passwords: the .fwz opens with its product-line name (SCHEDULER), and the inner archives use bticino, which the image itself names in HEADER_btweb_app.txt. Both are in the catalog as known packaging strings. The tool tries that list and nothing else.

The MH200N is ARM Linux 2.4 like MyHOMEServer1, with an ext2 rootfs, a recovery rootfs and an application archive that holds bt_luci, bt_device, bt_termo and bt_multi. So the emulator approach from OWNd#77 should carry over.

Proposal: download from Legrand, store nothing ourselves

@anotherjulien, I checked the source_url of every firmware entry in the Encyclopedia's artifact manifest. All ten can still be downloaded from Legrand without a login. The five assets.legrand.com links return exactly the size in your manifest. The five homesystems-legrandgroup.com checkout links return the zip but no Content-Length, so for those only the SHA-256 check after download proves it's the same file.

So I'd like to make the vendor URL the primary source in every catalog entry. The pipeline then needs no R2 token, no firmware environment and no copy of anything, and we never redistribute: we only download from Legrand, the way any installer does. Only images whose hash or tool version changed get rebuilt, so we don't hit their servers on every run. Your bucket stays the archive for the day Legrand removes or replaces a file. When that happens, the hash check fails loudly and we add an r2: source for that one image.

Does that work for you? If so, the next catalog entries are F454 2.0.51 and MH202 1.0.24, using the SHA-256 values from your manifest.

Next

• @gdluck: could you post the version and SHA-256 of the MyHOMEServer1 image you used in OWNd#77? That ties your results to an exact image, and phase 2 can start from there: which bus frames make bt_luci / bt_device emit *1*19*WHERE##, and what each WHO 1001 DIM 11 mask bit means.
• @xtimmy86x, @Interstellar0verdrive: thanks for the offers. Once phase 2 has results, I'll post a concrete probe list with the expected answer for each frame.

[anotherjulien]

Yeah sure, either way works fine 👍
I'm archiving them mostly because they are bound to disappear some day...

Did you settle on a fixed output format for the tests?
I'd love to have something that can be easily ingested in the Encyclopedia and used as source material!

[gdluck]

this is the firmware I used

[GreenGrassBlueOcean]

@anotherjulien Yes! Here is how the oracle output is structured and how it bridges into the Encyclopedia:

1. Oracle Core Format (in own-firmware-oracle)

The raw test runs produce deterministic, header-annotated, sorted TSVs (results/<product>/<version>/oracle/<harness>/<suite>.tsv):

• Header: captures metadata including target product, firmware version, unpacked image SHA-256, tool version, framer (line), responder (pic), and harness (full or unit:<daemon>).
• Columns: direction\tinput\treply\tverdict\toutput
  • direction: down (OpenWebNet sent to gateway via port 20000 command session) or up (bus frame injected at serial UART).
  • input: The exact OpenWebNet frame or PIC frame sent.
  • reply: The gateway's immediate socket response (ack, nack, or - if no frame returned on the command session).
  • verdict: ack (accepted), nack (rejected), silent (dropped/ignored), out (triggered bus UART command or client event notification).
  • output: Space-separated list of bus UART commands emitted (e.g. bus:06D13101420D0D0100) or client event frames (e.g. own:*1*1*31##).

Because rows are sorted and timestamp-free, re-running the test suite gives a zero git diff unless emulator logic or gateway firmware behavior changes.

2. Ingestion into the Encyclopedia (PR #46)

To ingest these directly into the Encyclopedia's evidence system without requiring physical installation topology fields (such as physical installation_id, bus_id, or physical DUT SKU), we opened OpenWebNet-Encyclopedia#46:

• Adds evidence_class: "firmware_emulation" to evidence/schema/evidence-manifest.schema.json.
• Adds epistemic_status: "firmware_observed" and capture_kind: "firmware_oracle".
• Conditions the physical environment fields (installation_id, bus_id, gateway_id, device_under_test) so they are strictly required for physical captures (public_experiment, public_capture) but optional/nullable for firmware emulation packages.

This allows any oracle suite run to be converted into an official Encyclopedia evidence package (evidence/<capture_id>/manifest.json + frames.jsonl) referencing the exact firmware SHA-256 and TSV row.