Where should canonical evidence live?

[anotherjulien]

I’d like to get some thoughts on where we should keep the canonical field evidence behind the Encyclopedia.

A lot of the traces we collect are not really collected just for testing MyHOME. We collect them because we want to understand and document OpenWebNet protocol behavior and actual device behavior: what a gateway emits, how a device responds, which dimensions are supported, where implementations differ, and so on.

And the natural place for that knowledge is the Encyclopedia.

Right now, quite a lot of valuable real-world evidence lives under MyHOME/tests/fixtures: captures from physical installations, specific device/gateway combinations, firmware observations, controlled tests, etc. Those files are useful to MyHOME, but I’m not sure MyHOME should be their canonical home.

My instinct is that the Encyclopedia should hold the evidence for the claims it makes.

That would give us a much cleaner chain:

field capture → documented evidence → Encyclopedia claim → Machine KB

It would also make the Encyclopedia much easier to audit. When a page says that a device behaves a certain way, we should ideally be able to point back to the exact capture, hardware context, firmware, test action, and provenance supporting that statement.

This matters because descriptions and READMEs can sometimes drift beyond what the preserved wire trace actually proves. Keeping the evidence close to the documentation makes it easier to distinguish direct observations from interpretations, tester reports, and unresolved questions.

It also avoids making MyHOME the accidental historical archive for knowledge that belongs to the whole Organization. The same F418U2, F461, or gateway capture may be useful to the Encyclopedia, Machine KB, MyHOME, OWNd, or future tools.

For MyHOME, I don’t think this needs to make testing more complicated.

A clean approach could be:

• canonical, sanitized captures live in the Encyclopedia;
• each capture gets a stable evidence ID and hash;
• MyHOME keeps a small manifest of the evidence its tests use;
• a sync tool generates deterministic MyHOME test fixtures from a pinned Encyclopedia revision;
• generated fixtures remain checked into MyHOME, so normal tests stay self-contained and offline;
• CI verifies that those fixtures still match the canonical evidence.

So the separation would be:

Encyclopedia owns the evidence and the documented knowledge derived from it.
MyHOME owns the tests derived from that evidence.

Synthetic parser/test fixtures would of course stay in MyHOME.

Before moving anything, we could simply classify the current fixture corpus into canonical field evidence, derived regression fixtures, synthetic fixtures, and implementation-specific data.

Would people be comfortable with the Encyclopedia becoming the Organization’s canonical home for publishable field evidence, especially when that evidence was collected specifically to understand and document protocol or device behavior?