Add some documentation about matter idl tools that we have for valida…

…ting data (#30198)

* Add some documentation about matter idl tooling

* Restyle

* Add zigbee only note, because that caught me a bit

* Restyle

* Add quotes on cmdline examples

* clearer docs a bit

* Add an external data model for the data model md (make sphinx happy)

* make build succeed

* Some better orphan markers to let sphinx process these files without a TOC entry. I was unsure how to make a proper toc here

* fix spelling

---------

Co-authored-by: Andrei Litvin <andreilitvin@google.com>

data_model/README.md

@@ -1,3 +1,7 @@
+---
+orphan: true
+---
+
 ## Contents
 
 This folder contains a machine-readable representation of matter clusters.

docs/conf.py

@@ -32,9 +32,6 @@
     "examples/thermostat/nxp/linux-se05x/README.md",
     "examples/common/m5stack-tft/repo",
     "docs/guides/README.md",
-    "scripts/tools/memory/memdf/README.md",
-    "scripts/tools/memory/platform/README.md",
-    "scripts/tools/memory/README-GitHub-CI.md",
 ]
 
 
@@ -67,6 +64,7 @@
 
 external_content_contents = [
     (MATTER_BASE / "docs", "[!_R]*"),
+    (MATTER_BASE, "data_model/**/*.md"),
     (MATTER_BASE, "README.md"),
     (MATTER_BASE, "examples/**/*.md"),
     (MATTER_BASE, "examples/**/*.png"),
@@ -82,5 +80,6 @@
     "README",  # cannot detect README.md
     "scripts/",
     "examples/android/",
+    "data_model/",
 ]
 external_content_link_extensions = [".md", ".png", ".jpg", ".svg"]

docs/guides/index.md

@@ -55,6 +55,7 @@ ti/ti_platform_overview
 
 -   [Access Control](./access-control-guide.md)
 -   [IP Commissioning](./ip_commissioning.md)
+-   [Matter IDL tooling and validation](./matter_idl_tooling.md)
 
 ## Setup Guides
 

docs/guides/matter_idl_tooling.md

@@ -0,0 +1,86 @@
+# The `.matter` IDL file format
+
+The matter IDL file format is designed to be a human-readable representation of
+data structures, cluster definitions and endpoint composition.
+
+Since it is designed to be easy for both machine and humans to read, it is the
+basis of some tools to make validating zap-based cluster definitions easier.
+
+More details on the format in
+[matter_idl/README.md](../../scripts/py_matter_idl/matter_idl/README.md).
+
+## Parsing CSA XML Data definitions
+
+The SDK contains a copy of CSA XML data definitions in `data_model/clusters`.
+
+The files there are updated by running a scraper against the official matter
+specification and are a good source of truth for actual definitions. Update
+information available in [data_model/README.md](../../data_model/README.md).
+
+NOTE: scraper is a work in progress, XML data may be incomplete or have errors
+still.
+
+The script `./scripts/py_matter_idl/matter_idl/data_model_xml_parser.py` has the
+ability to parse one or more CSA data model XML files and output their content
+in `.matter` format. For example:
+
+```sh
+./scripts/py_matter_idl/matter_idl/data_model_xml_parser.py data_model/clusters/BooleanState.xml
+```
+
+The tool supports several options that are useful for development:
+
+| Argument(s)             | Description                                                                         |
+| ----------------------- | ----------------------------------------------------------------------------------- |
+| `-o/--output PATH`      | Output the matter file into a path instead of STDOUT.                               |
+| `--compare PATH`        | Also read another `.matter` file for compare. MUST be used with `--compare-output`. |
+| `--compare-output PATH` | Output the subset of `--compare` clusters that matches XML into PATH.               |
+
+Using `--compare` AND `--compare-output` produce output that is easier to
+compare as opposed to using existing zap-generated matter files because it
+strips out comments and it also alpha-sorts elements so that diffs are
+human-readable.
+
+### Comparing a `.matter` file against the spec
+
+Combining arguments to the tool allows getting a diff between SDK and
+specification:
+
+-   `data_model/clusters/*.xml` are assumed to be the official specification
+    definitions
+-   `src/controller/data_model/controller-clusters.matter` contains _all_
+    clusters defined in the SDK
+
+As such one can run compares such as:
+
+```sh
+./scripts/py_matter_idl/matter_idl/data_model_xml_parser.py         \
+     -o out/spec.matter                                             \
+     --compare-output out/sdk.matter                                \
+     --compare src/controller/data_model/controller-clusters.matter \
+     data_model/clusters/DoorLock.xml                               \
+  && diff out/{spec,sdk}.matter
+```
+
+NOTE: due to specification data scraper still being in development, the diff
+should be human-validated (e.g. for tool errors or Zigbee-only markers in the
+spec).
+
+## Linting `.matter` files for devices
+
+For device validation, `./scripts/idl_lint.py` provides the ability to validate
+a matter file for some basic conformance logic. These rules are expressed in
+`scripts/rules.matterlint`.
+
+The rules generally are:
+
+-   pre-loaded from silabs XML files (validates that mandatory attributes are
+    present)
+-   Hard-coded rules (e.g. mandatory clusters and attributes on specific
+    endpoints)
+
+Usage:
+
+```sh
+./scripts/idl_lint.py ./examples/window-app/common/window-app.matter
+```

scripts/tools/memory/README-GitHub-CI.md

@@ -1,3 +1,7 @@
+---
+orphan: true
+---
+
 # Scripts for GitHub CI
 
 A set of `gh_*.py` scripts work together to produce size comparisons for PRs.

scripts/tools/memory/memdf/README.md

@@ -1,3 +1,7 @@
+---
+orphan: true
+---
+
 This package contains routines to to collect, aggregate, and report memory
 usage, using Pandas `DataFrame` as the primary representation.
 

scripts/tools/memory/platform/README.md

@@ -1,3 +1,7 @@
+---
+orphan: true
+---
+
 A config file is a Python (nested) dictionary used by
 `memdf.util.config.Config`.