Add hardware metadata files, generate setup scripts and hardware docs from them. #883

petejohanson · 2021-07-27T02:31:12Z

Draft PR for the ongoing work. I am approaching a reasonable minimal metadata format to address our immediate pain points, that I believe can be used as a starting point for our further work on this, more advanced setup web utility from @joelspadin , etc.

Overview

The PR proposes a standard ${item_id}.zmk.yml filenaming convention, where the item might be a board, shield, or interconnect. Those files contain a standard YAML structure, and I have also created a JSON schema that can be used to validate these files, e.g. west metadata check, including a GH Action to perform these checks.

Each item has a required type property of board, shield, or interconnect. By having a dedicated interconnect file, we can include information about the interconnect without duplicating it, and also achieve "promote a board ID to an interconnect ID" and keep the metadata about both uses distinct.

Examples

boards/arm/nrfmicro/nrfmicro_13.zmk.yml:

id: nrfmicro_13
name: nRFMicro 1.3/1.4
type: board
arch: arm
outputs:
  - usb
  - ble
url: https://github.com/joric/nrfmicro/
exposes: [pro_micro]

boards/shields/corne/corne.zmk.yml:

id: corne
name: Corne
type: shield
url: https://github.com/foostan/crkbd/
requires: [pro_micro]
exposes: [i2c_oled]
features:
  - keys
  - display
siblings:
- corne_left
- corne_right

boards/interconnects/pro_micro.zmk.yml:

id: pro_micro
name: Pro Micro
type: interconnect
url: https://www.sparkfun.com/products/12640
manufacturer: SparkFun
description: |
  The SparkFun Pro Micro grew popular as a low cost ATmega32U4 board with sufficient GPIO and peripherals
  to work for many keyboard needs. Since the original Pro Micro, many pin compatible boards have appeared,
  with various changes or improvements, such as the Elite-C w/ USB-C, nice!nano with nRF52840 wireless.

Hardware Docs Page

Link: https://deploy-preview-883--zmk.netlify.app/docs/hardware

Screenshot:

TODO

Ensure everyone is happy w/ current minimalist format.
Investigate "layout" metadata, and if we need to account for that in this first pass at metadata + schema.
Add docs for how to create such a file when writing a board or shield.
Complete metadata for existing boards/shield.
Determine regression w/ styling of code blocks being lost.

Nicell · 2021-07-27T05:31:19Z

Generally, I think this metadata format is a good start. I like the exposes/requires stuff, nice. Should things like the pro_micro expose pro_micro? or is that inherent to its ID?

Maybe we can consider rolling all the power profiler stuff into here too. Although that may require a bit of a refactor on how we currently do it, including getting readings of different SoCs.

Do we have a plan on how to update/extend this metadata? I can list 100 more things I could see us having on this, but I don't think we need them right now.

Nicell · 2021-07-27T05:44:50Z

Besides getting rid of the hardcoded setup scripts and hardware docs page, we also need to update the GitHub workflows to not require hardcoding every board and shield as well. Or maybe we can try doing an overhaul with only the minimum being built with PRs (#502).

petejohanson · 2021-07-27T11:49:17Z

Generally, I think this metadata format is a good start. I like the exposes/requires stuff, nice. Should things like the pro_micro expose pro_micro? or is that inherent to its ID?

Hmm... Good question. Let me think on that.

Maybe we can consider rolling all the power profiler stuff into here too. Although that may require a bit of a refactor on how we currently do it, including getting readings of different SoCs.

I could see getting some power related stuff in here, yeah, e.g. quiescent at idle, perhaps add a vcc_cutoff option to the features list, etc

Do we have a plan on how to update/extend this metadata? I can list 100 more things I could see us having on this, but I don't think we need them right now.

A few thoughts:

We can add a version: 1 to these right now and make it required. That acknowledges we'll have future versions.
We can add additional optional properties easily, and then backfill additional values.
When we settle on the final new set of required items, we bump to a new major version, and update those to add to the required list
Any code in the meantime leverages schema to add null checks to verify optional props they want to use.

docs/package.json

docs/src/components/hardware-list.tsx

docs/package.json

joelspadin · 2021-07-30T05:43:12Z

schema/hardware-metadata.schema.json

+  "$defs": {
+    "id": {
+      "type": "string",
+      "pattern": "^[a-z0-9_]+$"


Is it okay for an ID to start with a number? (Not sure if these IDs need to match C identifier rules.)

These do not need to be C identifiers, and we should be a bit more flexible here, for when someone has a keyboard named funky things like _33ble (Looking at you, @tominabox1 )

That works. Just wanted to make sure we didn't need this to be [a-z_][a-z0-9_]*

schema/hardware-metadata.schema.json

docs/src/components/hardware-list.tsx

Nicell

Some small proofreading suggestions. Also, can we use package-lock.json version 2 for this PR?

docs/docs/development/hardware-metadata-files.md

docs/src/components/hardware-list.tsx

docs/src/templates/setup.sh.mustache

joelspadin · 2021-08-03T23:08:28Z

Also, can we use package-lock.json version 2 for this PR?

Agreed. It would be nice to stop completely rewriting the lock file every few changes. (Upgrade to Node 14.x which includes NPM 7.x, or just upgrade NPM on its own.)

petejohanson · 2021-09-09T04:25:14Z

Also, can we use package-lock.json version 2 for this PR?

Agreed. It would be nice to stop completely rewriting the lock file every few changes. (Upgrade to Node 14.x which includes NPM 7.x, or just upgrade NPM on its own.)

This was done from our dev container. I have a feeling, dependabot keeps upgrading it, causing the flip floppiness.

petejohanson · 2021-09-09T20:30:37Z

Also, can we use package-lock.json version 2 for this PR?

Agreed. It would be nice to stop completely rewriting the lock file every few changes. (Upgrade to Node 14.x which includes NPM 7.x, or just upgrade NPM on its own.)

Fixed this up to avoid the lock file version flip flopping.

app/boards/shields/jian/jian.zmk.yml

Nicell

🎉

app/scripts/west_commands/metadata.py

docs/docs/development/hardware-metadata-files.md

joelspadin · 2021-09-11T02:54:10Z

docs/docs/development/hardware-metadata-files.md

+
+The array should contrain the complete harware IDs of the siblings that combine in the logical device, e.g.:
+
+```yaml


"Siblings" might be a confusing term to use here. It seems to me like "corne" is more of a top-level item, and "corne_left" and "corne_right" are its children. Siblings makes me think that you'd have something more like

id: corne_left siblings: [corne_right]

id: corne_right siblings: [corne_left]

(You also need to scroll up to the top of this page to see that this array belongs in the "corne" file and not in both left and right files.)

docs/src/components/hardware-list.tsx

joelspadin · 2021-09-11T03:15:39Z

docs/src/components/hardware-list.tsx

+      </p>
+      <ul>
+        {grouped["onboard"]
+          .sort((a, b) => a.name.localeCompare(b.name))


If we ever start getting into things with double digit version numbers, we might want to pull in something like https://www.npmjs.com/package/string-natural-compare to sort numbers embedded in strings numerically instead of lexicographically.

docs/src/hardware-schema-typescript-plugin/index.js

docs/src/setup-script-generation-plugin/index.js

* Initial hardware metadata JSON schema. * GH Action to validate all schemas for boards/shields.

* Generate TS types from new hardware metadata schema. * Aggregate all hw metadata YAML into one aggregate JSON file for consumption by others.

* Standalone and pro-micro format board metadata added.

* No longer recommending that all new shields get added to the build matrix in `build.yml`, so we avoid conflicts.

* Add interconnect details. * Add m60 and nrf52840_m2 metadata files.

petejohanson added documentation Improvements or additions to documentation enhancement New feature or request board PRs and issues related to boards. user configs shields PRs and issues related to shields labels Jul 27, 2021

petejohanson self-assigned this Jul 27, 2021

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch from 3bf4a2c to e144133 Compare July 27, 2021 02:36

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch 4 times, most recently from f86bb2f to e49a8fd Compare July 28, 2021 13:24

joelspadin reviewed Jul 30, 2021

View reviewed changes

docs/package.json Show resolved Hide resolved

joelspadin reviewed Jul 30, 2021

View reviewed changes

Nicell reviewed Aug 1, 2021

View reviewed changes

docs/src/components/hardware-list.tsx Outdated Show resolved Hide resolved

Nicell mentioned this pull request Aug 2, 2021

feat(ci): Use metadata for builds #889

Merged

6 tasks

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch 3 times, most recently from d6aba4a to bdd2033 Compare August 3, 2021 02:32

Nicell reviewed Aug 3, 2021

View reviewed changes

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch 2 times, most recently from 25aff76 to c6b6580 Compare September 9, 2021 04:22

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch from c6b6580 to c632551 Compare September 9, 2021 20:19

petejohanson marked this pull request as ready for review September 9, 2021 20:23

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch from c632551 to 30f8a38 Compare September 9, 2021 20:42

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch 3 times, most recently from 5fb0c1c to 8e00e6a Compare September 10, 2021 14:49

krikun98 reviewed Sep 10, 2021

View reviewed changes

app/boards/shields/jian/jian.zmk.yml Outdated Show resolved Hide resolved

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch 2 times, most recently from 160f92a to 8425c52 Compare September 10, 2021 16:59

Nicell approved these changes Sep 10, 2021

View reviewed changes

joelspadin reviewed Sep 11, 2021

View reviewed changes

petejohanson added 2 commits September 10, 2021 23:34

feat: Add hardware metadata schema.

a806673

* Initial hardware metadata JSON schema. * GH Action to validate all schemas for boards/shields.

feat: Add a few sample hardware metadata files.

2a8dde9

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch from 8425c52 to 01ab542 Compare September 11, 2021 03:34

feat(docs): Add TS type gen, metadata JSON gen

7616c50

* Generate TS types from new hardware metadata schema. * Aggregate all hw metadata YAML into one aggregate JSON file for consumption by others.

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch 3 times, most recently from 7e0034c to 390e25b Compare September 11, 2021 04:17

petejohanson added 2 commits September 11, 2021 00:24

feat(docs): Add dynamic hardware list component.

c4f1662

feat(docs): Type check the docs TSX components.

02f05fa

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch from 390e25b to 8bc3ab9 Compare September 11, 2021 04:25

petejohanson added 8 commits September 11, 2021 00:35

feat: Generate setup scripts from metadata.

2af7a21

docs: Add docs on hardware metadata files.

87aab0a

refactor(shields): Add pro-micro shield metadata.

af7f54f

fix(docs): Deploy when boards/shields change.

45ce988

refactor(boards): Add metadata for boards.

064d717

* Standalone and pro-micro format board metadata added.

refactor(docs): Remove build.yml for new shield.

f404ad1

* No longer recommending that all new shields get added to the build matrix in `build.yml`, so we avoid conflicts.

feat(metadata): Add MakerDiary nRF52840 M.2 interconnect

8508a2a

* Add interconnect details. * Add m60 and nrf52840_m2 metadata files.

feat(metadata): Add YAML check/format npm scripts.

5023ae8

petejohanson force-pushed the boards-shields/hardware-metadata-schema branch from 8bc3ab9 to 5023ae8 Compare September 11, 2021 04:35

petejohanson merged commit c7e5136 into zmkfirmware:main Sep 11, 2021

dxmh mentioned this pull request Sep 14, 2021

fix(setup.sh): Add missing shields to setup script #769

Closed

caksoylar mentioned this pull request Apr 3, 2022

Board & Shield Documentation #35

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add hardware metadata files, generate setup scripts and hardware docs from them. #883

Add hardware metadata files, generate setup scripts and hardware docs from them. #883

petejohanson commented Jul 27, 2021 •

edited

Loading

Nicell commented Jul 27, 2021 •

edited

Loading

Nicell commented Jul 27, 2021

petejohanson commented Jul 27, 2021

joelspadin Jul 30, 2021

petejohanson Aug 3, 2021

joelspadin Aug 3, 2021

Nicell left a comment

joelspadin commented Aug 3, 2021

petejohanson commented Sep 9, 2021

petejohanson commented Sep 9, 2021

Nicell left a comment

joelspadin Sep 11, 2021

joelspadin Sep 11, 2021


		The array should contrain the complete harware IDs of the siblings that combine in the logical device, e.g.:

		```yaml

Add hardware metadata files, generate setup scripts and hardware docs from them. #883

Add hardware metadata files, generate setup scripts and hardware docs from them. #883

Conversation

petejohanson commented Jul 27, 2021 • edited Loading

Overview

Examples

Hardware Docs Page

TODO

Nicell commented Jul 27, 2021 • edited Loading

Nicell commented Jul 27, 2021

petejohanson commented Jul 27, 2021

joelspadin Jul 30, 2021

Choose a reason for hiding this comment

petejohanson Aug 3, 2021

Choose a reason for hiding this comment

joelspadin Aug 3, 2021

Choose a reason for hiding this comment

Nicell left a comment

Choose a reason for hiding this comment

joelspadin commented Aug 3, 2021

petejohanson commented Sep 9, 2021

petejohanson commented Sep 9, 2021

Nicell left a comment

Choose a reason for hiding this comment

joelspadin Sep 11, 2021

Choose a reason for hiding this comment

joelspadin Sep 11, 2021

Choose a reason for hiding this comment

petejohanson commented Jul 27, 2021 •

edited

Loading

Nicell commented Jul 27, 2021 •

edited

Loading