ZMK Behavior Editing

[nickcoutsos]

Regarding the creation and customization of existing and new behavior definitions within the Keymap Editor interface.

TL;DR

Supporting ZMK's pre-defined behaviors has required manually defining the parameters expected for binding cells (e.g., that layer-tap expects a layer index and a keycode), but this isn't feasible for staying on top of new behaviors introduced by ZMK, or customizations defined in user keymaps (like custom hold-taps). Creating custom editors for each behavior is also not ideal, but making something entirely data-driven has its own challenges.

In more detail

Each behavior is described schemas in the ZMK codebase. From this we can see what properties are available and their types, as well as the number of binding cells. This level of detail is, I think, just what's required for Zephyr, and isn't enough on its own explain how to come up with the appropriate values.

As an example, the mods property (in zmk,behavior-caps-word and zmk,behavior-mod-morph) is described as an int but the fact that it's a collection of constants describing modifiers from the HID spec joined together as a binary union is something you'd only get by reading ZMK documentation or the actual source code implmenting that behavior. The point being that this requires some amount of human planning, but hopefully this can be done in a way that makes things more manageable.

My goal right now is to keep things modular so that custom editors for specific property types are reusable across different behaviors and to ensure that there's always a fallback to editing as a raw value or else ignoring so that even unrecognized behaviors (something new that I haven't had a chance to review yet, or something like mouse emulation that isn't in ZMK's main branch yet) can still be set up outside of the app without causing conflicts.

Roadmap

• Define parameter types for built-in behaviors
• Resolve parameter types for new behaviors that reference other behaviors in a bindings property
• Bare-bones support for editing raw property values of a devicetree node
• Schema-based behavior editing with defined read-only and required fields based on the behavior's compatible property.
• Handling for default property values
• Ability to delete optional properties
• Special-purpose editor components for various field types
  • a mods selector for zmk,behavior-caps-word, and zmk,behavior-mod-morph that accepts 0 or more values from the list of HID modifier keys and serializes them as a union.
  • a bindings selector that accepts only behaviors (for hold tap and others)
  • a bindings selector that accepts behaviors with binding cells (for tap dance and others)
  • improvements to the key position selector (used in combo editing) to drag-select larger sets of keys -- this seems important for positional hold-tap setups for homerow mods
• Validation for input fields
• Support simpler node/reference/label editing
• Share more behavior code between the API and browser application. Right now all of the devicetree parsing happens only in the backend, so even if you can use the app to create a custom hold-tap behavior with the raw value editor, it won't know what the bind parameters are until it is saved and retrieved. In the (much) longer term I should look at refactoring some of the code into packages but I'll probably start by symlinking or something.
• Behavior customizer -- different from the behavior editor where you might create "new" variations of known behaviors like a hold-tap, this would let you re-define properties of existing behaviors like the &caps_word's mods list, or &mt's tapping-term-ms.

[nickcoutsos]

Some of the groundwork for this is in place already. If you're using features like mouse emulation which isn't merged into ZMK yet you may see that the behaviors appear in the behavior selector (if they're defined in your keymap file directly) and that even though there isn't a proper picker for its parameters you can view/modify raw values.

The "raw" behavior editor was dark-launched recently. A lot had to happen just to get to this point so I haven't been able to test it much yet, but even without others using it directly I want to make sure that nothing else was broken unexpectedly. Once I'm able to get a better look at things I'll make the option to use behavior editing a user-facing option.

[nickcoutsos]

The switch from editing "raw" property values to parsed (in situations where I can confidently parse a property value into reasonable types) was a challenge, but after defining a few value extraction functions based on ZMK's documentation I got to a point where I could have specific editing components for basic types.

I spent some time this weekend fleshing this out with some customizations.

The hold-trigger-key-positions property for hold-tap is of type array (of integers) but in an overlay for the schema I specify a "helper" that reuses the same key selector component in the Combo Editor since the underlying values are expressed in the same way.

I also created a component to process the mods lists in mod-morph and caps-word behaviors. These properties are a "list" of modifiers expressed as a boolean union, presented in the UI for now as a list of checkboxes. This isn't a specific "helper" like the key positions because the idea of selecting one or more items from a pre-defined list is generic and in this case it's reasonable enough to present that list of choices directly to the user. In the future I might add a little more nuance to this by doing something like looking at the length of the list to decide if it should be presented more concisely.

I think the next thing I'll look at is a bindings picker since I'll get to reuse some UI components from the keybinds.

[nickcoutsos]

It's been a little while since I've touched this, but it hasn't been forgotten. Once I had basic behavior editing functionality in place I had to pause that work to focus on performance.

Parsing speed

Devicetree searching
During my own testing and reviewing server logs I noticed that parsing a keymap file was taking an unusually long time. There were a lot of factors here but mostly this relates to how I try and understand the general layout of a keymap immediately after parsing the devicetree. Although conventionally keymaps will look the same...

/ {
  behaviors {...};

  macros {...};

  combos {...};

  keymap {...};
};

I can't rely on that. These groups of nodes can be at any depth in the tree, and because each behavior and macro node specifies its own compatible property, they technically don't have to exist as part of a group. To deal with this I perform these lookups by searching through the tree for nodes with certain compatible property values. The problem with this is that it searches the entire tree. It was also searching the tree depth-first, which means once it starts looking at things like the keymap node, it will dig into every layer and every binding and every behavior/identifier listed for no reason.

I've already re-implemented this as a breadth-first search to take advantage of conventions without relying on them. I can further refine this by giving myself a way to rule out certain nodes during traversal. I'm looking for behavior nodes, which can be just about anywhere, but if I encounter compatible = "zmk,combos"; or `compatible = "zmk,keymap"; I know that digging deeper won't help me.

Update buffering
Another performance issue came from all of the updates that happen when saving changes to a keymap. Whenever I want to make a change to the devicetree (such as adding a property to a behavior node), then any further changes (like adding another property to that same node) rely on the new state of the tree. Right now every change I make triggers a re-parse of the tree which is a performance hit on its own, but it also means invalidating what I know about the structure and requires re-doing those lookups. I have been working on buffering these edits to minimize re-parsing the tree. I haven't tested this enough yet to deploy, but in some of my basic comparisons I'm seeing a parse and update of a keymap taking a quarter of the time it was taking without buffering.

Edge-cases/unconventional device trees
There's one more area where I'd like to improve performance, and it relates to how much effort I put into supporting any kind of keymap that ZMK can compile. Some time ago I encountered some keymaps that have multiple root nodes, like:

/ {
  behaviors {...};
};

/ {
  combos {...};
};

/ {
  keymap {...};
};

which ZMK (or Zephyr?) will accept and merge into a single node. As this was technically acceptable I wanted to make sure that the keymap editor would be able to handle it, so the parser will first look for any root nodes in the file, and then search each of them for more specific nodes. You can see how this gets out of hand. Being able to rule out these edge cases would allow for much more targeted searches and improve performance significantly.

Somewhat related to this, I've seen errors come up from keymaps that use custom macros to simplify things like homerow mods, or zmk-nodefree to abstract devicetree syntax altogether. I think these are both good ways to simplify code and improve readability and code reuse, but they're problematic for an editor where the code is as much the end product as it is the storage medium. Pre-processing these macros into devicetree syntax that the editor can already parse is one thing, but spitting out keymap code that uses these macros as intended would require building the editor around those concepts specifically.

Long story short, I'm recognizing that as nice as it is to be flexible it shouldn't be at the expense of the more typical use cases. I've added some sanity checks to look out for use of the node-free macros and warn the user of the compatibility issues, and I can do the same for C macros to alias actual key bindings or unusual device tree structures.

Memory leaks

At the same time, for a little while now, I've been having memory issues with the application causing increasingly frequent crashes. This wasn't a huge issue because the backend is largely stateless and as soon as the process is back up and running the app continues functioning. Still, not ideal.

After a lot of digging I found out that, thankfully, the leaks were less of an issue of writing code that can't be garbage collected and more to do with improper use of Tree Sitter. This really just boils down to using tree.delete() when I'm finished with a tree object. I'm not an expert in web assembly, but I suspect that when you have data that lives in "both worlds" you can't clean up allocated memory automatically when objects are garbage collected. Long story short, anywhere I'm having the parser give me a syntax tree I need to use its .delete() method when I'm done, and suddenly I'm seeing memory usage stay below 100MB instead of steadily increasing with every keymap fetch/update.

[nickcoutsos]

Since improving performance the other week I've been putting some effort into getting the codebase more organized. The next steps for behavior editing will require utilizing some of the existing functionality in the frontend (so that if you do something like create a new behavior based on hold-tap the app can immediately determine parameters and allow you to bind the appropriate values) and I've started breaking sections of the codebase into chunks with clearer boundaries.

Some of this is already being incorporated into the web app behind the scenes. I have a sub-package for the data that gets extracted from the ZMK repo and augmented (behaviors and keycodes, and also the list of supported ZMK keyboards), and the web application bundles this same package so that its no longer necessary to query the API for what is fairly static data.

There's also a sub-package with my helper code for parsing/mutating devicetree documents and while I'm not using this in the browser yet I've had to spend a day or two just trying to make sense of that landscape. web-tree-sitter references node packages that aren't being polyfilled by webpack anymore, and my create-react-app bootstrapped app wouldn't allow me to set the appropriate configuration to handle the problem. I looked at craco, react-app-rewired, and vite as alternatives, but in the end I just used createapp.dev to generate a simplified webpack configuration. It was awful but it's another step forward.

Right now I'm going back to shore up some of my sorely lacking test coverage because I'm very likely to break things as I try and find better ways to organize them.

[nickcoutsos]

Finally making some real progress. I had to take some time away from the project, but I've been able to slowly separate some of the complex logic to help share code between the server and application. I've also looked at areas of inconsistency in how different parts of the system handle behavior properties.

Having taken care of that, it is now possible to create a new behavior in the frontend application, and immediately bind it with the correct parameter types. I have not tested this exhaustively, so here's hoping nothing has broken.

There are still limitations to this, however. It is currently possible to create a new behavior such as a hold-tap with &mo and &sl bindings and determine that two layer parameters are needed, and it is also possible to create a behavior that uses a custom behavior as one of its bindings resolving those parameters as needed. The problem right now is that modifying a referenced behavior will not currently update its parameters.

This isn't exactly a new problem. Even from the beginning there have been inter-dependencies like binding a layer index as a parameter to &mo or any other behavior. Deleting or re-arranging layers will not automatically update that parameter. Likewise, renaming a macro will not automatically search for references to it throughout other macro/combo/keymap bindings to update them as well. That will have to come later, and it won't be a whole lot of fun.