Support KMK Firmware

[yoichiro]

As the title. I want to support KMK Firmware in Remap. My idea is below:

• Not use the VIA protocol to communicate between the Remap and the KMK Firmware.
• Instead, the Remap accesses to each file of KMK Firmware via File Access API provided by Web browser.
• The Remap parses the Python code which has key mapping data and translates them to key codes which are defined in Remap based on the QMK Firmware.
• The Remap generates the Python code according to the key codes and updates the key mapping file with them.

Probably, if I can implement a code set based on the idea above to Remap, users can customize key mappings on Remap UI without any touch for KMK Firmware core file set.

Basically, each keyboard designer who wants to support KMK Firmware, he/she intends to create two files: kb.py and main.py.

The main.py file has a keyboard.keymap variable. It has all key mapping data. Usually, it is written by Python.

To become the code parsing simply, the variable should be written by an individual file, I guess.

I also have an option which I choose a JSON file format to describe the key mapping data. But, this idea is not good, I guess. Because, when the JSON file is updated, it is not reloaded automatically, I guess.

If the file is written by the Python code and is updated by the Remap, it will be reloaded automatically by CircuitPython, I guess. Probably. I need to confirm whether this idea is correct or not.

In addition, I will be able to adopt the same idea for other firmware which can mount the filesystem and can access to each file via File Access API, I guess.

If KMK Firmware disables the auto-reload feature for py files, this idea will not be able to achieve the goal. But, currently the boot.py does not have the logic to turn off the feature.

[s-ol]

I'm also very new to KMK and CircuitPython, so I don't feel confident suggesting the "best" way to do it.

But I just wanted to mention that CircuitPython includes JSON encoding/decoding libraries out of the box, so it would not be an issue to use an existing JSON format. From the python side, it's possible to read non-python files on the mass storage device, so it should also be possible to reload a JSON config, as far as I can tell. The benefit of this approach is that you don't have to try and parse Python, which will never be able to work in 100% of cases.

[yoichiro]

@s-ol Thank you for the suggestion! Yes, I agree that the portability and parse-ability of the JSON format is very attractive. I think that it is possible to use the JSON format to store key mapping data in the MCU.

But, actually, there is both pros and cons for the JSON format and the Python format. Especially, I think that the representation of each key code is important in this case. I try listing up them here:

JSON format

• Pros
  • Parsing of the JSON format string is very easy and strict.
  • It is possibility to be able to use the key mapping file written by the JSON format for other firmwares as well, if I can define a standard format.
• Cons
  • It is necessary to translate the complex key code representation. In the Python code main.py, each key code is written by each constant value like KC.A which actually has two values like code: 4 and has_modifier: None. Also, there are more complex value structures like KC.LCTRL(KC.LGUI(KC.RGHT)). That is, each key code representation cannot be serializable in the JSON format easily.
  • If KMK Firmware upgrades and each key code is changed, each key code in the JSON file becomes incorrect.
  • I don't want to add codes to the KMK Firmware side, but it is necessary to add a logic to parse the JSON file. Also, the logic to translate each key code representation is needed.

Python format

• Pros
  • It is unnecessary to define any translation rule of the key code representation between the JSON format and the Python world. The KMK firmware can recognize the KC.A and KC.LCTRL(KC.LGUI(KC.RGHT)) directly without special logic.
  • It is unnecessary to treat any internal key code value. If KMK Firmware changes the key code values, it does not affect to the generated Python file which has key mapping data represented by the KC.* format.
• Cons
  • Actually, it is difficult to parse the Python format string than the JSON format

Generally, parsing the Python code is difficult than the JSON code. However, as the architecture image above, my idea does not include parsing the main.py. Instead, the Remap generate the remap.py file automatically, and it is unnecessary to edit the remap.py file by users. That is, the cons of the Python format does not become cons.

In this issue, it is important to store all key codes without any translation logics and any changes into the file. Also, even if KMK Firmware updates key code values, the file is necessary to keep it as valid.

[yoichiro]

I have just tested this idea in the real keyboard "Lunakey Pico" which has Raspberry Pi Pico and KMK Firmware is working.

First, I have created the "remap.py" file which has my key mappings like the following:

def create_keymap(KC):
  return [
    [KC.GESC, KC.Q, KC.W, KC.E, KC.R, KC.T, KC.Y, KC.U, KC.I, KC.O, KC.P, KC.BSPC, KC.MT(KC.TAB, KC.LCTRL), KC.A, KC.S, KC.D, KC.F, KC.G, KC.H, KC.J, KC.K, KC.L, KC.SCLN, KC.MT(KC.QUOT, KC.LCTRL), KC.LSFT, KC.Z, KC.X, KC.C, KC.V, KC.B, KC.N, KC.M, KC.COMM, KC.DOT, KC.SLSH, KC.MT(KC.MINS, KC.LSHIFT), KC.LALT, KC.LANG2, KC.MO(1), KC.SPC, KC.ENT, KC.MO(2), KC.LANG1, KC.RALT], [KC.BSLS, KC.CIRC, KC.EXLM, KC.AMPR, KC.PIPE, KC.DLR, KC.AT, KC.ASTR, KC.PLUS, KC.EQL, KC.PERC, KC.BSPC, KC.TRNS, KC.N1, KC.N2, KC.N3, KC.N4, KC.N5, KC.N6, KC.N7, KC.N8, KC.N9, KC.N0, KC.DQT, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.COLN, KC.LABK, KC.RABK, KC.QUES, KC.UNDS, KC.LGUI, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.MO(3), KC.TRNS, KC.LGUI], [KC.BSLS, KC.CIRC, KC.EXLM, KC.AMPR, KC.PIPE, KC.DLR, KC.AT, KC.ASTR, KC.PLUS, KC.EQL, KC.PERC, KC.BSPC, KC.HASH, KC.GRV, KC.LBRC, KC.RBRC, KC.LPRN, KC.RPRN, KC.PGUP, KC.HOME, KC.UP, KC.END, KC.TRNS, KC.DQT, KC.TRNS, KC.TILD, KC.TRNS, KC.TRNS, KC.LCBR, KC.RCBR, KC.PGDN, KC.LEFT, KC.DOWN, KC.RGHT, KC.TRNS, KC.TRNS, KC.LGUI, KC.TRNS, KC.MO(3), KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.LGUI], [KC.TRNS, KC.RESET, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.F1, KC.F2, KC.F3, KC.F4, KC.F5, KC.F6, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.F7, KC.F8, KC.F9, KC.F10, KC.F11, KC.F12, KC.TRNS, KC.TRNS, KC.VOLD, KC.VOLU, KC.MUTE, KC.TRNS, KC.LSFT(KC.LGUI(KC.PSCR)), KC.LSFT(KC.PSCR), KC.CAPS, KC.TRNS, KC.LCTRL(KC.LGUI(KC.RGHT)), KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS, KC.TRNS]
  ]

Second, I have replaced the existing key mappings definition in the "main.py" file with using above like the following:

...
import remap
...
keyboard.keymap = remap.create_keymap(KC)
...

After replacing them in my devices, these files work normally. Also, when editing the key mapping in the "remap.py" file, the KMK Firmware is reloaded automatically and the latest key mappings are applied.

Probably, when Remap updates the "remap.py" file, the updated content will be applied by auto-reloading the file by the KMK Firmware.

The current my idea is that it would be a better way that Remap loads the "remap.py" file and pares it and writes it after customizing the key mapping on the Remap.

[kdb424]

In this issue, it is important to store all key codes without any translation logics and any changes into the file. Also, even if KMK Firmware updates key code values, the file is necessary to keep it as valid.
I can say that we do our absolute best to avoid changing any values of internal keys. User defined keys can go hog wild, but we do our absolute best to leave these as "reserved" to not be toyed with. Where possible, even if keys aren't defined yet, we have a history of locking keycode values for later use.

Love the project, and wish I had more time to dedicate to code, but I'm keeping a mostly quiet eye on it. Hope this gets somewhere that you are happy with it!

[yoichiro]

@kdb424 Thank you for your advice. Basically, this idea is that Remap edits the file which has key mappings instead of the user. All users need to understand and follow key code changes of KMK Firmware. As same as it, the implementation of this idea in the Remap side needs to continue following them.

[yoichiro]

The draft of the user flow is the following:

1. A user opens the Remap. The user sees the new menu item "+ KEYBOARD WITH KMK FIRMWARE" like the following:

2. The user clicks its menu item. A directory picker UI is opened. The user chooses the directory of the device with KMK Firmware mounted as a drive.

3. The Remap finds a file named "remap.py" from the selected directory. If exists, the Remap detects the keyboard type by reading and parsing the file.

4. If the file is not found or the keyboard type could not be detected, the Remap shows the user the UI to select a keyboard type from the keyboard list supporting KMK Firmware. The user selects a keyboard type from the list.

5. The Remap reads and parses the "remap.py" file and load the key mappings. The Remap translates the key mappings information into each an internal key code value using the key code structure of QMK Firmware. Then, the Remap loads the keyboard definition JSON file from the server and renders it on the Remap page.

6. The user assigns and replaces each key mapping on the Remap.

7. The user clicks the "FLASH" button. The Remap translates each key code into an expression which is used in the KMK Firmware like "KC.A" and generate the key mapping array using Python format. The Remap updates the python code of the key mappings in the "remap.py" file. The KMK Firmware automatically reloads the "remap.py" file, and the new key mappings are applied.