Krux on Maix Amigo IPS

[odudex]

Jeff, thanks for the quick fix for the firmware.
Amigo has its own issues, but progress is being made

• Running ok
• Display ok (had to be mirrored than have to deal with flipped X coordinates)
• Camera ok
• 3rd button "Previous" ok

Next comes bigger fonts and touch (this won't be easy)

[odudex]

Fried my brain here trying to make larger fonts work, but I was not able to. The scrips seem to not be able to
handle large fonts. Here is what I think why:
Scripts and fonts.c seems to read one byte hex value for each line of pixels to draw the characters.
1 byte= 8 bits, each bit is 1 pixel.
A font character that is 8x14 will have 14 lines of 1 byte each.
Larger fonts, however, need more bytes for each line.
hexfill.py, for example, is not able to handle multiple bytes/line and will make all zeros
I tried a 12x24 font, modified hextodkz.py to make a 2Bytes x 24 matrix instead of actual 1Byte x 14, but it didn't work. I think font.c only handle 1 byte matrices.

[odudex]

Changes did help, font with whole byte widths work (ex 1632) but fonts with intermediary sizes don't (1224).
Some small bug in hexfill.py: font_byte_length = math.ceil(width / 8 * height) should be:
font_byte_length = math.ceil(width / 8) * height

[ghost]

I had a feeling intermediary sizes may not work... Can you file an issue for this?

[jdlcdl]

I'm glad that this old discussion is available and that I've found it.

Regarding ko-KR fonts, I find that Korean glyphs can fit into a square-ish bounding box and that they do not fit well in a tall rectangle. While prototyping for my amigo, I've tried making Korean fonts that are 12x24 (which is much too small), 32x24 (which seems really readable, but only for Korean, and appears uselessly double-spaced when switching to other languages), and finally 16x24 (still too small I suspect in Korean, but as wide as possible without expressing fonts as 3 bytes wide in font.c. This is still too much space between characters in other languages). I don't believe that a happy medium can be found, with regard to a common width:height ratio, between ko-KR fonts and existing supported languages; my guess is that it might only get worse for other written languages that krux will want to support in the future.

My attempts in mid-to-late September 2023 in this branch are here

I'm working with the assumption that in krux font format, all characters in a font.c's unicode array must be the same size (ie: same amount of bytes wide times same height), so the widest character in the font set defines the amount of bytes necessary for all other characters, even though those parts of the bitmap can be unset. Further, that font.c has a hard coded width that allows fonts to be trimmed horizontally (ie: For amigo, fonts are 2 bytes wide... they could be as big as 16x24 but making them 12x24 makes for tighter character spacing).

My last thoughts on solving for ko-KR and other wide-ish character sets in the future is to have flexibility in font.c so width is no longer hard coded and could be changed as new languages are selected (I imagine display needs to do this too). This way, if it turns out that 28 wide by 24 high is optimum for a particular language while other languages are best at 12x24, then font.c could still be built to have unicode with all characters 4 bytes wide (many with blank bitmaps on the right side) and take advantage of flexibility for each language to define the width so that they could be horizontally trimmed to display with proper spacing.

Side thought: Maybe fonts.c remains hardcoded for width but only represents a subset of all fonts, then switching languages on a particular device dynamically swaps out fonts? maybe users can select different font sizes like this too?

Notes I've kept along the way are here
Unfinished hacks I've used to visualize bdf/hex/kff are here

I suspect that you @jreesun will have invaluable insight into how this might best be accomplished, so I thank you in advance for your thoughts on this topic.

[ghost]

Love the thoroughness you bring to this project @jdlcdl !

You may have noticed that the languages listed in this issue all share something in common: they use the latin alphabet (with modifications of course). In other words, I knew asian and semitic languages would be difficult and chose to ignore the problem until someone like you came around :)

[We can continue to wait for someone to come around asking about support for arabic.]

I'm working with the assumption that in krux font format, all characters in a font.c's unicode array must be the same size (ie: same amount of bytes wide times same height)

Yes, that's correct. The fact they are the same size is used to index into the byte array in constant time. Here's the "spec" of the format if you missed it: https://github.com/selfcustody/krux/tree/main/firmware/font#format-spec

My last thoughts on solving for ko-KR and other wide-ish character sets in the future is to have flexibility in font.c so width is no longer hard coded and could be changed as new languages are selected (I imagine display needs to do this too)

Side thought: Maybe fonts.c remains hardcoded for width but only represents a subset of all fonts, then switching languages on a particular device dynamically swaps out fonts? maybe users can select different font sizes like this too?

Yes, we could change fonts.c so that it comes preloaded with several "BuildIn" fonts instead of one: https://github.com/selfcustody/MaixPy/blob/master/projects/maixpy_amigo_tft/compile/overrides/components/micropython/port/src/omv/img/font.c#L551-L553

Each translation JSON file could have a new "font" key specifying the font index to use, which can be referenced in display.py. This might necessitate changing the json structure to something like:

{
  "font": 0, // latin, or 1 for asian
  "translations": {
      "About": "Über", 
      ...
    }
}

We would need to expose the font_width(), font_height(), and font_load() functions in font.c as new functions on the lcd (py_lcd) module so display.py could call them.

Also, to be clear, I don't think each language should have its own unique font; ideally there would now just be two stored fonts (latin + asian) to avoid glyph redundancy and save on space, and each translation file would point to one or the other.

What do you think?

[jdlcdl]

On "What do I think?": I feel like I've hit a wall, having no experience and plenty of difficulty with c-code, so I would much rather follow and learn than to lead. Thank you for taking time to share these thoughts.

Also, I don't believe this is a pressing matter, yet. I'm on this topic only because of my own personal agenda to enable hardened/selfcustody individuals around the world. Still, I believe that a valuable translator can arrive, translate everything within days, and then be gone before a solution exists to show them what their diligent work looks like. I want to keep them interested and engaged until the next release to reach that new market, and maybe to get them invested like an initial ambassador to those newcoiners. A Korean bitcoiner has already expressed some interest, and I'd like to have given thoughts toward a solution to match the work they may deliver any day now. If that happens soon, I'll make sure that a korean-only krux branch exists to hook them in, buying time for a solution in a future release.

On redundancy of fonts, there are 2 issues that come to mind, both with solutions (I think):

1. Whenever translations don't exist for a localization, redundancy of at least the lower ascii set might help the user to understand and/or report the problem. But a better solution is not to have any missing translations.
2. Whenever a user is in Settings/Languages, they'll need to be able to read their language in their own language ("ko-KR" might be meaningless to some, the current translated slug "Locale" is helpful). But perhaps another solution is to use flag icons?

On having several BuildIn fonts instead of just one, so that there are multiple font structs in font.c, and so they could be loaded dynamically, I like this idea; I think this is closer to what I was thinking. Two thoughts come to mind:

1. Can we get two extra free wins in the way of 1) user-selectable-font-sizes and 2) easier maintenance by sharing font.c files across multiple projects/boards with similar screen sizes while we also solve for the current problem of how to display different font sets via being able to font_load() dynamically?
2. Is it really the case that any user will want a firmware bloated with all of the languages that krux deserves to be translated in (billions of households speaking hundreds of languages need a signing device), or is it more likely that most users will really just need firmware for their language, and maybe a couple others for their friends/family?
   Maybe it's best to cross that bridge when we get to it, because there is no pressing bloat issue at the moment. I want to at least keep this in mind while considering the possible solutions. Maybe fonts and languages are a "pack" that gets loaded dynamically so that font.c can remain as lean as possible.

p.s. My lack of C knowledge is simply fact, but it's not unsolvable. I will work my way thru this (was able to expose font_width() and font_height() to python yesterday after 2 failures, and starting to understand recent edits odudex made around flash.erase). I will appreciate any insight/solutions/methods others can share.

[ghost]

Awesome video, it's looking really good on the Amigo!

After watching that, I see some issues I'll need to address:

• Consolidating all 24 words into one screen if the display height is large enough
• Increasing the padding passed to draw_centered_text when rendering an xpub/key expression so it doesn't go all the way out to the end of the screen like that when there is plenty of vertical space to wrap sooner. Alternatively, this could potentially be solved more generally by having the default padding value come from the board config.
• Or, like you had wanted, if the font size is increased to better utilize the screen space these issues may go away naturally.

Does the Amigo not have a backlight? The reason I ask is because the QR code is noticeably dimmer on this device than on the M5StickV. With the M5StickV, its backlight is so bright that using pure white and black to render QR codes was causing glare problems, hence why I changed the default colors to be light and dark grey. On the M5StickV, this change is barely noticeable, but on the Amigo it looks strange...

Maybe for devices that don't have a backlight, I should default to white and black rather than the light/dark grey. Thoughts?

[odudex]

Sure some tweaking will be necessary on screens. I was already using some conditional formatting on Bit. But I think increase font size is a priority now.

Both Amigo and Bit have backlight, but a regular one. M5stickV seems to have a too powerful backlight, so the need of dimming.
I notice the QR code, even when black and white are used, are faded if compared with text. I think the QR module already does some dimming to avoid glare, so if you use shades of grey it may be too much dimming for regular displays like the Bit and Amigo.

[odudex]

Capturing touch data
Maixpy c driver don't work.. I found a partially working pure python driver, it crashed a lot, but after reducing sampling rate think it's stable now.

[ghost]

Thinking about how you could incorporate touchscreen support into Krux without massive changes to the code (and tests 😰)...

There are basically three usages of wait_for_button in Krux:

1. Select Selecting one item out of a list. Examples include menu navigation, keypad input, etc.
2. Prompt Answering a "Yes" or "No" kind of prompt. Examples include Done?, Print to QR?, Sign?, Continue?, etc.
3. Acknowledge Waiting for user acknowledgement before proceeding. Examples include displaying a QR code, the mnemonic, an error message, etc. until the user presses any button to continue.

The first one requires knowing where the items are on the screen, and therefore is probably best off being solved by the functions generating those items (Page.capture_from_keypad and Menu.run_loop). It's already the case that they handle the selection logic, so it should just be a matter of keeping track of the bounding box of each item and adding a new case to their code to allow selecting an item directly instead of paging to it.

To solve 1) and 3), I would:

1. Add a new BUTTON_TOUCH constant in input.py and update wait_for_button to return this if the user touches the screen at all
2. Add a new method on Input to query where the touch occurred, so something like touch_position() returning an (x, y) tuple

This way, the existing wait_for_button flow can remain intact, and the functions referenced above can additionally start checking for BUTTON_TOUCH and then fetch touch coordinates to figure out which particular "button" was selected. This also solves 3) because the return value in those cases are ignored since it's just waiting for any button press which would now include touches.

To solve 2) without too drastic of a codebase change, I would:

1. Create two new constants PROMPT_YES, and PROMPT_NO in pages/__init__.py, that have the same values as BUTTON_ENTER and BUTTON_PAGE, respectively.
2. Add a new Page.prompt method that can be called by subclasses of Page (i.e., Login and Home) and encapsulates the logic to handle being on either a button-based device like M5StickV or a touchscreen one like Amigo.

Something like:

def prompt(self, text, offset_y=None):
        if self.ctx.input.has_touch():
            if offset_y:
                self.ctx.display.draw_hcentered_text(text + "\n\n" + t("Yes\tNo"), offset_y=offset_y - 2 * font_size())
            else:
                self.ctx.display.draw_centered_text(text + "\n\n" +  t("Yes\tNo"))
        else:
            if offset_y:
                self.ctx.display.draw_hcentered_text(text, offset_y=offset_y)
            else:
                self.ctx.display.draw_centered_text(text)
        btn = self.ctx.input.wait_for_button()
        if btn !=  BUTTON_TOUCH:
            return btn
        pos = self.ctx.input.touch_position()
        return PROMPT_YES if pos[0] <= self.ctx.display.width() / 2 else PROMPT_NO

Then, the places in the code that act like prompts would be updated to call prompt() instead of their draw_centered_text and wait_for_button calls, but because PROMPT_YES and PROMPT_NO have the same values as BUTTON_ENTER and BUTTON_PAGE, the rest of the code could remain unchanged.

So, this:

        self.ctx.display.draw_hcentered_text(t("Proceed?"), offset_y=200)
        btn = self.ctx.input.wait_for_button()

would become:

        btn = self.prompt(t("Proceed?"), offset_y=200)

[odudex]

Menus, prompt method and acknowledge are working. For that I created a simplified code that uses y regions only.
Next are keypads, for that I'll have to use some kind of array indexing.

The font used here is a 1224, configured as 1624 to be possible to render.

[ghost]

Great work!

[odudex]

Keypads working
Now needs some more testing, fine tuning and running it on M5stickV.
Talking about M5stickV and code sharing, do you think it is bad to let the dark grey background rectangles on buttons on M5stickV? I think for the touch version it gets more intuitive for user to know what is going to happen according to where the screen is touched

[ghost]

Looking good. I like the decision to remove the button backgrounds from disabled letters. Is it possible to increase the (vertical) padding around the buttons to make them look larger?

I like the button backgrounds and think they make sense for touchscreen devices (also glad you chose a monochrome color!), but probably not for the M5StickV. I'm thinking it would just add clutter, but I'm open to seeing what it would look like first before deciding. Definitely not for the Yes/No prompt buttons, though, there just isn't enough space for those.

It would be worth exploring using a white button outline effect (just a 1px white line going around the button) across all devices to indicate which button is currently selected instead of the existing > or - characters. That might help to bring the two UIs closer together and get rid of the odd > and - characters on buttons on the Amigo.

[odudex]

Yes, it is possible to increase vertical padding. Now it is a font height multiple to facilitate calculations. Font could be a little bigger, or padding could added.
I Will keep M5stickV without backgrounds and Yes/No prompts.
I have already tested outlines in all devices, it looks nice. lcd module don't have draw_rectangle method so I used image module to draw menus with outline on selected item, but seemed to be RAM hunger so I gave up. I will try again using lcd's fill_rectangle and draw four rectangles to build the 1px outline.
We could remove the selection index indicators on Amigo, but I like the idea users could still use it with buttons even if touch screen stops working.

[odudex]

Exploring selection outline

[ghost]

That's looking really good!

[odudex]

Exploring touch menus:

• Dynamically adjusted menu items height
• Removing backgrounds as all screen is filled with active buttons
• Only showing outline selection in touch devices in case user press one of the physical buttons.

[odudex]

Applied the same style to Yes/No, but not dynamic height as I think it would need to much customization code.
Keypads got dynamic height, but I think toggling backgrounds makes the typing experience faster, more intuitive and comfortable.
Numeric keypad

Talking about keypads, I think we need an "abort" or "leave" key. What do you think? Should be a 3 or less letters word or symbol.

[ghost]

Would it be possible to use the same faint outline around the keypad characters (rather than the solid colored background now) that disappears when the character is inaccessible? Just to keep the UI looking consistent throughout.

I like what you did with having the white selection outline not be visible until first button press on touchscreen devices; makes sense to me!

Yes, I agree. Maybe call it "Esc" like on a keyboard? We may also want a "Clr" option (if it fits) for clearing the input string. I noticed that this may be necessary now with impossible letters + autocomplete if you've "gone too far" and want to go back a few letters.

[odudex]

The faint outline seems a good idea, I will try it out.
I like "Esc" and "Clr" too.
Also for next PR I fixed a small bug: "del" key wouldn't work after autocomplete returns something, now it works.
Something else we could do: Disable "Go" key until a valid word is on buffer

[ghost]

Okay, if you fix the Del key behavior so users can't get stuck then I don't think Clr is necessary to add.

Disabling the Go key may be a good idea, just need to make sure it works that way for all keypad input methods.

[odudex]

Del is fixed, but I did a Clr button to test, I think it can be helpful
Did not disabled Go yet
This is how it's going

[odudex]

Just an update: Greened all tests (that indeed was a pain).
Next: Increase coverage, than simulator

[odudex]

Had a use case where an Amigo bricked after a few hours of use. It reached me and I concluded it got physically and permanently damaged. I suspect it was a factory defect, or, it may be overheating.
Amigo uses a tiny child board for K210, so it has poor heat dissipation. Under normal use I noticed its possible to feel the heat from K210 through the plastic case. So I decided to underclock it to 300Mhz on my Maixpy fork. It runs cooler and don't have a performance drawback. @jreesun do you think it's a good idea? Or we should keep default 400MHz?

[ghost]

As long as the device remains stable while underclocked, I think that sounds like a great idea. Krux doesn't need the extra performance anyway.