Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bug: Incomplete display involving Chinese character #238

Closed
luwenjin opened this issue Oct 28, 2022 · 17 comments
Closed

Bug: Incomplete display involving Chinese character #238

luwenjin opened this issue Oct 28, 2022 · 17 comments
Labels
arduino bug Something isn't working display-tech
Milestone

Comments

@luwenjin
Copy link

luwenjin commented Oct 28, 2022

Case 1:

截屏2022-10-28 下午12 12 50

image

I set Unit name to "秒1" (a Chinese character + "1"), after uploading, the display only show the Chinese character, "1" is missing.
It seems only the first character is shown.
However If I only use ASCII for Unit name, everything is fine.

Case 2:

截屏2022-10-28 下午12 25 52

image

I set row in EnumMenuItem with Chinese character "1分钟", after uploading, the display only shows "1分", the last character is missing.
However If I only use ASCII for the rows, everything is fine.

I use this font

    installMonoBorderedTheme(renderer, MenuFontDef(u8g2_font_wqy12_t_gb2312b, 12), MenuFontDef(u8g2_font_wqy12_t_gb2312b, 12), true);
@luwenjin luwenjin changed the title Incomplete display involving Chinese character Bug: Incomplete display involving Chinese character Oct 28, 2022
@davetcc
Copy link
Collaborator

davetcc commented Oct 28, 2022

Could you please send an example generated *_menu.cpp file, I suspect this is a UTF8 issue between Java and embedded libraries.

@luwenjin
Copy link
Author

luwenjin commented Oct 28, 2022

ui.zip
This is the files from my project, and I've checked the code, the texts are complete.

You may also past these text containing Chinese characters to your experimental menuitems;
for enum rows: 1分钟
for unit name: 秒1

@davetcc
Copy link
Collaborator

davetcc commented Oct 29, 2022

So my initial suspicion was incorrect, I thought that there could have been a UTF-8 issue rendering the C++ file, this is not the case and the file is correctly generated with all the UTF-8 characters entact.

Two things strike me here straight away:

For unit, there is the possibility of the field being too short, it is only 5 bytes long with one byte needed for the null terminator. At least one of those characters you're using needs 3 bytes, and therefore may be very close on space. In the next version I'll NAME_SIZE_T and UNIT_SIZE_T build flags to allow the name and unit fields to be made longer. However, this does not apply to enum labels, they allow at least 20 characters.

Given that as far as I can tell, the string is correctly generated, I would actually turn on IoLogging's maximum debug level and see what's getting printed into the buffer that's sent to the graphics library text printing function. To do that set the following build flag:

-DIO_LOGGING_DEBUG=1

And then in your sketch main add the following include:

#include <IoLogging.h>

Lastly, add the following code at the start of setup:

serEnableLevel(SER_TCMENU_DEBUG, true);

Let's start here, and see what we find, if need be we'll add an extra log line right before the text printing to dump the contents of the string using the hex dump logging function.

@davetcc davetcc added question Further information is requested awaiting-feedback labels Oct 29, 2022
@luwenjin
Copy link
Author

I'll try that on Monday.

@luwenjin
Copy link
Author

Before I can try that, I have a clue.

The bug may be due to length calculation, in unicode 1分钟's length is 3,
but in UTF-8 encoding, it's length is 1+3+3 = 7, so the "钟" is out of range, and is missing on the display.

As I recall, I've tried to add more ASCII code after the last Chinese character, eg: 1分钟abcdefghi.
In this case, all the Chinese characters can be seen, but the last few ASCII characters are missing.

Hope this clue can help you find out the cause of the bug.

@davetcc
Copy link
Collaborator

davetcc commented Oct 30, 2022

You may be the first to use Kanji but I know there are users that have used Cyrillic and Arabic, they both ran into issues
with the display library (not tcMenu per se). IIRC both were fixable.

Let's get the debug output and take it from there, in the case that there is not enough detailed information from the present logging, a few lines of extra logging code may be needed.

@luwenjin
Copy link
Author

Designer UI

image

Display(selected the 2nd row)

image

Log

I started the MCU, nav to the enum menuitem 自动关机, entered edit mode, selected the 2nd row, exited edit mode.

13:35:33.495 > 417-IOA:Switches initialized (pull-up, int, encPoll)1 1 0
13:35:33.495 > 418-TCD:changeMenu: 17
13:35:33.502 > 418-TCM:root has changed
13:35:33.502 > 419-TCM:Recalculate display order, safe=0
13:35:33.507 > 429-TCD:Add title
13:35:33.507 > 429-TCD:Add manual id at row17 1
13:35:33.514 > 430-TCD:Add manual id at row42 2
13:35:33.514 > 430-TCD:Add manual id at row35 3
13:35:33.520 > 430-TCD:Add manual id at row40 4
13:35:33.520 > 441-TCD:Add manual id at row5 5
13:35:33.523 > 441-TCD:Add manual id at row71 6
13:35:33.523 > 442-TCM:Set items in menu (size, offs) 6 1
13:35:33.529 > 452-TCD:activate item 1
13:35:33.529 > 452-TCD:Change active (V, ID) 1 17
13:35:33.534 > 453-TCM:Set dimensions: 128 64
13:35:33.534 > 453-WRN:EEPROM Key NOT found FADE
13:35:33.633 > 555-TCD:changeMenu: 17
13:35:33.633 > 555-TCM:root has changed
13:35:33.633 > 555-TCM:Recalculate display order, safe=0
13:35:33.638 > 555-TCD:Add title
13:35:33.638 > 556-TCD:Add manual id at row17 1
13:35:33.644 > 557-TCD:Add manual id at row42 2
13:35:33.644 > 567-TCD:Add manual id at row35 3
13:35:33.649 > 568-TCD:Add manual id at row40 4
13:35:33.649 > 569-TCD:Add manual id at row5 5
13:35:33.655 > 569-TCD:Add manual id at row71 6
13:35:33.655 > 581-TCM:Set items in menu (size, offs) 6 1
13:35:33.660 > 581-TCD:activate item 1
13:35:33.666 > 581-TCD:Change active (V, ID) 1 17
13:35:33.782 > 703-TCD:Complete redraw
13:35:33.783 > 707-TCD:Drawing at: 0 128 18
13:35:33.783 > 709-TCD:intTx Soccer Bar  10 -1
13:35:33.788 > 711-TCD:draw item (pos,id,chg)1 17 1
13:35:33.788 > 711-TCD:Drawing at: 18 128 15
13:35:33.794 > 714-TCD:intTx 训练 >> 10 18
13:35:33.794 > 716-TCD:draw item (pos,id,chg)2 42 1
13:35:33.799 > 716-TCD:Drawing at: 34 128 15
13:35:33.805 > 721-TCD:intTx 灯光提示 >> 10 34
13:35:33.805 > 721-TCD:draw item (pos,id,chg)3 35 1
13:35:33.812 > 723-TCD:Drawing at: 50 128 15
13:35:33.812 > 737-TCD:intTx 倒计时 >> 10 50
13:35:36.136 > 3055-TCD:activate item 2
13:35:36.136 > 3056-TCD:Change active (V, ID) 2 42
13:35:36.160 > 3082-TCD:draw item (pos,id,chg)1 17 1
13:35:36.160 > 3084-TCD:Drawing at: 18 128 15
13:35:36.166 > 3087-TCD:intTx 训练 >> 10 18
13:35:36.166 > 3089-TCD:draw item (pos,id,chg)2 42 1
13:35:36.171 > 3090-TCD:Drawing at: 34 128 15
13:35:36.171 > 3095-TCD:intTx 灯光提示 >> 10 34
13:35:36.495 > 3414-TCD:activate item 3
13:35:36.496 > 3415-TCD:Change active (V, ID) 3 35
13:35:36.502 > 3424-TCD:Screen Row moved 1 2
13:35:36.502 > 3425-TCD:Drawing at: 0 128 18
13:35:36.502 > 3427-TCD:intTx Soccer Bar  10 -1
13:35:36.508 > 3428-TCD:draw item (pos,id,chg)2 42 1
13:35:36.513 > 3430-TCD:Drawing at: 18 128 15
13:35:36.513 > 3435-TCD:intTx 灯光提示 >> 10 18
13:35:36.519 > 3435-TCD:draw item (pos,id,chg)3 35 1
13:35:36.519 > 3439-TCD:Drawing at: 34 128 15
13:35:36.525 > 3442-TCD:intTx 倒计时 >> 10 34
13:35:36.525 > 3443-TCD:draw item (pos,id,chg)4 40 1
13:35:36.532 > 3456-TCD:Drawing at: 50 128 15
13:35:36.532 > 3459-TCD:intTx 秒表 >> 10 50
13:35:36.875 > 3793-TCD:activate item 4
13:35:36.875 > 3794-TCD:Change active (V, ID) 4 40
13:35:36.902 > 3824-TCD:Screen Row moved 2 3
13:35:36.902 > 3825-TCD:Drawing at: 0 128 18
13:35:36.902 > 3827-TCD:intTx Soccer Bar  10 -1
13:35:36.908 > 3829-TCD:draw item (pos,id,chg)3 35 1
13:35:36.913 > 3830-TCD:Drawing at: 18 128 15
13:35:36.913 > 3834-TCD:intTx 倒计时 >> 10 18
13:35:36.919 > 3835-TCD:draw item (pos,id,chg)4 40 1
13:35:36.919 > 3837-TCD:Drawing at: 34 128 15
13:35:36.927 > 3850-TCD:intTx 秒表 >> 10 34
13:35:36.927 > 3851-TCD:draw item (pos,id,chg)5 5 1
13:35:36.932 > 3852-TCD:Drawing at: 50 128 15
13:35:36.932 > 3855-TCD:intTx 设置 >> 10 50
13:35:37.303 > 4222-TCD:activate item 5
13:35:37.303 > 4223-TCD:Change active (V, ID) 5 5
13:35:37.329 > 4251-TCD:Screen Row moved 3 4
13:35:37.329 > 4252-TCD:Drawing at: 0 128 18
13:35:37.329 > 4254-TCD:intTx Soccer Bar  10 -1
13:35:37.335 > 4255-TCD:draw item (pos,id,chg)4 40 1
13:35:37.340 > 4257-TCD:Drawing at: 18 128 15
13:35:37.340 > 4260-TCD:intTx 秒表 >> 10 18
13:35:37.346 > 4260-TCD:draw item (pos,id,chg)5 5 1
13:35:37.346 > 4272-TCD:Drawing at: 34 128 15
13:35:37.352 > 4275-TCD:intTx 设置 >> 10 34
13:35:37.352 > 4275-TCD:draw item (pos,id,chg)6 71 1
13:35:37.357 > 4277-TCD:Drawing at: 50 128 15
13:35:37.357 > 4279-TCD:intTx Debug >> 10 50
13:35:38.434 > 5355-TCM:Menu edit end 17
13:35:38.434 > 5356-TCM:NavigateTo pushed 0 17 17
13:35:38.434 > 5356-TCD:changeMenu: 50002
13:35:38.439 > 5358-TCM:root has changed
13:35:38.439 > 5358-TCM:Recalculate display order, safe=0
13:35:38.445 > 5368-TCD:Handling back item50002
13:35:38.450 > 5370-TCD:Add manual id at row16 1
13:35:38.450 > 5370-TCD:Add manual id at row56 2
13:35:38.456 > 5370-TCD:Add manual id at row20 3
13:35:38.456 > 5371-TCD:Add manual id at row55 4
13:35:38.461 > 5382-TCD:Add manual id at row57 5
13:35:38.461 > 5382-TCD:Add manual id at row9 6
13:35:38.467 > 5384-TCM:Set items in menu (size, offs) 6 0
13:35:38.472 > 5384-TCD:activate item 0
13:35:38.472 > 5394-TCD:Change active (V, ID) 0 50002
13:35:38.478 > 5396-TCD:Complete redraw
13:35:38.478 > 5399-TCD:Drawing at: 0 128 18
13:35:38.478 > 5403-TCD:intTx 设置 [..] 10 -1
13:35:38.483 > 5404-TCD:draw item (pos,id,chg)1 16 1
13:35:38.489 > 5404-TCD:Drawing at: 18 128 15
13:35:38.489 > 5405-TCD:item: LED亮度 15 18
13:35:38.495 > 5418-TCD:draw item (pos,id,chg)2 56 1
13:35:38.495 > 5420-TCD:Drawing at: 34 128 15
13:35:38.501 > 5421-TCD:item: 感应距离 15 34
13:35:38.501 > 5425-TCD:draw item (pos,id,chg)3 20 1
13:35:38.506 > 5427-TCD:Drawing at: 50 128 15
13:35:38.506 > 5428-TCD:item: 音量 15 50
13:35:39.334 > 6256-TCD:activate item 1
13:35:39.334 > 6257-TCD:Change active (V, ID) 1 16
13:35:39.334 > 6257-TCD:Drawing at: 0 128 18
13:35:39.340 > 6260-TCD:intTx 设置  10 -1
13:35:39.340 > 6262-TCD:draw item (pos,id,chg)1 16 1
13:35:39.345 > 6263-TCD:Drawing at: 18 128 15
13:35:39.351 > 6264-TCD:item: LED亮度 15 18
13:35:39.981 > 6903-TCD:activate item 2
13:35:39.982 > 6904-TCD:Change active (V, ID) 2 56
13:35:39.982 > 6905-TCD:draw item (pos,id,chg)1 16 1
13:35:39.987 > 6907-TCD:Drawing at: 18 128 15
13:35:39.993 > 6908-TCD:item: LED亮度 15 18
13:35:39.993 > 6912-TCD:draw item (pos,id,chg)2 56 1
13:35:39.998 > 6913-TCD:Drawing at: 34 128 15
13:35:39.998 > 6914-TCD:item: 感应距离 15 34
13:35:40.391 > 7310-TCD:activate item 3
13:35:40.391 > 7311-TCD:Change active (V, ID) 3 20
13:35:40.400 > 7322-TCD:Screen Row moved 1 2
13:35:40.400 > 7323-TCD:Drawing at: 0 128 18
13:35:40.400 > 7325-TCD:intTx 设置  10 -1
13:35:40.406 > 7327-TCD:draw item (pos,id,chg)2 56 1
13:35:40.411 > 7328-TCD:Drawing at: 18 128 15
13:35:40.411 > 7329-TCD:item: 感应距离 15 18
13:35:40.417 > 7333-TCD:draw item (pos,id,chg)3 20 1
13:35:40.417 > 7335-TCD:Drawing at: 34 128 15
13:35:40.423 > 7335-TCD:item: 音量 15 34
13:35:40.423 > 7349-TCD:draw item (pos,id,chg)4 55 1
13:35:40.428 > 7350-TCD:Drawing at: 50 128 15
13:35:40.428 > 7351-TCD:item: 自动关机 15 50
13:35:40.923 > 7842-TCD:activate item 4
13:35:40.923 > 7843-TCD:Change active (V, ID) 4 55
13:35:40.923 > 7850-TCD:Screen Row moved 2 3
13:35:40.928 > 7851-TCD:Drawing at: 0 128 18
13:35:40.928 > 7854-TCD:intTx 设置  10 -1
13:35:40.934 > 7855-TCD:draw item (pos,id,chg)3 20 1
13:35:40.939 > 7856-TCD:Drawing at: 18 128 15
13:35:40.939 > 7857-TCD:item: 音量 15 18
13:35:40.939 > 7859-TCD:draw item (pos,id,chg)4 55 1
13:35:40.945 > 7862-TCD:Drawing at: 34 128 15
13:35:40.950 > 7863-TCD:item: 自动关机 15 34
13:35:40.950 > 7876-TCD:draw item (pos,id,chg)5 57 1
13:35:40.956 > 7878-TCD:Drawing at: 50 128 15
13:35:40.956 > 7882-TCD:intTx 电量校准 >> 10 50
13:35:41.711 > 8633-TCM:Edit start 55
13:35:41.719 > 8642-TCD:Drawing at: 0 128 18
13:35:41.720 > 8644-TCD:intTx 设置  10 -1
13:35:41.720 > 8646-TCD:draw item (pos,id,chg)3 20 0
13:35:41.725 > 8646-TCD:Drawing at: 18 128 15
13:35:41.731 > 8647-TCD:item: 音量 15 18
13:35:41.731 > 8649-TCD:draw item (pos,id,chg)4 55 1
13:35:41.736 > 8651-TCD:Drawing at: 34 128 15
13:35:41.736 > 8652-TCD:item: 自动关机 15 34
13:35:41.742 > 8665-TCD:draw item (pos,id,chg)5 57 0
13:35:41.742 > 8667-TCD:Drawing at: 50 128 15
13:35:41.750 > 8671-TCD:intTx 电量校准 >> 10 50
13:35:42.443 > 9365-TCD:draw item (pos,id,chg)4 55 1
13:35:42.444 > 9367-TCD:Drawing at: 34 128 15
13:35:42.450 > 9368-TCD:item: 自动关机 15 34
13:35:44.389 > 11311-TCM:Menu edit end 55
13:35:44.389 > 11312-TCM:Set items in menu (size, offs) 6 4
13:35:44.395 > 11312-TCD:activate item 4
13:35:44.395 > 11313-TCD:Change active (V, ID) 4 55
13:35:44.400 > 11320-TCD:Drawing at: 0 128 18
13:35:44.400 > 11322-TCD:intTx 设置  10 -1
13:35:44.406 > 11324-TCD:draw item (pos,id,chg)3 20 0
13:35:44.406 > 11324-TCD:Drawing at: 18 128 15
13:35:44.412 > 11325-TCD:item: 音量 15 18
13:35:44.412 > 11338-TCD:draw item (pos,id,chg)4 55 1
13:35:44.417 > 11339-TCD:Drawing at: 34 128 15
13:35:44.417 > 11340-TCD:item: 自动关机 15 34
13:35:44.423 > 11346-TCD:draw item (pos,id,chg)5 57 0
13:35:44.428 > 11346-TCD:Drawing at: 50 128 15
13:35:44.428 > 11349-TCD:intTx 电量校准 >> 10 50

@davetcc
Copy link
Collaborator

davetcc commented Nov 1, 2022

I'm going to need to try and debug this, I have a Nucleo with ST-Link debugger that has an OLED attached, which display library are you using (u8g2)? Is the font standard?

@luwenjin
Copy link
Author

luwenjin commented Nov 1, 2022

I use u8g2 library, the font is MenuFontDef(u8g2_font_wqy12_t_gb2312b, 12) and is included in u8g2 library.

@davetcc davetcc added bug Something isn't working arduino display-tech and removed question Further information is requested awaiting-feedback labels Nov 5, 2022
@davetcc
Copy link
Collaborator

davetcc commented Nov 14, 2022

I've Isolated it into a simple sketch and managed to work out what's wrong, I've fixed the U8G2 renderer so you can switch on UTF-8 encoding from a flag during code generation.

Editing UTF-8 text may be a different proposition, one which would probably be possible for you to implement if needed once I add custom multi-edit soon.

@davetcc davetcc added this to the 3.0.0 milestone Nov 14, 2022
@davetcc
Copy link
Collaborator

davetcc commented Nov 14, 2022

So I've also checked TFT_eSPI and UTF8 should work there too without any changes on our side, it just seems like the UTF8 library attribute would need to be turned on.

I think for AdafruitGFX it would be better if I fixed the font renderer I have for frame buffers to work with Unicode fonts, and then handle it that way. Rather than rely on the fork that seems to use malloc in the main print loop.

@luwenjin
Copy link
Author

luwenjin commented Nov 14, 2022

How about making UTF-8 the default encoding ? For at least 3 reasons:

  1. ASCII is a subset of UTF-8 encoding, so it is not likely to affect English characters' display.
  2. Most modern OS use UTF-8 as it's default encoding.
  3. Less config to the library, less complexity.

@davetcc
Copy link
Collaborator

davetcc commented Nov 14, 2022

The problem is that to the best of my knowledge based on the youtube video watching, there's at least four thousand and maybe as high as ten thousand users of tcMenu. I just don't want to break existing sketches, and UTF8 support can be turned off by the user in U8G2 too, meaning it would break things. By default the designer always outputs UTF8, Arduino on most boards compiles using UTF8, but for the graphics libraries, its a bit more hit and miss:

  • Adafruit, it needs a fork of the lib, the fork does a malloc during print. I would not put that into production for that reason unles it was on a very, very large board.
  • U8G2, the support can be turned off the by the user, leading to drawing problems (and issues here or in the forum).
  • TFT_eSPI seems the best of all, it looks to be supported out of the box, and only needs a user toggle to enable it.

The flags will only be for U8G2 and Adafruit.

@luwenjin
Copy link
Author

I see 👍

@davetcc
Copy link
Collaborator

davetcc commented Nov 28, 2022

If you take a look at https://github.com/davetcc/tcUnicodeHelper you'll see what I've been working on. Based on some old code I'd written for mbed, it now is able to render Unicode properly regardless of the display technology, and even has a new screen in designer to create custom fonts with complete control over the ranges.

I've tested it with up to 4 character long sequences.

It will be available as an option for all graphical displays in TcMenu 3.0

@davetcc
Copy link
Collaborator

davetcc commented Dec 1, 2022

So the new option in the designer will allow for font creation, initially it will be quite basic:

image

You'd then switch on TcUnicode in code generator (which then limits the font dialog to only allow the right font types to be created.

image

And all the way through you can pick between TcUnicode and AdafruitGFX fonts (both for export from UI, and loading on the device). I've got a fair bit of documentation ready to go at the same time as the release.

@davetcc
Copy link
Collaborator

davetcc commented Dec 1, 2022

image

and here is the output on the display. The title is rendered using an AdafruitGFX font using TcUnicode renderer, and the items are rendered using a tcUnicode font with Latin, Latin extensions and Cyrillic enabled. If you take a look at either the arduino32 pico example or the mbed frame buffer example, these two have been converted.

What's more, the library is even usable standalone, without TcMenu, and it allows Adafruit_GFX users to work with Unicode without needing any special forks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arduino bug Something isn't working display-tech
Projects
None yet
Development

No branches or pull requests

2 participants