Porting to ESP-IDF with C++ support

[v1993]

Hello, I'm interested in porting this great library to ESP32 without Arduino compatiblity layer. However, for me it seems that there is a problem with cppsrc/U8x8lib.cpp: it use arduino specific stuff.

Is it possible to add support for other platform in it or it isn't required for C-side calls implementations and headers will be enough (i. e. not build this file at all)?

Is it doing only low-level stuff which I can do in any plain C file, or some things which are reqired to get C++ working too?

[olikraus]

U8g2 itself is a pure C library. For Arduino however it calls the C++ Arduino HAL and adds a C++ lib API wrapper. Both, Arduino wrapper and HAL are put together in the same c++/h files. As a consequence: If you want the C++ U8g2 API but not the C++ Arduino HAL, then some code needs to be copied out the existing C++ code.

I am not 100%, but I think this has been done by the nodemcu people already: https://nodemcu.readthedocs.io/en/dev-esp32/modules/u8g2/

In general porting instructions are here: https://github.com/olikraus/u8g2/wiki/Porting-to-new-MCU-platform

[v1993]

Thank you. Now I think about architecture: is it good idea to place ESP32 specific stuff in sys/esp-idf? It will be included from CMakeLists.txt or component.mk. I also plan adding config (for ESP32 menuconfig menu) for 16bit mode and fonts to make u8g2 feel like nice component.

Also, where can I find pure C exmples? I feel like there is a lack of them.

[olikraus]

Now I think about architecture: is it good idea to place ESP32 specific stuff in sys/esp-idf? It will be included from CMakeLists.txt or component.mk

I have no idea about this. CMakeLists.txt or component.mk in the u8g2 are part of a pull request. I can not tell you something about these files.

Also, where can I find pure C exmples?

Have a look here: https://github.com/olikraus/u8g2/tree/master/sys
All directories except "arduino" will contain pure c examples.

Here is a "hello world" example for the STM32L031:
https://github.com/olikraus/u8g2/blob/master/sys/arm/stm32l031x6/u8x8_test/main.c

Once I created a blog entry also:
https://drolliblog.wordpress.com/2017/11/02/stm32l031x6-eval-board/

I feel like there is a lack of them.

I personally feel, that I created more docs and examples than many other Arduino libs...

[v1993]

Thanks you again and sorry for blaming you about no docs. Your Arduino/C++ reference is just great! I just had some problems about pure C documentation.

[olikraus]

:-)

Let me know if I can help you with u8g2 related problems.

[v1993]

Now I think about architecture: is it good idea to place ESP32 specific stuff in sys/esp-idf? It will be included from CMakeLists.txt or component.mk

I have no idea about this. CMakeLists.txt or component.mk in the u8g2 are part of a pull request. I can not tell you something about these files.

There are no problems with changing those files, just wonder is sys/platformname a right place to store platform-specific stuff (I/O frunctions for ESP32 in this case).

[olikraus]

There are no problems with changing those files, just wonder is sys/platformname a right place to store platform-specific stuff (I/O frunctions for ESP32 in this case).

You mean below u8g2? Well sure, it you like to do so and send a corresponding pull request then I can accept this.

[v1993]

Thank you! Another question: how presice should be all the U8X8_MSG_DELAY_* stuff should be at waiting?

There are three options: use vTaskDelay (round time to ticks, pretty rough), ets_delay_us for busywait correct time (much better at timing, but not let anything else run while waiting), or ets_delay_us with disabeling interrupts with portDISABLE_INTERRUPTS/portENABLE_INTERRUPTS (100% correct timing, but extremely bad at long and not that presice waitings due to not be able to handle interrupts).

I suggest using vTaskDelay on ≥ 10ms delays and ets_delay_us on smaller scale. Are there any parts which should be more accurate at timing (so disable interrupts)? If so, it's still better to disable them in actual calling code and not callback.

Thanks.

[v1993]

Ok, I took a look at files.

/* generated code (codebuild), u8g2 project */

Where code generator and template files are? It looks like I'll have to add ESP32 support there.

My current idea it to leave csrc as is and place EPS32 stuff in some other directory, like sys/esp-idf (and auto generated stuff there too).

[olikraus]

Another question: how presice should be all the U8X8_MSG_DELAY_* stuff should be at waiting?

It always can be more than requested, so: It does not need to be precise.

Are there any parts which should be more accurate at timing (so disable interrupts)?

I think the only sensitive message is the I2C delay. But this is only required for software emulated I2C.

Where code generator and template files are?

You probably do not need to take care on the code generator. "codebuild" only connects the displays with the communication interfaces: It defines which types of interfaces are available for which display.

The hardware specific files are either in the sys//common directory or (an exception) for Arduino in the u8g2/cppsrc directory.

My current idea it to leave csrc as is

Yes please. "csrc" should not contain any hardware specific code.

[v1993]

Are Arduino display C++ binding auto generated? If yes, because I want to support them on ESP-IDF too, it will require tweaking code generator.

Am I right?

My goal is to provide C++ bindings compatible with Arduino interface too, because C++ is commonly used on ESP32 too.

[olikraus]

Are Arduino display C++ binding auto generated?

If you mean the constructor classes here, then yes:
https://github.com/olikraus/u8g2/blob/master/cppsrc/U8g2lib.h#L404

[v1993]

I also want to auto generate bindings for ESP-IDF to make migration/usage as comfortable as possible. Therefore, I have to tweak generator, right?

[olikraus]

Or you create a copy of the codebuild Code and add your modifications there.

[v1993]

To be fair, current code generator looks like hell for me ☺.

I'll look at it when get pure C working.

Do you have anything againist re-writing in in something non-C? I'm good (enough to write such thing) with C++ and lua.

[v1993]

Can you take a look an my test implementation for gpio_and_delay, please?

uint8_t u8x8_gpio_and_delay_espidf(u8x8_t *u8x8, uint8_t msg, uint8_t arg_int, U8X8_UNUSED void *arg_ptr)
{
    uint8_t i;
    switch(msg)
    {
    case U8X8_MSG_GPIO_AND_DELAY_INIT:
        for( i = 0; i < U8X8_PIN_CNT; i++ )
            if ( u8x8->pins[i] != U8X8_PIN_NONE )
            {
            bool success;
                if ( i < U8X8_PIN_OUTPUT_CNT )
                {
					success = u8x8_setpinoutput_espidf(u8x8->pins[i]);
                }
                else
                {
					success = u8x8_setpininput_espidf(u8x8->pins[i]);
                }
                if (!success) return 0;
            }

        break;

    case U8X8_MSG_DELAY_NANO:
        ets_delay_us(arg_int==0?0:1);
        break;

    case U8X8_MSG_DELAY_10MICRO:
        /* not used at the moment */
        break;

    case U8X8_MSG_DELAY_100NANO:
        /* not used at the moment */
        break;

    case U8X8_MSG_DELAY_MILLI:
        vTaskDelay(arg_int);
        break;
    case U8X8_MSG_DELAY_I2C:
        /* arg_int is 1 or 4: 100KHz (5us) or 400KHz (1.25us) */
        ets_delay_us(arg_int<=2?5:2);
        break;
    case U8X8_MSG_GPIO_I2C_CLOCK:
    case U8X8_MSG_GPIO_I2C_DATA:
		auto pin = u8x8_GetPinValue(u8x8, msg);
        if ( arg_int == 0 )
        {
            if (
            (!u8x8_setpinoutput_espidf(pin)) ||
            (gpio_set_level(pin, 0) != ESP_OK)
            ) return 0;
        }
        else
        {
            if (!u8x8_setpininput_espidf(pin)) return 0;
        }
        break;
    default:
        if ( msg >= U8X8_MSG_GPIO(0) )
        {
            i = u8x8_GetPinValue(u8x8, msg);
            if ( i != U8X8_PIN_NONE )
            {
                if ( u8x8_GetPinIndex(u8x8, msg) < U8X8_PIN_OUTPUT_CNT )
                {
                    gpio_set_level(i, arg_int);
                }
                else
                {
                    u8x8_SetGPIOResult(u8x8, gpio_get_level(i));
                }
            }
            break;
        }

        return 0;
    }
    return 1;
}

Are there any visible flaws? Thanks.

[olikraus]

To be fair, current code generator looks like hell for me

then, it is better to work on a copy... ;-)

Do you have anything againist re-writing in in something non-C?

I do not care if you work on your own repo ;-)
For pull requests: New files in new directories should be ok. For the existing code I prefer C over C++

[v1993]

Hmm, seems there is a problem…

Arduino introduce global SPI state, but ESP-IDF does not. Doing so would mean big problems with more than displays running in parallel. It would be really nice to have non-interputtable version where SPI state don't have to be stored between calls. Any ideas?

Possible solution is to add SPI and I2C states to u8x8_t struct on ESP32. How bad is it? In my opinion, that's ok, but you're main developer after all…

[v1993]

BTW, do you have an IRC/Jabber? It probably would be faster to communicate there (nothing personal, feel free to reject with or without reason).

[v1993]

Allright, got it! There IS global state on hardware level (SPI1 and SPI2), so I make one "universal" function which can work with any SPI descriptor and call it from SPI(1/2) handlers.

[v1993]

Well, just got that I was doing HW SPI all wrong… It's better try to get ESP-IDF driver working and NOT try to adapt one from arduino-esp32.

I have some questions now:

1. How big data transfers usually are? There are huge optimisations for < 32 byte ones.
2. Do all SPI stuff have to be syncronized?
3. Do U8G2 read anything? I'm pretty sure it isn't, just want to be 100% clear about it.

[olikraus]

Possible solution is to add SPI and I2C states to u8x8_t struct on ESP32. How bad is it?

Yes, it will break the independence from SPI and I2C

BTW, do you have an IRC/Jabber?

No

How big data transfers usually are? There are huge optimisations for < 32 byte ones.

I2C transfers are limited to 32 bytes. Any other transfers can have any length.

Do all SPI stuff have to be syncronized?

Against what?

Do U8G2 read anything? I'm pretty sure it isn't, just want to be 100% clear about it.

U8g2 does not read from displays (but from GPIOs)

Just a general notes: I do not have much time to work on U8G2. So I can not do more than give comments on your questions.
The core part of u8g2 has to be C and it must not depend on any low level, system specific code... at least if you want me to accept pull requests.

[v1993]

Ok, thank you.

Do all SPI stuff have to be syncronized?

Against what?

In this case syncronized means no caching in driver, i.e. use polling (write means waiting until transfer is complete) and not interrupt (things goes into queue and are done in background) SPI mode (ESP-IDF specific stuff).

P.S.: I was talking about HW SPI transfers, or to be presice arg_int for U8X8_MSG_BYTE_SEND message. Sorry for not being clear.

[v1993]

Also, if adding stuff to u8x8_t is bad, can I use U8X8_WITH_USER_PTR and store my stuff by that pointer?

[v1993]

Hi. My porting is going ok yet. There is a question: can I be sure that memory pointed by arg_ptr is HW SPI callback is on stack? Thanks.

[olikraus]

Also, if adding stuff to u8x8_t is bad, can I use U8X8_WITH_USER_PTR and store my stuff by that pointer?

Yes, sure, that is the purpose of the user ptr

There is a question: can I be sure that memory pointed by arg_ptr is HW SPI callback is on stack?

probably not.

[v1993]

Interesting thing: it's possible to use two displays with all same pins except CS (and DC I think) which is handled by driver (even from two cores)!

However, I want to know one thing: is it possible to trace destruction of u8x8? I know that it's rare thing, but it would be nice to delete device in such a case.

Thanks.

[v1993]

And another (more actual) question: can I make few SPI transactions instead of one?

[v1993]

I do not know a single case where the page buffer is bigger then the frame buffer.

https://github.com/olikraus/u8g2/wiki/u8g2setupc#max7219-32x8 is a good example. There are also ones which have the same size as full one. Removing eqal ones would work as hint to developer that they can use full mode with same memory usage.

I usually call these u8x8_d_xxx functions as device function. What do you mean by "args/vars/functions"?

I've already called them somehow, don't remember how exactly. Under args/vars/functions I mean ones in code/docs generation.

Oh, i reread the previous posts, but did not found a specific question. Can you repeat the same?

Did the same. Turns out I actually removed them at some point of editing reply before posting… I've solved them in temporary way and plan to discuss later when making PR.

I also think that generating big portions of constructor references is bad idea especially when there are two different C++ constructors sets. I plan to generate table like this:

Controller/Display pair	1 buffer	2 buffer	f buffer	SPI	[other interfaces will be here in columns]	UART
SSD1306_128X32_NONAME	☐	☐	☐	☑		☐

This allows having one table for C and both C++ interfaces. As for constructor reference, I want to put it like this:

The communication protocol is part of the constructor name (see above). They differ for Arduino and ESP-IDF.
The following communication protocols are supported on Arduino:

Communication	Description	Constructor
4W_SW_SPI	4-wire (clock, data, cs and dc) software emulated SPI	U8G2_X_X_4W_SW_SPI(args_here)

What do you think about it? I'll continue working in this direction until you answer, so that would be sad if I'll have to throw out just written piece of code :-P

[olikraus]

No doubt, this looks better than the current wiki :-)

https://github.com/olikraus/u8g2/wiki/u8g2setupc#max7219-32x8 is a good example. There are also ones which have the same size as full one. Removing eqal ones would work as hint to developer that they can use full mode with same memory usage.

ok....

[olikraus]

Ok, I said it looks better. However i think the constructor name should be mentioned.
Additionally, the checkboxes imply, that the 1, 2 and f constructors might not be there. However, they are always created (even in those cases, where they are useless). So having checkboxes is a little bit useless.

[v1993]

Additionally, the checkboxes imply, that the 1, 2 and f constructors might not be there. However, they are always created (even in those cases, where they are useless). So having checkboxes is a little bit useless.

I partially agree with you.

At first, constructors creation/docs generation are now united: if there is constructor, there will be docs and vice versa. But you make a good point for F constructor because it is always here.

However i think the constructor name should be mentioned.

I'll think about it… don't think that listing all constructors like now is required, but giving good examples and description is important.

BTW u8x8setupc.md looks kinda lonely…

[olikraus]

BTW u8x8setupc.md looks kinda lonely…

Not sure what you mean.

In general the generated docs are a little bit less frequently used. Most users just see the constructor list in the examples :-(

I really like the effort, you put into this topic. Hope it is worth to do all this.

[v1993]

u8x8setupc.md contain only generated list and no hints at all.

I really like the effort, you put into this topic. Hope it is worth to do all this.

Thank you! I really want to make ESP-IDF support feel complete: have C++ bindings (done) and nice docs (WIP). To keep thing in one place I do the same to Arduino part obviously (united docs/C++ generator). Hope this will be useful for other ESP-IDF community members too.

[v1993]

That's some progress on docs: https://github.com/v1993/u8g2/blob/master/tools/newcodebuild/docs/u8g2setupcpp.md. What do you think?

NOTE: interface tables are auto-generated. Look at https://github.com/v1993/u8g2/blob/master/tools/newcodebuild/resources/Arduino/interfaces.lua and https://github.com/v1993/u8g2/blob/master/tools/newcodebuild/resources/ESP-32/interfaces.lua to see original data (it is also used for constructor generation).

[v1993]

Here it is — controller/display interface table: https://github.com/v1993/u8g2/blob/master/tools/newcodebuild/docs/u8x8controllertable.md!

[olikraus]

Looks good. Still a list of all constructors would be good.

[v1993]

Well, I personally think that list of all C++ constructors is pretty bad idea: having only Arduino one seems kind of unfair to ESP-IDF users and having both would result in giant page.

[v1993]

Wow! Thanks to table I've just fixed generation for I²C constructors.

Turns out it is really helpful tool even for development stage.

[v1993]

@olikraus I've noticed that U8x8 C++ Setup and U8g2 C++ Setup are extremely close to each other. Do you think that it is good idea to merge them into one file?

I'll do so myself.

Also, how can I send you non-generated changed/added docs from wiki?

[olikraus]

U8g2 and u8x8 are split to allow projects with pure u8x8 code.

What do you mean by send code? My email is mentioned in all c codes.

[v1993]

U8g2 and u8x8 are split to allow projects with pure u8x8 code.

I was talking about doc pages there…

What do you mean by send code? My email is mentioned in all c codes.

Well, I know that. For code part I can make pull request via GitHub, but what about docs? What would be a convinient way to review and merge wiki changes for you?

[v1993]

@olikraus If it isn't clear, I'm waiting for response.

[v1993]

@olikraus Umm, sorry to annoy you a bit more again… just saw that you're active in other issues. I'm waiting for your help here too. I'll have less free time in next few weeks, so I would be happy to finish docs as soon as possible (also complete notes while I do remember some details about code) and merge pull request.

[olikraus]

Sorry, as mentioned before, I am away from all kinds of computer hardware. Only mobile phone is there. I will be absent for another 7 days

Regarding wiki pull requests. I'd not know how to do this. But: is it required? Docs should be generated and not added via pull requests.

[v1993]

Regarding wiki pull requests. I'd not know how to do this. But: is it required? Docs should be generated and not added via pull requests.

Yep, but… I'm going to change non-generated docs (and write some new from scratch). What about those?

Again, do you thing that it is a good idea to merge u8g2 c++ setup and u8x8 c++ setup because they are very close and basicly dublicate each other in most places?

[olikraus]

What about those?

I could give you access rights.

u8g2 c++ setup and u8x8 c++ setup should be kept separatly.

[v1993]

I could give you access rights.

Would it be ok to create some docs before pull reuest?

u8g2 c++ setup and u8x8 c++ setup should be kept separatly.

Why? If I was unclear once again, I'm talking about their docs, not headers. I said that in previous reply.

[olikraus]

If someone wants to work with u8x8 only, then why shell that user read about u8g2? I think it is better to have two different setup docs.

[v1993]

@olikraus Eh, u8g2 setup and u8x8 setup (references) are basicly dublicates (they literally contain tons of eqal text) with very minor changes. Merging them is rather trivial and removes requiment to support two copies of docs.

[v1993]

@olikraus Oh, and before I forget: ESP-IDF CMakeLists.txt include csrc/ file list.

Do you think that it is a good idea to auto-generate it too? Can I assume any platform (linux/windows/etc.) in such case and for example run find ../../csrc -type f internally?

[olikraus]

How can I answer this?
My fear is, that with all your modifications, my projects becomes more difficult to maintain. Or I need to learn a lot of new things to maintain it. Please answer by your self and let yourself guide by the goal not to modify the existing structure.

[v1993]

My fear is, that with all your modifications, my projects becomes more difficult to maintain.

Why? There is no way merging two docs which repeat each other a lot may hurt development!

Or I need to learn a lot of new things to maintain it.

Well, it may apply to new codebuild, but not to docs. In fact, lua it rather easy to understand in most casesand I'll be glad to help with it.

Please answer by your self and let yourself guide by the goal not to modify the existing structure.

So again, what is a problem of merging two pretty much dublicated docs?

Also, what about my last reply? You repo contain .sh scripts so I suppose it is fine to assume linux?

[olikraus]

So again, what is a problem of merging two pretty much dublicated docs?

I just think it is better to just get exactly the information which you want to see.

Also, what about my last reply? You repo contain .sh scripts so I suppose it is fine to assume linux?

Yes, I do all the development with Linux

[v1993]

I just think it is better to just get exactly the information which you want to see.

Oh, I see. I'll still try to megre them, but will keep what you said in mind.

Yes, I do all the development with Linux

Great! Means that I can freely run find :-)

[olikraus]

Yes, I do all the development with Linux

Great! Means that I can freely run find :-)

Sure all the usual commands are there...

[v1993]

@olikraus I can't merge your last changes because of C++ part. Can you, please, tell what was changed in generation mechanism what I have to incorportate?

New displays/controllers, different functions, etc.?

[olikraus]

I have added a new display.
#884