Add more gradient features

[kisvegabor]

Problem to solve

Continue #5527

Having more gradient features could

• save to use of images in some cases
• allow the use of animations of on images
• improve the responsiveness

Success criteria

1. Improve lv_grad_dsc_t to describe: skew, radian, conic gradiens too
2. Support these gradients at least in SW rendering

Solution outline

As we follow CSS for most the style properties we should do the same here as well.

lv_grad_dsc_t can be updated like this:

typedef struct _lv_grad_dsc_t{
    lv_gradient_stop_t   stops[LV_GRADIENT_MAX_STOPS]; /**< A gradient stop array */
    uint8_t              stops_count;                  /**< The number of used stops in the array */
    uint32_t             angle;                        /**< 0..3600, 0.1 resolution, for linear and conic types*/
    lv_grad_type_t       type;                         /**< linear, conic, radial  */
    struct _lv_grad_dsc_t *next;                       /**< Next gradient to render more gradients*/                                        
} lv_grad_dsc_t;

It could effectively describe the main properties and can be easily extended later.

---

The implementation in SW rendering should be a generic solution to allow easily using it later on arcs, bg images, borders or shadows too. To be more precise we should be able to render any partial gradients into an ARGB8888 buffer, which can be used as an image during rendering.

Rabbit holes

The performance of SW rendering is still a questions.

Testing

The usual unit testing should work.

Teaching

Adding docs, example and using it in demos should be enough.

Considerations

No response

[kisvegabor]

cc @XuNeo @bjsylvia @kdschlosser

[kdschlosser]

Right now how the current gradients work the building of the gradient and also the rendering of it seem to take place in a single step; I believe the best thing is going to be to separate the 2. this way the "gradient" itself is able to be reused. we also need to have a dithering function to smooth out what the gradient displays as for RGB565 displays.

I can shoot the code over for rendering say a radial gradient and you guys can optimize it. Meaning we can have it so that it only renders an 1/8th of the data and you can transform that around to make the entire gradient. I don't know how to go about doing that last bit in LVGL. I don't even know if it would be faster to do that.

That brings up a thought. can we build in some kind of mechanics to be able to time function calls? This would require a higher time precision than milliseconds but for testing purposes it would be extremely helpful to have.

[kisvegabor]

From my first comment:

To be more precise we should be able to render any partial gradients into an ARGB8888 buffer, which can be used as an image during rendering.

I think we need such a function. Of course the gradient can be cached, but we can decide to render it in smaller parts in case of PARTIAL buffering or in FULL/DIRECT mode to save some memory. It can be an improved version of lv_gradient_get().

I can shoot the code over for rendering say a radial gradient and you guys can optimize it.

Even a non optimal code would be great as a proof of concept and as a starting point.

That brings up a thought. can we build in some kind of mechanics to be able to time function calls? This would require a higher time precision than milliseconds but for testing purposes it would be extremely helpful to have.

Take a look at the built-in profiler.

[kdschlosser]

I am going to dink about with this a little bit in the next few days. I have added support to the micropython binding I have been messing about with. I have to test to see if it works properly. Once I know it is working then I will add it to LVGL and submit a PR for it. The PR is only going to be for proof of concept and it should not be merged. It gets somewhat complicated because technically speaking we should be able to allow compound gradients, like adding a radial and a conical together.

I am also not going to optimize the code to do things like integer math. I will leave that up to the folks that have more experience in C code.

[zjanosy]

we also need to have a dithering function to smooth out what the gradient displays as for RGB565 displays

I second that. Gradient backgrounds look cool, but on a 16-bit display it just does not work because of the severe color banding. A workaround is to use a properly dithered (using, e.g., Floyd-Steinberg dithering) bitmap as a background image with tiling in one direction. This trick kind of solves the problem, but certainly it needs more storage (especially if one would want to use different colors on different screens), and it needs additional manual work to generate the bitmaps.
LVGL v8 did have a dithered gradient, but it has been removed from v9. Maybe it should be revisited. Certainly, performance is a big issue, especially because (part of) the background needs to be redrawn almost always when something changes -- not to mention scrolling. So a proper implementation should use some kind of caching mechanism (e.g., render the gradient on a canvas first, and use it as a background image).

[kdschlosser]

Take a look here..

https://github.com/kdschlosser/lvgl_micropython/blob/main/ext_mod/lvgl_addons/src/color_addons.c

This takes a raw pixel buffer to render the gradients to. then the buffer can be passed to lv_image. Dithering is a step that can be done between the 2 if needed. The dithering is done in place so no additional memory is needed just the time to do it. These kinds of things you have done at startup and only once.

[zjanosy]

@kdschlosser Nice. What is this dither algorithm called? Seems to be efficient, because it is local to a pixel, and does not rely on other pixels. I wonder how well it performs (in terms of quality) compared to F-S?

[kdschlosser]

You can see the results here.

#5764 (comment)

It shows the original, then as RGB565 and then RGB565 dithered. The results are quite good actually. It's faster than Floyd-Steinberg dithering. I do know that. a lot less calculations and the calculations that are done are bitwise operations which are the fastest thing that can be done. The quality is really good..

[kdschlosser]

Here are some examples of what those functions can do

#5764 (comment)

#5764 (comment)

[zjanosy]

Thank you for the examples, they look very convincing. I think we should try them on actual hardware to see how they perform. The dithering should be fine, but I'm a bit worried about the radial and conical gradients, since they use floating point calculation.

[zjanosy]

@kdschlosser I have started to work on adding this feature, based on your input.

[kdschlosser]

There is an attached zip file with C code already written.. I don't know if you saw it.

https://github.com/lvgl/lvgl/files/14594940/lvgl_addons.zip

It compiles, I have not tested it tho. It's a good starting point. The gradients can definitely be optimized I am sure, especially the radial gradient. I am also sure that you can set it up to use integer math as well. I gave a starting point. If possible I would move those large arrays for the dithering so they get allocated on the stack. I think that in most use cases the dithering is not something that would run real time in the sense that it would be done in the middle of the running program. I would imagine it would be done at startup and the user would cache the dithered image. It really depends on the performance I would guess. I know that allocation when the function runs would take up a little bit of time. We would need to run performance tests on it, real world tests to see if the memory allocation is worth the tradeoff.

might have to add a macro around it to enable or disable the dithering code.

[zjanosy]

I have some preliminary results. I have implemented a radial gradient algorithm with both floating point and fixpoint arithmetics. It has 5 parameters:

• starting circle center
• starting circle radius
• ending circle center
• ending center radius
• extend mode (pad, repeat, mirror)
  Some examples (the starting and ending circles are drawn on top of the gradient):
  
  
  
  The screenshots were made in the Windows simulator. The code has been tested to run on an ESP32-S3, but the performance needs some improvement. Currently the algorithm does not take advantage of the symmetries or special cases.

[kisvegabor]

Wonderful! 😍

As we have discussed in the end we need to render the gradient line by line to integrate into the SW render. But I think it wont be an issue.

[kdschlosser]

The code is also not working with the alpha channel at all. Can this be adjusted so it does? It doesn't look like it is using anti-aliasing either.

[kdschlosser]

how long does it take to render on an ESP32?

[lvgl-bot]

We need some feedback on this issue.

Now we mark this as "stale" because there was no activity here for 14 days.

Remove the "stale" label or comment else this will be closed in 7 days.

[kdschlosser]

not stale

[zjanosy]

I'm more or less finished with the new gradient support in LVGL. In this PR linear, radial and conical gradients are supported with various extend modes (pad, repeat, reflect). Alpha maps are supported as well. Currently the algorithms uses exclusively integer arithmetic. However, on platforms with an FPU it may be faster to use floats instead.
The performance is acceptable on an ESP32-P4, but there is a lot of math involved to calculate the pixels, so it may not be reasonable to use these on low end platforms.
I will add some more examples and test cases.

[kdschlosser]

and how on earth did you get your hands on a P4 to test it with? There are no devkits yet from espressif and I haven't seen any that have been made by anyone else unless you order 1000 at a time.

[zjanosy]

Gábor has good connections... :-)

[kdschlosser]

one of the things you also need to rememver is that the application of the gradients is not something that will typically occur on demand. It is something that is going to be done on startup. The other thing is it is typically not going to be used as something that would take up the entire screen. It will be used for much smaller sizes. Using the conical gradient for adding a gradient to the arc widget indicator. That kind of thing.

Depending on how much "extra" you added that extra might be overboard and it would be increasing the math involved. For example, if I wanted to render a simple radial gradient using the center of the supplied buffer as the center point for the gradient and I didn't want to do any perspective changes the math involved would be vastly easier to accomplish that. So having all of the extra features will make more work to render a normal radial gradient.

What would be the use case for rendering this kind of a gradient??

I can see where there would be a use case for rendering a normal radial gradient. If you combine the ability to render a radial gradient and a conical gradient together you can now render a color wheel.

[kdschlosser]

The only thing I could see where you would use the kind of gradient above is if you were wanting to do something like a 3D tunnel animation. I do not think the MCU's that LVGL runs on would be up to the task. Don't get me wrong, it would make for a really cool demo of LVGL and it's abilities that's for sure. Maybe that would be the reason for adding it.

[zjanosy]

The radial gradient code handles the special cases (like concentric limit circles) separately, but they are still much more CPU intensive than just drawing a horizontal or vertical gradient because of the square root involved. The conical gradient uses atan2, which needs some CPU cycles, too. I did my best to find the fastest algorithms, but take a look at the code, maybe you can find a way to optimize it further.
As I wrote above, the floating point implementation may be actually faster (especially with a fast square root algorithm). I did not add this, but I have also implemented the radial gradient using floating point. On the ESP32-S3 is was a bit slower than the integer version, on the P4 is was a bit faster. I did not test it on other platforms.
Gradients can be used as a background, or add lighting effects. Unfortunately when something is above them (e.g., the needle of a meter or the handle of an arc), the area below that object needs to be recalculated. We could use a canvas as the background, render the gradient into the canvas, and then use it as a background image. But this uses a lot of memory, so it is a tradeoff.

[kdschlosser]

where is the code?

There are also MCU's that do have dedicated FPU's and I would image those would be faster no? When compiling if an MCU doesn't have an FPU the math library that is apart of the stdlib doesn't have the float functions like powf and atanf. What I had done was in those cases I used a software calculation to handle it.

See here....

https://github.com/kdschlosser/lvgl_micropython/blob/c55a6b02142f6c9291759ee45feb849a198e11b5/ext_mod/lvgl_addons/src/soft_math.c

I have not tested the code to see if it works as I was only looking to get it to compile properly which it does.

[kdschlosser]

I also believe that due to the nature of what we are working with as far as the hardware is concerned the rendering doesn't need to be perfect. What I mean by that is say for the radial gradient we don't have to create a gradient that has a different color for each pixel across the radius. That can be scaled down so say 1/3rd of the radius. Once the rendering of the gradient has finished dithering can then me used to smooth out the banding. That is going to work a lot faster due to the dithering being mostly bit operations.

I did come across this for a software sqrf function.

float soft_sqrtf(float number)
{
    union {
        float    f;
        uint32_t i;
    } conv;

    conf.f = number;
    conv.i  = 0x5F3759DF - (conv.i >> 1);
    conv.f *= 1.5f - (number * 0.5f * conv.f * conv.f);
    return conv.f;
}

dunno how that would fair in terms of performance and accuracy compared to using only integers and lookup tables. The sqrt function that you have written has a sizeable table that it uses. Memory on some of these MCU's is more important than speed. an example is to look at the the STM32 SOC's. decent processor speed no memory to speak of really. This is what makes it really hard but you have other MCU's that will fair better in the memory department but not have a whole lot of processor speed. The ESP32 fairs better in the memory department especially when you get into the octal SPIRAM which would be twice as fast as the SPIRAM available for the STM32 MCUs But the processor speed is not as fast when compared to some of the higher end STM32 MCU's

I know I know the P4. But those are to be considered not yet available to the general public and they won't really be fully supported for about a year.

[kdschlosser]

Right now as it sits if someone wants to add a gradient that is outside the scope of what LVGL is currently able to do they have to create it into an image like a PNG and load that image which takes up a much larger amount of memory then if it could be rendered to a buffer and that buffer stays resident. That is what I am trying to get away from is the extra amount of memory used and also the extra amount of flash used to store the image file.

So if we can reduce the flash consumption and the memory consumption and trade that that some additional processor overhead at the start of the program that is a win and that should be the goal. Give the user the option of what they want speed or memory and that is exactly what would be given if adding the gradient rendering to LVGL.

The user can decide which one they want to use.