Feature Request: Pygame-like display interface #137
Description
I would like to request a feature:
Either as part of adafruit_rgb_display or as a separate module that may use adafruit_rgb_display as a dependency:
I would like to see a Pygame-like low level display library for CircuitPython.
What makes this different from displayio?
displayio is designed to be a high level UI library. It requires heavy use of object heirarchies that cost additional memory and CPU time, and it is not terribly well suited to beginners or to the development of real-time video games, due to this higher level of complexity. For casual games, where performance isn't terribly important, it works well, and for GUI control and data feedback interfaces, it is a good choice. For high performance applications where highly customized UIs are ideal, real-time video games for example, it just gets in the way. And for beginners who are just trying to put some basic text or images onto the screen, the UI object heirarchy is just a series of unnecessary hurdles. CircuitPython needs a lower level display library to be well suited to one of the most common types of application people learn programming for (making games).
What is adafruit_rgb_display missing?
adafruit_rgb_display is missing a ton of stuff that is necessary for it to be useful with CircuitPython. While it does contain a function for rendering images to the display, the images must by in PIL format, and there is no PIL port for CircuitPython. adafruit_rgb_display works with Blinka, so on a platform with standard Python, like the Raspberry Pis, for which there is a PIL port, it can work, but there's no way to load images in the correct format in CircuitPython. One major thing that is needed is the ability to load images either from program memory or microSD cards, ideally using the CircuitPython filesystem. It would be nice if images were loaded in 565 format 16-bit color, to avoid the need to do the conversion every time they are rendered.
adafruit_rgb_display uses no buffering. The drawing functions draw directly to the display instantly. This results in rapid flashing when trying to do animation, as the screen is constantly cleared and rewritten. In theory this effect could be eliminate with per-pixel selective writes, but keeping track of dirty pixels, and the process of working out and rendering the pixels that have been modified is computationally expensive. This strategy only works when changes between frames are rare and fairly small, and even then it requires significantly more code. Many of Adafruit's microcontroller boards aren't fast enough to handle this. The solution is to provide the ability to create a use a framebuffer. Pygame automatically creates a framebuffer when a display surface (a window) is initialized. I'm not sure this is the best strategy in this case, as there are applications where unbuffered writes (or only partially buffered writes) might be the best option. Instead, it might be best to provide the ability to create buffer surfaces that can be rendered to along with a method of very quickly sending the entire buffer to the display. This might be expanded to allow smaller buffers to be sent to the display at a specific location, so that users have the option of using multiple smaller buffers, each for a different region of the screen, either for splitting the screen into multiple viewports for different kinds of data or for spitting the screen to display different points-of-view. These surfaces would work much like Pygame surfaces, though perhaps with more limited color depth and fewer features, to keep the code more compact and fast.
adafruit_rgb_display also lacks drawing functions. It can only draw filled rectangles, horizontal lines, and vertical lines. Pygame's draw submodule can do rectangles (filled or empty), polygons, circles, ellipses, arcs, lines (not exclusive horizontal and vertical), and anti-aliased lines. Most of these are not computationally difficult, and for some there are published highly optimized algorithms that could be used. What I would specifically like to see are rectangles, triangles, ellipses, and circles (filled and non-filled) as well as lines (of any angle). Connected triangle and line arrays would also be nice, but those can be done easily enough with for loops of triangles or lines (though less optimally). Anti-aliasing is likely too computationally expensive and should probably be omitted. (C++ code for much of this already exists here: https://github.com/adafruit/Adafruit-GFX-Library)
Pygame also has a font module. The Learn guide here (https://learn.adafruit.com/2-0-inch-320-x-240-color-ips-tft-display/python-usage) uses PIL's ImageFont for this, but again, there's no PIL port for CircuitPython. As before, there is already some Adafruit code in (https://github.com/adafruit/Adafruit-GFX-Library) for that, which might be easier to port to CircuitPython than it would be to rewrite it from scratch.
Lastly, adafruit_rgb_display seems to be quite slow. Specifically, I've written up a bit of Python code myself, attempting to partially implement some Pygame features for adafruit_rgb_display, and even using ulab arrays, I'm only getting peak framerates around 12.5FPS (80ms per frame). (I'm using an RP2350 in a Fruit Jam, with the 2.0" 240x320 SPI TFT.) I don't currently know exactly what the problem is, but I find it likely it is the result of either pure Python code being used in adafruit_rgb_display for handling the SPI writes, or possibly too many levels of function calls or indirection behind the scenes. (I don't have a logic analyzer yet, so I can't really troubleshoot to work out the source of the problem.) If it is possible to implement the display drivers in adafruit_rgb_display in C or C++, with the write functions built into those or using SPI write functions also written in C or C++, that might solve the write speed problem.
Other Pygame features that might be useful?
Pygame's Rect type is used heavily in Pygame's rendering, for passing around dimensional information. It is quite useful for avoiding excessively long argument lists. It would be nice to keep the API as close to what Pygame uses as possible, and that would require the use of the Rect type in a lot of places. In addition Rect includes collision detection, positioning, and clipping functions that are useful both in rendering and in games. Because Rect objects don't interact directly with peripheral hardware, it might be possible to use Pygame's code direct, with limited need for modification (https://github.com/pygame/pygame/blob/main/src_c/rect.c). (It is worth noting that it is licensed under the Library GPL, which is likely not relicensable under any other more permissive license.) Even if it can't just be directly ported though, most of the operations done in Rect are pretty easy to reimplement.
Other Notes:
I've already started an attempt at this. I wanted to do it in pure Python, as I don't have time to do it in C right now. Unfortunately, I ran into the above mentioned framerate issue. I've implemented basic surfaces, rects, and a display submodule that handles drawing a surface to the display. For testing, I'm using a Fruit Jam (https://www.adafruit.com/product/6200) and a 2.0" 240x320 TFT display (https://www.adafruit.com/product/4311). I'm using ulab for surface buffers, in an attempt to optimize speed and size. I've ran into several problems, including ulab using the wrong endianness for the display and the framerate issue mentioned above. Writing to the ulab array buffer in CircuitPython seems to be quite fast. I haven't implemented any drawing functions, but I've got a couple of loops that draw a 1px border around the screen, and commenting them out doesn't seem to make a significant difference in framerate.
I have significant experience with Pygame and some experience with SDL and SDL2 (which is what Pygame uses under the hood). I also have some experience with video game development, and I've written basic rendering engines many times (I taught undergrad video game design for several years, and part of the course was in-class coding demos, where I wrote a basic game engine, including the rendering engine, over the course of a few weeks). I'm willing to help with with the Python side of this, and I'm will to providing consulting for design and such on lower level stuff. I might even be able to help a bit with C/C++ code, so long as I don't have to deal with the Python bindings. (I mostly work in C and Python in my daily work. I generally avoid C++ specific stuff, but I can work well within C++, so long as I don't have to deal with too much C++ specific stuff that isn't also part of C.)
Are there any alternative options that might be viable instead?
Yes, there's one I can think of. Adafruit_GFX_Library (https://github.com/adafruit/Adafruit-GFX-Library) for Arduino might be more easily ported than it would be to make a cut down Pygame-like implementation. I don't know exactly what of the above features Adafruit_GFX_Library has. I know it doesn't have the ability to load images from a filesystem and instead expects images to be embedded in the code as data. It shouldn't be difficult to copy the image loading functionality from displayio though. It also doesn't have drivers built in. Adafruit's Arduino libraries do include many display drivers written in C++ that are compatible with Adafruit_GFX_Library, which could probably be ported. The main complication I see arising from that is replacing calls to Arduino's pin selection functions. I don't think this would be terribly difficult though, and it would likely improve speed significantly. (I've been down the rabbit hole of Arduino's pin selection functions, and it's disturbingly deep, with many layers of function calls and pointer dereferencings. For any microcontroller where pin/port addresses don't absolutely have to be hardcoded, it should be extremely easy and much more efficient to replace Arduino's code. If it is absolutely necessary to keep Arduino's pin selection code, it can probably be pulled out and ported itself. In fact, CircuitPython probably already has its own alternative to that which could be used instead...) There may be other Arduino dependencies that require more effort, but in my experience, Arduino dependencies aren't generally terribly difficult to remove/replace.
If you managed to get to the end of this, thank you! I really appreciate your time.