03 Software simulator for fast creating and testing of LVGL GUIs
If you adjust the software to your needs, most likely you will need several iterations. Each time you have to compile the firmware, upload it to your ESP32 and test the result. Especially creating the LVGL screens is new to most of the users, so a lot of iterations are needed.
This is where the software simulator comes in place. You can compile and run the OMOTE code on your local Windows or Linux machine, showing the LVGL screens. Exactly the same code is used in the simulator as when you compile the firmware.
The simulator runs in Visual Studio Code (not Visual Studio) with PlatformIO. This is what you need anyway for compiling the firmware. No need for any other compiler or development environment. No Visual Studio is needed as often used in other LVGL simulators.
What is not working in the simulator is anything where the ESP32 hardware is needed:
- the hardware keys of the OMOTE (maybe support for simulating the hardware keys will be added in a future version)
- sending and receiving IR messages
- BLE keyboard (bluetooth keyboard) for controlling e.g. a Fire TV or other media players
- battery voltage
- sleep and wakeup
- the last used scene and screen are not saved. After restart, you start with no scene and the first screen.
Sending MQTT messages is supported in all environments (Linux, WSL2, Windows).
Using the simulator saves a lot of time. And using Linux/WSL2 over Windows also saves a lot of time (both for compiling the firmware and using the simulator). Later it will be briefly explained on how to setup WSL2.
| environment | ESP32, incl. flashing | simulator |
|---|---|---|
| Windows, full rebuild | 220 sec | 55 sec |
| Linux/WSL2, full rebuild | 85 sec | 7 sec |
| Windows, single file change | 80 sec | 12 sec |
| Linux/WSL2, single file change | 55 sec | 2 sec |
Flashing the ESP32 needs 30 sec (included in the numbers above).
This is how the running simulator looks like on Windows. In the background you can see the log messages.
The simulator was tested with Windows 11, WSL2 (Ubuntu) and Ubuntu 22.04 (standalone). In PlatformIO IDE, you have to
- choose the PlatformIO environment you want to use
- wait until PlatformIO has switched the environment. This may take some time if you never used this environment before
- click on
BuildorUpload
There are two environments available:
| env:linux_64bit | env:windows_64bit | env:windows_32bit | |
|---|---|---|---|
| running with | Linux, WSL2 | Windows (MINGW64) | Windows (MINGW32) |
| LVGL memory (default) | 64k | 64k | 32k (same as on ESP32) |
| remarks | more memory is needed, which is not a problem, but LVGL memory usage shows different values than on ESP32 | more memory is needed, which is not a problem, but LVGL memory usage shows different values than on ESP32 | LVGL needs exactly the same memory as on the ESP32 |
The example in the GIF above was recorded on Windows 11, using WSL2, and is in real time.
Note that in this animated GIF the simulator is running on WSL2, but Visual Studio Code still runs on Windows. The simulator window looks slightly different than a native Windows window. Please read the section below about setting up WSL2 on how to do this.
But prior to the compiler being successful, you have to install some software.
- Go to MSYS2, download the installer (tested with
msys2-x86_64-20240113.exe) and install MSYS2. - The next steps depend on if you plan to use 64 bit or 32 bit
| 64 bit | 32 bit | |
|---|---|---|
| add the following paths to your environment variable PATH |
C:\msys64\usr\bin C:\msys64\mingw64\bin |
C:\msys64\usr\bin C:\msys64\mingw32\bin |
| go to Windows start menu and start a shell | "MSYS2 MINGW64" | "MSYS2 MINGW32" |
| in the MINGW shell, install the following packages |
pacman -S mingw-w64-x86_64-gcc pacman -S mingw-w64-x86_64-gdb pacman -S mingw-w64-x86_64-SDL2 |
pacman -S mingw-w64-i686-gcc pacman -S mingw-w64-i686-gdb pacman -S mingw-w64-i686-SDL2 |
- you need to restart Visual Studio Code so that the changed path is recognized. After restart, open a terminal in VSC and check if the path is correct
The debugger gdb is only needed if you plan to debug the software. Try it and see if it is helpful for you. To me it is extremely helpful. Have you ever tried debugging ESP32 code? A nightmare.
You can install the packages both for MINGW64 and MINGW32, but you can only use one at the same time. Switching the environment in PlatformIO is not enough. For using the other compiler, you have to change the path and to restart Visual Studio Code. After restart, open a terminal in VSC and check if the path is correct.
Choose env:windows_64bit when using MINGW64.
Choose env:windows_32bit when using MINGW32.
env:windows_64bit with MINGW32 works, but will set LVGL memory to 64k.
env:windows_32bit with MINGW64 will not work and will lead to error messages from the linker. See comments in file platformio.ini for reasons.
- Install the following packages
- sudo apt install git
- sudo apt install gcc
- sudo apt install g++
- sudo apt install libsdl2-dev
Always choose env:linux_64bit. 32 bit will not work, because default installations of Linux and WSL2 are 64 bit.
All code in folder src\ is not depending on a specific hardware or compiler and almost not on the Arduino framework. Only Serial.print statements, delay() and millis() are still allowed in the code in folder src\.
In case of the simulator, the remaining Arduino related code in folder src\ is mocked with file src\applicationInternal\hardware\arduinoLayer.cpp
All access to the hardware has to be done in file src\applicationInternal\hardware\hardwarePresenter.cpp. This file includes the file hardware\hardwareLayer.h. In this header file, the magic happens.
Depending on if you are compiling the firmware for an ESP32, or if you want to use the simulator on Windows/WSL2/Linux, the correct hardware related header files are included.
All files in folder hardware\ESP32\ are for the ESP32 - including a lot of Arduino related code.
All files in folder hardware\windows_linux\ are for Windows/WSL2/Linux - having no Arduino related code at all.
hardware/
├── ESP32/
│ ├── lib/
│ ├── include_hal_esp32.h
│ └── ...
├── windows_linux/
│ ├── include_hal_windows_linux.h
│ └── ...
└── hardwareLayer.h ←───────────────┐
│
src/ │
├── applicationInternal/ │
│ └── hardware/ │
│ ├── hardwarePresenter.cpp ──┘
│ ├── hardwarePresenter.h ←───┐
│ └── arduinoLayer.cpp/.h ←───┤
├── devices/ ───────────────────────┤
├── devices_pool/ │
├── guis/ ───────────────────────┤
├── scenes/ ───────────────────────┤
└── main.cpp ───────────────────────┘
This section is only for Windows users.
WSL2 is not directly related to OMOTE or the LVGL simulator, nor is it needed. But because of the much higher compiler speed (factor 3 for compiling ESP32 firmware, factor 6-8 for compiling the simulator, both compared to Windows) it might be worth for you having a look.
- Install WSL2: https://learn.microsoft.com/en-us/windows/wsl/install
Basically it is just the following Powershell commands:- wsl --install
- wsl --install Ubuntu
- Read Developing in WSL to understand how it works. In short, it works like this:
- Visual Studio Code is running on Windows
- a "Visual Studio Code Server" is automatically installed in WSL2 (or any other Linux machine you connect to with SSH)
- files, compiler, debugger and all other tools are running in WSL2
- Visual Studio Code runs on Windows and is only the frontend to it
- Install some VSC extensions. A good start is:
- You can also try these VSC extensions:
This is the hardest part.
If you want to flash your firmware to the ESP32, the USB port from windows has to be made available in WSL2.
Most Linux distributions provide a tool called USBIP, which can provide an attached USB device via an IP connection to a different machine. See this nice picture on how it works. And if you are interested, please also read the documentation from Microsoft on how to connect USB devices to WSL2.
Your windows machine is the "USB/IP server daemon", and WSL2 is the "USB/IP client" (could also be a Hyper-V guest).
First you need to install the USBIP server software for Windows.
Following are the most important steps to do. If you need more detailed information, please read the documentation linked above.
You need to bind the attached OMOTE once with administrative privileges in Powershell or Command Prompt.
- usbipd list
- usbipd bind --busid=<BUSID>
After this initial bind, the rest can be done without administrative privileges.
There are two ways to attach the OMOTE to WSL2. From Windows you have to use:
usbipd attach --wsl --busid=<BUSID>
or
usbipd attach --wsl --busid=<BUSID> --auto-attach
Use --auto-attach to automatically re-attach when the device is detached or unplugged.
If you attach the OMOTE, you get a mesage like
usbipd: info: Using WSL distribution 'Ubuntu' to attach; the device will be available in all WSL 2 distributions.
usbipd: info: Using IP address <your IP address> to reach the host.
Go to WSL2 and check with ls /dev/ttyUSB0 if the USB port is now available in WSL2.
You can detach the device from Windows with usbipd detach --busid=<BUSID>
Sometimes, especially if you have several WSL2 distros and/or Hyper-V guests, the USB device is not automatically available in the distro you want to have it. If so, go into your WSL2 distro and do
usbip list --remote=<your IP address>
sudo usbip attach --remote=<your IP address> --busid=<BUSID>
Check with usbip port and ls /dev/ttyUSB0 if the USB port is now available in WSL2.
You can detach the device from WSL2 with
usbip port
sudo usbip detach -p "00"
or any other number provided when using usbip port
If the device is available in WSL2 (ls /dev/ttyUSB0), but you don't have write access to it, you can change access rights with sudo chmod a+rw /dev/ttyUSB0. This needs to be repeated each time you connect the USB device.
You can also try to use udev rules which automatically set access rights for certain devices being connected to a Linux machine.
If I remember correctly, I installed PlatformIO Core (CLI) in WSL2 with the installer script (without the Visual Studio Code IDE, VSC will run on Windows) which automatically installed a bunch of udev rules. From that on, all known ESP32 devices were automatically available with write access without having to use sudo chmod a+rw /dev/ttyUSB0 anymore.
Sometimes /dev/ttyUSB0 doesn't show up in WSL2, no matter if you use Way 1 or Way 2.
If you think you attached the device correctly, you can check it like this.
- check if USB device is available in WSL2
$ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 1a86:7523 QinHeng Electronics CH340 serial converter
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Here the device in the middle is the OMOTE. So at least it was correctly attached and is available in the correct WSL2 distro.
- check if driver has been loaded
$ dmesg
...
[ 146.442476] vhci_hcd vhci_hcd.0: pdev(0) rhport(0) sockfd(3)
[ 146.443572] vhci_hcd vhci_hcd.0: devid(327682) speed(2) speed_str(full-speed)
[ 146.444630] vhci_hcd vhci_hcd.0: Device attached
[ 146.720435] vhci_hcd: vhci_device speed not set
[ 146.790433] usb 1-1: new full-speed USB device number 2 using vhci_hcd
[ 146.880438] vhci_hcd: vhci_device speed not set
[ 146.950438] usb 1-1: SetAddress Request (2) to port 0
[ 147.028412] usb 1-1: New USB device found, idVendor=1a86, idProduct=7523, bcdDevice=81.34
[ 147.029668] usb 1-1: New USB device strings: Mfr=0, Product=2, SerialNumber=0
[ 147.030695] usb 1-1: Product: USB Serial
Here something is missing. It should look like the following. Look at the last four lines.
$ dmesg
...
[ 505.243758] vhci_hcd vhci_hcd.0: pdev(0) rhport(0) sockfd(3)
[ 505.244847] vhci_hcd vhci_hcd.0: devid(327682) speed(2) speed_str(full-speed)
[ 505.245940] vhci_hcd vhci_hcd.0: Device attached
[ 505.520433] vhci_hcd: vhci_device speed not set
[ 505.590430] usb 1-1: new full-speed USB device number 6 using vhci_hcd
[ 505.680430] vhci_hcd: vhci_device speed not set
[ 505.750445] usb 1-1: SetAddress Request (6) to port 0
[ 505.828903] usb 1-1: New USB device found, idVendor=1a86, idProduct=7523, bcdDevice=81.34
[ 505.830091] usb 1-1: New USB device strings: Mfr=0, Product=2, SerialNumber=0
[ 505.831198] usb 1-1: Product: USB Serial
[ 505.855612] usbcore: registered new interface driver ch341
[ 505.856506] usbserial: USB Serial support registered for ch341-uart
[ 505.857411] ch341 1-1:1.0: ch341-uart converter detected
[ 505.885824] usb 1-1: ch341-uart converter now attached to ttyUSB0
So what happended? Can't tell you. What was the solution?
In my case, I have several WSL2 distros on my Windows machine: Ubuntu, Debian and two others. Ubuntu is what I'm using for developing with PlatformIO. But I have to first start distro Debian and then to do a sudo usbip attach --remote=<your IP address> --busid=<BUSID> in Ubuntu, otherwise it will not work. So a running, but not used Debian is needed for me to attach the USB device to Ubuntu ... This weird behaviour started after installing Ubuntu as Hyper-V guest.