02 How to understand and modify the software

Getting started

OMOTE can be used with almost any device that can be controlled with

• IR signals (infrared)
• MQTT messages
• a BLE keyboard

The software comes with a predefined set of devices that can be taken as starting point.

device	what is it used for	communication	remark
device_samsungTV	TV showing the picture	IR (SAMSUNG)	full set of IR commands for Samsung UE32EH5300
device_yamahaAmp	AV receiver	IR (NEC)	full set of IR commands for Yamaha RX-V359
device_appleTV	Apple TV media player	IR (SONY)	only a small set of IR commands
device_smarthome	control your smart home with MQTT	MQTT	example on how to switch and set brightness of a bulb
device_keyboard_ble	BLE keyboard that can be connected to a media player	BLE	can be used to connect to e.g. a Fire TV or anything else that can be controlled with an BLE keyboard
device_keyboard_mqtt	a special "keyboard" that can send MQTT messages	MQTT	Can be used together with another ESP32 emulating a hardware keyboard that can be connected to a media player with an USB port. Use this if you can't or don't want to connect your media player with BLE to the OMOTE (BLE needs lot of resources on the OMOTE)

The devices above are used for the demo application. Commands of all these devices are either used by the gui or by hardware keys.

In folder /src/devices_pool there are more devices. If you want to use them, please read below in section "How To".

Devices included in folder /src/devices_pool are (list is probably not complete):

category	device
TV	LG 42LA6208-ZA
AV receiver	Denon AVR-S660H LG SOUNDBAR Remote AKB73575421
media player	LG BLURAY Remote AKB73896401 SAMSUNG BLURAY AKB73896401 NVIDIA SHIELD TV

What is the minimum you have to do?

If you simply want to use this setup and to see how it looks like, then

• copy file secrets_override_example.h to secrets_override.h
• put in this file your WiFi credentials and the address of your MQTT broker
• if you want to use WiFi, check if -D ENABLE_WIFI_AND_MQTT=1 is set in platformio.ini
• if you want to use the BLE keyboard, check if -D ENABLE_BLUETOOTH=1 and -D ENABLE_KEYBOARD_BLE=1 is set in platformio.ini
• compile the firmware and upload it to your OMOTE

If you want to understand how to adjust the software to your needs, please continue reading the next sections.

Basic concepts

Devices

In OMOTE, devices are described with their capabilities they can provide. A device can be anything that can be controlled via IR signals, MQTT messages or the BLE keyboard provided by OMOTE.

• a TV
• an AV receiver
• media players like DVD player, Apple TV, etc.
• smart home devices that can be controlled with MQTT messages
• the BLE keyboard is a special device that can be used to control any device which is paired to the OMOTE, e.g. a Fire TV. Such devices don't need to be defined in OMOTE, they simply have to be paired to the BLE keyboard.
  Remark: currently it is only possible to pair one single device via BLE to the OMOTE. Multiple connections, e.g. to a Fire TV and a Apple TV, are not supported. As a workaround, you could pair one device to the BLE keyboard and connect the other to a MQTT keyboard. You can have as many MQTT keyboards as you want.

Each device definition is placed in a separate folder. The definition of a device is simply a list of commands a device can receive.

• the *.h file lists all available commands (e.g. Volume up, Power toggle etc.)
• the *.cpp file needs to provide a function that is called in main.cpp to register this device to be available at runtime.

A lot of devices can be available in your source code, but they are only active at runtime if e.g. register_device_samsung(); is called in function setup() in main.cpp.

All devices in folder /src/devices_pool won't get compiled by default. You first have to move them to folder /src/devices if you want to use them.

Take care: every device you register needs memory. Memory is limited, so don't register all available devices.

Commands

As explained in section "devices", all devices can be registered with the commands they can receive. Wherever you are in code, a command like executeCommand(SAMSUNG_POWER_OFF); can be send. The parameter can be any command that was registered by any device.

Commands can also be used for switching scenes or to do some special purposes. Both will be explained later.

Scenes

Scenes are made up of a set of devices, selecting the correct inputs for each device and other things. In most cases scenes are something like:

• watching TV
• watching DVD
• watching Apple TV, Fire TV or Google TV

For each scene you can define:

• a start sequence to start up all the devices needed for that scene
• a stop sequence that is executed if you leave a scene
• a keymap that defines which hardware keys on the OMOTE will send which commands

The software comes with a predefined set of scenes:

scene	what is it used for
TV	turn on TV and AV receiver, set correct input of TV and receiver
Fire TV	turn on TV and AV receiver, set correct input of TV and receiver, hit HOME on BLE keyboard to bring Fire TV to the start page
Chromecast	turn on TV and AV receiver, set correct input of TV and receiver
Apple TV	turn on TV and AV receiver, set correct input of TV and receiver
Off	turn off all devices

By default, the four scenes are assigned to the four hardware keys at the bottom of the OMOTE. The scene 'off' is selected with the power button in the upper right corner of the OMOTE.

Programmatically, a scene can be selected e.g. with executeCommand(SCENE_FIRETV);

Logitech Harmony users are used to know "Activities", which is something very similar.

User frontends

There are two kind of user frontends available in OMOTE: hardware keys and the touchscreen.

Hardware keys

All hardware keys can be fully customized. They can send any command registered by a device, can switch a scene or can be bind to a special command that can do anything you want to do in code.

For each key you can define a command that is executed if a key is short pressed, and another command (if wanted) that is executed if a key is long pressed. And you can define if the command for a short press is repeated if the key is hold.

enum repeatModes {
  // only as fallback
  REPEAT_MODE_UNKNOWN,
  // if you short press or hold a key on the keypad, only one single command from keyCommands_short is sent
  // -> best used if you do not want a command to be sent more than once, even if you press the key (too) long, e.g. when toggling power
  SHORT,
  // if you hold a key on the keypad, a command from keyCommands_short is sent repeatedly
  // -> best used e.g. for holding the key for "volume up"
  SHORT_REPEATED,
  // if you short press a key, a command from keyCommands_short is sent once.
  // if you hold a key on the keypad, a command from keyCommands_long is sent (no command from keyCommands_short before)
  // -> best used if a long key press should send a different command than a short press
  SHORTorLONG,
};

A default definition for the hardware keys is provdided in scenes/scene__defaultKeys.cpp and looks like this:

  key_repeatModes_default = {
                                                                                                             {KEY_OFF,   SHORT            },
    {KEY_STOP,  SHORT            },    {KEY_REWI,  SHORTorLONG      },    {KEY_PLAY,  SHORT            },    {KEY_FORW,  SHORTorLONG      },
    {KEY_CONF,  SHORT            },                                                                          {KEY_INFO,  SHORT            },
                                                         {KEY_UP,    SHORT            },
                      {KEY_LEFT,  SHORT            },    {KEY_OK,    SHORT            },    {KEY_RIGHT, SHORT            },
                                                         {KEY_DOWN,  SHORT            },
    {KEY_BACK,  SHORT            },                                                                          {KEY_SRC,   SHORT            },
    {KEY_VOLUP, SHORT_REPEATED   },                      {KEY_MUTE,  SHORT            },                     {KEY_CHUP,  SHORT            },
    {KEY_VOLDO, SHORT_REPEATED   },                      {KEY_REC,   SHORT            },                     {KEY_CHDOW, SHORT            },
    {KEY_RED,   SHORT            },    {KEY_GREEN, SHORT            },    {KEY_YELLO, SHORT            },    {KEY_BLUE,  SHORT            },
  };
  
  key_commands_short_default = {
                                                                                                             {KEY_OFF,   SCENE_ALLOFF_FORCE},
  /*{KEY_STOP,  COMMAND_UNKNOWN  },    {KEY_REWI,  COMMAND_UNKNOWN  },    {KEY_PLAY,  COMMAND_UNKNOWN  },    {KEY_FORW,  COMMAND_UNKNOWN  },*/
  /*{KEY_CONF,  COMMAND_UNKNOWN  },                                                                          {KEY_INFO,  COMMAND_UNKNOWN  },*/
                                                     /*  {KEY_UP,    COMMAND_UNKNOWN  },*/
                   /* {KEY_LEFT,  COMMAND_UNKNOWN  },    {KEY_OK,    COMMAND_UNKNOWN  },    {KEY_RIGHT, COMMAND_UNKNOWN  },*/
                                                     /*  {KEY_DOWN,  COMMAND_UNKNOWN  },*/
    {KEY_BACK,  SCENE_SELECTION  },                                                                        /*{KEY_SRC,   COMMAND_UNKNOWN  },*/
    {KEY_VOLUP, YAMAHA_VOL_PLUS  },                      {KEY_MUTE,  YAMAHA_MUTE_TOGGLE},                  /*{KEY_CHUP,  COMMAND_UNKNOWN  },*/
    {KEY_VOLDO, YAMAHA_VOL_MINUS },                   /* {KEY_REC,   COMMAND_UNKNOWN  },*/                 /*{KEY_CHDOW, COMMAND_UNKNOWN  },*/
    {KEY_RED,   SCENE_TV_FORCE   },    {KEY_GREEN, SCENE_FIRETV_FORCE},  {KEY_YELLO, SCENE_CHROMECAST_FORCE},{KEY_BLUE,  SCENE_APPLETV_FORCE},
  };

Remark: you can both use the command COMMAND_UNKNOWN or simply comment out the definition to have no definition for a key.

For each scene, you can override the key definitions. Have a look in files scenes/scene_allOff.cpp, scenes/scene_TV.cpp etc. for an example. In a scene file, you can override none of the keys, only some, or all.

The same is possible for GUIs. They can also override key definitions as long as the GUI is shown. Have a look in file gui_smarthome.cpp how it is done.

At runtime, a GUI specific definition is used first. If not available, the scene specific definition is used. If not available, the default definition is taken. If no default definition is available, then no command is executed.

Touchscreen

The touchscreen can also be used to get status information about devices or to send commands. The touchscreen is organized in so called tabs. You can swipe left or right to go to the next tab.

If you want a specific action to happen, simply call the commandHandler like executeCommand(SAMSUNG_POWER_OFF); in the callback function of a GUI element (button, slider etc.). See one of the examples on how to do it, e.g. devices/misc/device_smarthome/gui_smarthome.cpp.

You can trigger scene specific commands based on the currently active scene from the gui. You know at runtime which scene currently is active, so that you can do context specific actions. But it has to be done in code by yourself. See how it is done in guis/gui_numpad.cpp.

OMOTE is using the Light and Versatile Graphics Library. If you want to learn more about how to use lvgl, this is a good starting point.

The library is very powerful, but also resource demanding, at least for an ESP32. Lack of memory has been an issue in the past. Most of the problems are solved, but you should read section memory.

Important: If you are creating new guis, you may find it difficult and time consuming to write correct lvgl code and to test it on the ESP32. You can significantly speed up the development of guis with use of the software simulator

Scene specific guis

There is a so called "main_gui_list" which consists by default of all guis you have registered. You can swipe horizontally between the guis. By default, the first gui in the "main_gui_list" is the "scene selection gui".

Additionally, you can define scene specific gui_lists which are used whenever a scene is active. When a scene is active, you can switch between the scene specific gui list and the "main_gui_list", e.g. for starting a different scene or changing some settings of the OMOTE.

You can have the same gui in one or more of these lists. E.g. the numpad can be used to send numbers to different devices, based on the scene (see "Scene TV" and "Scene Fire TV").

Switching from the "main_gui_list" go the scene specific list is done simply by selecting a scene on the "scene selection gui".
Remark: if you select a scene on the "scene selection gui", the start sequence of the scene is only sent if the scene was not yet active. If you want to force the start sequence of the scene to be resent, you need to long press the scene on the gui or use the corresponding hardware key.

Switching from the scene specific list back to the scene selection gui in "main_gui_list" can be done in several ways:

• do a "swipe down" from top of the screen
• click on the scene name in the header bar at top of the screen
• click on the gui name in the page indicator at bottom of the screen
• hit the "back" button on the hardware keyboard
• programmatically with executeCommand(SCENE_SELECTION);

You can also directly jump from the scene specific list to the "main_gui_list" and back. The last active GUI from each list is automatically activated again. This can be achieved by

• pressing the hardware key "record"
• programmatically by executeCommand(SCENE_BACK_TO_PREVIOUS_GUI_LIST);

This is how the gui lists of the predefined example look like:

If you want to have only one single list of guis (the "main_gui_list"), available in all of the scenes, and no scene specific gui lists, then set -D USE_SCENE_SPECIFIC_GUI_LIST=0 in platformio.ini.
If you also don't want to have the "scene selection gui", comment line register_gui_sceneSelection(); in file main.cpp

How to

Normally, you only have to add your WiFi credentials, devices, guis and scenes. So only these files are of interest:

src/
├── devices/
│   ├── AVreceiver/
│   │   └── device_yamahaAmp/
│   ├── keyboard/
│   │   ├── device_keyboard_ble/
│   │   └── device_keyboard_mqtt/
│   ├── mediaPlayer/
│   │   └── device_appleTV/
│   ├── misc/
│   │   ├── device_smarthome/
│   │   └── device_specialCommands.cpp/.h
│   └── TV/
│       └── device_samsungTV/
├── devices_pool/
│   ├── AVreceiver/
│   │   ├── device_denonAvr/
│   │   └── ...
│   ├── keyboard/
│   │   ├── device_keyboard_ble/
│   │   └── device_keyboard_mqtt/
│   ├── mediaPlayer/
│   │   ├── device_lgbluray/
│   │   └── ...
│   ├── misc/
│   │   ├── device_smarthome/
│   │   ├── ...
│   │   └── device_specialCommands.cpp/.h
│   └── TV/
│       ├── device_lgTV/
│       └── ...
├── guis/
│   ├── gui_numpad.cpp/.h
│   └── ...
├── scenes/
│   ├── scene__defaultKeys.cpp/.h
│   ├── scene_TV.cpp/.h
│   └── ...
├── secrets_override.h
└── secrets.h

The devices have separate folders for TVs, AV receivers, media players, and so on. Try to put your devices into the correct folder. It is not necessary, but helps to keep the structure clean.

add WiFi credentials

Copy file secrets_override_example.h to file secrets_override.h. This file will never by under version control, so you can safely put your secrets in this file.

use one of the devices from the devices_pool

The folder /src/devices_pool is intended for a pool of devices already contributed by the community of OMOTE. Files in this folder won't get compiled by default.

If you want to use one of these devices, then

• copy the corresponding device folder from /src/devices_pool/... to /src/devices/...
• add the corresponding call to register_device_*() into /src/main.cpp (and include the device header file in /src/main.cpp as well)
• start using the commands defined by the device, e.g. by putting them in the hardware key map of scene__default.cpp or of a specific scene

register new device

• create new folder, similar to devices/TV/device_samsungTV/
• take an existing device as template and copy e.g. the two files from device_samsungTV/ to your new folder and rename files
• rename function register_device_samsungTV()
• list all the available commands in the header file
• register the commands in the cpp file
• register device in main.cpp with a call to register_device_*()
• if you need a different kind if IR code to be sent which is currently not supported, add it to enum IRprotocols in file /src/applicationInternal/hardware/hardwarePresenter.h and also add it in /hardware/ESP32/infrared_sender_hal_esp32.cpp or ask for support

Remark: to save memory, remove the devices not needed by you, at least by commenting the call to register_device_*()

register new gui

• take an existing gui as template and copy the two files (.h and .cpp)
• if the new gui is specific for a single device, then place the files in the same folder as the device
• if the new gui is for more than one device or not related at all to a device, place it in folder /src/guis/
• each gui must have a unique name which is defined in the .h file
• rename function register_gui_*()
• register gui in main.cpp with a call to register_gui_*()
• if you want to deactivate a gui, simply comment out the corresponding line in main.cpp
• if you want only specific guis to be available in the "main_gui_list", then adjust the following line in main.cpp:
  main_gui_list = {tabName_sceneSelection, tabName_smarthome, tabName_settings, tabName_irReceiver};

You can define a command to directly activate a GUI. See file gui_smarthome.cpp and command GUI_SMARTHOME_ACTIVATE on how it is done. As an example, the command GUI_SMARTHOME_ACTIVATE is used in the default key map for the hardware key "stop".

Remark: the combination of a specific command to activate a GUI and using the command SCENE_BACK_TO_PREVIOUS_GUI_LIST (by default mapped to hardware key "record") is very powerful to quickly jump to a specific GUI (for finetuning a device, controlling the smart home, ...) and jump back to where you have been before.

Important: If you are creating new guis, you may find it difficult and time consuming to write correct lvgl code and to test it on the ESP32. You can significantly speed up the development of guis with use of the software simulator

register a scene

• take an existing scene as template and copy the two files (.h and .cpp)
• place them in folder scenes/
• each scene must have a unique name which is defined in the .h file
• rename the function register_scene_*()
• define a command with which the scene can be activated
• define scene specific overrides for the hardware keys in scene_setKeys_*()
• define start and end sequences for the scene scene_start_sequence_*() and scene_end_sequence_*()
• define a scene specific gui list, if you want to have one. See "scene_TV" for a scene where such a list is defined, and "scene_chromecast" where no such list is defined
• register scene in main.cpp with a call to register_scene_*()
• if you want to deactivate a scene, simply comment out the corresponding line in main.cpp
• if you want only specific scenes to be available on the "scene selection gui", then adjust the following line in main.cpp:
  set_scenes_on_sceneSelectionGUI({scene_name_TV, scene_name_fireTV, scene_name_chromecast, scene_name_appleTV});

register special commands

• you can define own commands like MY_SPECIAL_COMMAND which is not binded to a single device. Can be used at runtime for whatever you like, e.g. for sending a sequence of commands. Search in the code for MY_SPECIAL_COMMAND to see an example on how to do it.

subscribe to MQTT topics

An example on how to subscribe to MQTT topics and to propagate received values to the gui is available.

Topics you want to subscribe to need to be added in mqtt_hal_esp32.cpp and mqtt_hal_windows_linux.cpp. Search for topic OMOTE/test to see an example.

Received MQTT messages are currently simply displayed on the receiver GUI tab.

Most likely you want to add special handling when a MQTT message has arrived. For this, you can extend function void receiveMQTTmessage_cb(...) in file /src/applicationInternal/commandHandler.cpp

You can send a MQTT message from OMOTE to your home automation software as soon as WiFi is available in OMOTE. As response, your home automation software could send the states of the smart home devices known to OMOTE. See function void receiveWiFiConnected_cb() in file /src/applicationInternal/commandHandler.cpp

Memory

The ESP32 has only limited memory. If you are using WiFi, BLE and lvgl at the same time, you quickly run into memory issues.

Please see this very detailed discussion about memory usage and optimization, if you are interested. If you are only interested in the most important results of this analysis, please stay here, and continue reading.

How memory optimization is done

The conclusion of the memory analysis is that you cannot have all the GUIs in memory at the same time. To keep memory usage low, OMOTE always has only three GUIs in memory at the same time. The visible GUI is always the one in the middle. Since during swiping you can already see the previous or next GUI, these two also need to be in memory. As soon as the swiping animation ended, all GUIs are deleted and the next three are created, with the new visible one in the middle.

It has been put a lot of effort into achieving this dynamic recreation of GUIs in a way that it is not recognizable by the user. It is not 100% perfect, but almost 😄 The big advantage of this approach is that in practice you are almost not limited in the number of GUIs you want to have. In tests 100 GUIs worked without any problem. Of course, the size of the flash where the program code is and other things are also limiting, but in reality, you should be able to have all the GUIs you want to have.

Of course, the three GUIs being in memory must not become so big that these three don't fit into memory.

How to see how much memory is used

There are two ways to get memory statstics:

1. on the serial output you get detailed information about memory usage every 5 sec
2. a small status text at top of the screen, updated every second, colored in red if resources are running out

To activate the status text on the screen, simply go the settings GUI and scroll to the bottom.

The first is "freep heap"/"total heap", the second is "free lvgl memory"/"total lvgl memory".

There are two things you must observe:

• lvgl memory must never exceed the fixed memory limit lvgl has (32k by default)
• ESP32 free heap should stay big enough for memory peaks Wifi or BLE need

Whenever lvgl memory is used more than 80%, the corresponding label turns red. Whenever ESP32 free heap will be below a certain threshold, the corresponding label turns red. The thresholds for the ESP32 heap are 15.000 bytes if WiFi and BLE are turned on and 5.000 bytes if both are deactivated. You can change these thresholds in file memoryUsage.cpp

How to optimize the size of lvgl static memory and ESP32 heap

What can you do if lvgl memory is getting low:

• reduce the number of widgets on the GUIs
• increase lvgl static memory (but to the cost of less ESP32 heap)

What can you do if ESP32 heap is getting low:

• reduce lvgl static memory, if possible (-D LV_MEM_SIZE= in platformio.ini)
• deactivate WiFi and MQTT if possible
• deactivate BLE and use the ESP32 emulating a hardware keyboard
• use only one draw buffer for lvgl, this saves 15360 bytes RAM. Comment out #define useTwoBuffersForlvgl in file guiBase.h

Architectural overview

This section is only for those who want to understand how everything works behind the scenes. It is not necessary to use and adapt the software to your needs.

The software is divided into components which have only loose coupling, following the MVP pattern.

commandHandler

• devices register at startup at the commandHandler with the commands they can execute
• can be called at runtime to execute a registered command
• uses the hardware which is needed to execute the command (IR sender, MQTT, BLE)

sceneRegistry

• scenes can register at setup to be available at runtime
• each scene must have a unique name
• each scene must provide functions to start and end a scene (which can use the commandHandler to send commands)
• can provide scene specific hardware key definitions

sceneHandler

• will be called by commandHandler to switch a scene. Since switching a scene is a more complex task, this is forwarded by the commandHandler to the sceneHandler

guiRegistry

• every tab available on the touchscreen needs to be registered at the guiRegistry
• each gui must provide functions the create the content of the tab
• guiRegistry will be called by guiBase (which is called by main) to create the content of the touchscreen
• if you want to deactivate a gui, simply comment out the corresponding line in main.cpp

keys

• hardware keys already have been explained in detail above

folder structure for applicationInternal/

Here you can find the corresponding files in the workspace:

src/
└── applicationInternal/
    ├── gui/
    │   ├── guiBase.cpp/.h
    │   ├── guiMemoryOptimizer.cpp/.h
    │   ├── guiRegistry.cpp/.h
    │   └── guiStatusUpdate.cpp/.h
    ├── hardware/
    │   ├── arduinoLayer.cpp/.h
    │   └── hardwarePresenter.cpp/.h
    ├── scenes/
    │   ├── sceneHandler.cpp/.h
    │   └── sceneRegistry.cpp/.h
    ├── commandHandler.cpp/.h
    ├── keys.cpp/.h
    └── memoryUsage.cpp/.h

components involved at setup

components involved for executing a simple command triggered by a hardware key

components involved for executing a scene switch