Reworked software - modular approach

[KlausMu]

Dear all,

I would like to start a discussion on how a good starting point of the software could look like for people willing to use OMOTE.

I was at the same point some weeks ago when I started using OMOTE. With the current software (with software I mean the file "main.cpp") I had some issues

• one single file "main.cpp" with more than 1000 lines, not easy to understand
• no bluetooth support
• only some basic code or examples, being far from complete what a full featured remote would do (this is not at all meant offensive)
• some things were simplified (keypad, wifi connection)

So I started to break up the existing code into several files. While doing this, I understood what everything is good for.
It's much easier to understand a software if you have separate files for every major thing.

Finally I ended up with a new version including the following features:

• definition of devices in separate files (one file for each device). The device files describes the capabilities of the device.
• definition of the gui for devices in separate files (one file for each screen)
• use of a command pattern so that frontends (keypad and touch panel) can send commands to a commandHandler without the need to know how to send a command to a device (IR, MQTT, BLE ...)
• the low level part of sending a command (IR, MQTT, BLE ...) is done by the command handler
• support of scenes (done by the command handler)
• separate files for the rest: battery, user_led, sleep, preferences
• keypad handling distinguishes short press, long press and hold
• support of a BLE keyboard, including navigation (UP, DOWN, ...), BACK, HOME, MENU, PLAYPAUSE, REWIND, FASTFORWARD, MUTE, VOLUME
• a robust wifi connection, with automatic wifi reconnect when connection got lost
• a gui for an IR receiver
• setting sleep timout via GUI now works, not only dropdown without functionality
• secrets as wifi credentials placed in a separate file not under version control
• and maybe some more minor things I forgot
• screens for "AppleTV" and "Smarthome" are unchanged, but still only there as an example. E.g. there is no device definition for them, and hence they don't use the commandHandler

From my perspective, this approach would be a much better starting point for people willing to use and to configure their OMOTE to their needs.

Don't get me wrong, people would still need to modify the source code. My approach is still far away from what e.g. Logitech Harmony is providing with their Windows App or Android App where you can configure everything with a nice user interface. To achieve this A LOT MORE effort had to be put into the software.

Here are some hints on how to use and to understand the code, if you are interested:

How to add a device:

Create two files similar to e.g.

device_yamaha.h
device_yamaha.cpp

and call init_<yourDevicename>_commands() in main.cpp

How to add a gui (a screen):

Example for numpad:

gui_numpad.h
gui_numpad.cpp

and call init_gui_numpad() and init_gui_pageIndicator_numpad() in the main gui file gui.cpp

How to add a completely new type of commands:

Currently the commands "IR global cache", "IR NEC", "IR SAMSUNG", "MQTT", "BLE Keyboard" and "Special" (e.g. for Scenes) are supported.
If you need more type of commands, add them in commandHandler.h
The actual sending of a command using the different technologies like infrared, MQTT and so on is done in commandHandler.cpp
Also the handling of scenes is done there.

How to define how the keypad works

This is done in keys.cpp
You can define for every key how it should behave in case of a long press. Three modes are supported:

  SHORT,             // if you short press or old a key on the keypad, only one single command from keyCommands_short is sent -> best used if you do not want a command to be sent more than once, even if you press the key (too) long
  SHORT_REPEATED,    // if you hold a key on the keypad, a command from keyCommands_short is sent repeatedly -> best used e.g. for holding the key for "volume up"
  SHORTorLONG,       // if you short press a key, a command from keyCommands_short is sent once.
                     // if you hold a key on the keypad, a command from keyCommands_long is sent (no command from keyCommands_short before)
                     // -> best used if a long key press should send a different command than a short press

For example I am using:

std::map<char, repeatModes> keyRepeatModes {
                                                                             {'o', SHORT         },
  {'=', SHORT         },   {'<', SHORTorLONG   },   {'p', SHORT         },   {'>', SHORTorLONG   },
  {'c', SHORT         },                                                     {'i', SHORT         },
                                         {'u', SHORT         },
                {'l', SHORT         },   {'k', SHORT         },   {'r', SHORT         },
                                         {'d', SHORT         },
  {'b', SHORT         },                                                     {'s', SHORT         },
  {'+', SHORT_REPEATED},                 {'m', SHORT         },              {'^', SHORT         },
  {'-', SHORT_REPEATED},                 {'e', SHORT         },              {'v', SHORT         },
  {'1', SHORT         },   {'2', SHORT         },   {'3', SHORT          },  {'4', SHORT         },
};

Then you can define for every key which command should be send for a short press, e.g.

std::map<char, std::string> keyCommands_short {
                                                                                               {'o', ALLDEVICES_POWER_TOGGLE},
/*{'=', },*/                      {'<', KEYBOARD_REWIND},      {'p', KEYBOARD_PLAYPAUSE},      {'>', KEYBOARD_FASTFORWARD},
  {'c', KEYBOARD_HOME},                                                                        {'i', KEYBOARD_MENU},
                                                {'u', KEYBOARD_UP},
                    {'l', KEYBOARD_LEFT},       {'k', KEYBOARD_SELECT},     {'r', KEYBOARD_RIGHT},
                                                {'d', KEYBOARD_DOWN},
  /* {'b', }, */                                                                               {'s', KEYBOARD_BACK},
  {'+', YAMAHA_VOL_PLUS},                       {'m', YAMAHA_MUTE_TOGGLE},                     {'^', SAMSUNG_CHANNEL_UP},
  {'-', YAMAHA_VOL_MINUS},                      /* {'e', }, */                                 {'v', SAMSUNG_CHANNEL_DOWN},
  {'1', SCENE_TV},                {'2', SCENE_FIRETV},         {'3', SCENE_CHROMECAST},      /*{'4', },*/
};

and for a long press:

std::map<char, std::string> key_commands_long {
  {'<', KEYBOARD_REWIND_LONG},
  {'>', KEYBOARD_FASTFORWARD_LONG},
};

If you need to, you can also change the repeat rate:

int repeatRate = 125; // in millisecons

If you are interested, please try the following branch:
https://github.com/KlausMu/OMOTE/tree/modular-approach

git clone -b modular-approach https://github.com/KlausMu/OMOTE

Looking forward to your feedback.

[CoretechR]

Hi Klaus, I am deeply impressed with how you have reorganized the code and totally agree with you that 1000+ lines of code in a single file is cluttered and confusing. I'm in favor of anything that makes the current software more accessible and easier to modify. And the way commands, devices and GUI screens are managed seems much cleaner and clearer. The long term goal for this project is still to be able to tweak these things without making changes to the code. But I think your approach is already very good groundwork for that. With some examples for devices and commands and a few new pages of documentation, this will definitely be very helpful for new users.

If that's OK with you, I would love to include parts (or all) of your branch in the software. What would be the best way to go about this?

If anyone else has any suggestions on this topic, please feel free to bring them in. I still think @MatthewColvin's Abstraction branch is great. I just have the fear that the object-oriented approach is too complicated to modify for some hobby users. At least that's the case for me.

[KlausMu]

Hi Max, I'm glad you like it :-)
Sure, you can take everything or only parts. I tried to commit in this branch only things that are of general use. But of course, if you think something should be changed, I would love to hear about it.

• so I would propose that in the end you take everything. If something needs to be changed before, please tell me
• I can also write a more detailed guideline on how to understand, use and modify the code. What would you like to be explained? Where would be the best place to have the guideline?
• we could also add more devices and screens as examples. People could then choose what to use or what could be the best starting point for their own devices. Any ideas about other devices?
• what also could easily be added is that the keypad and screens are aware of the scene we are currently in, so that they behave accordingly. For me there was no need for this, but maybe others see it differently. Would add some more complexity, but I think still understandable

I tried to keep the code as simple as possible and to be readable as possible. I'm afraid, if there will ever be a solution without any need to manually change the code, only having a nice GUI where you can click together everything you need, the code itself would be so complicated, that only few people will be able to maintain it. No matter if object oriented or not. But we will see what will happen in the future ...

I agree, as long as people have to modify the code, object oriented code can be overwhelming.

BTW: Linus Thorvalds also does not like C++, but sometimes he is very strict in his opinions ...

[CoretechR]

You are probably right that a completely UI-based customization will be extremely complex to implement. Maybe it's enough to start with a set of common UI screens and devices. I have some older IR remotes that I can try to turn into device profiles, just to create a bigger set of examples. But I'm not so sure about other types of (Bluetooth) devices. The fire TV and Chromecast should be very popular. And surely the community can also contribute new devices.
Is there an easy way to organize these device and user interface files into folders? Then it wouldn't matter how many more are added.

The documentation on this GitHub repo is definitely something that has to be expanded (Currently there are 4 whole lines of texts about the software). It would probably be best to split this into multiple pages about hardware, a basic code explanation, getting started, configuring a new device, etc. If you want to contribute a guide for your code and functions, that would be awesome. But of course you don't have to do this all by yourself.

So far, everything about the new code that I have been able to test works. One small thing is disabling Bluetooth. If "ENABLE_BLUETOOTH" and "ENABLE_KEYBOARD_BLE" are no longer defined, the compilation fails. Maybe I'm doing something wrong, but it would be nice to be able to turn Bluetooth off when it's not needed, since that increases the boot time considerably.

[KlausMu]

But I'm not so sure about other types of (Bluetooth) devices. The fire TV and Chromecast should be very popular.

FireTV is already fully supported through the BLE keyboard or the MQTT keyboard.

The old Chromecast cannot be controlled in any way. You can only switch the HDMI input of your TV or AV receiver. I have an old Chromecast.
The new Chromecast seems to be like a regular Android TV, comes with a remote and should also work with the already available BLE keyboard.

If someone wants to pair both a fire TV and a new Chromecast to the OMOTE BLE keyboard, this could need some more work. Seems to be possible, but not yet directly supported
I think, someone will complain if trying to do so :-) I can then have a look.

Is there an easy way to organize these device and user interface files into folders? Then it wouldn't matter how many more are added.

Sure. I reorganized everything into folders. Please have a look if you like it. Devices and corresponding GUI (if there even is one) are in the same folder.

While doing this, I also created devices for AppleTV (Sony IR) and smarthome (MQTT).

The documentation on this GitHub repo is definitely something that has to be expanded (Currently there are 4 whole lines of texts about the software). It would probably be best to split this into multiple pages about hardware, a basic code explanation, getting started, configuring a new device, etc. If you want to contribute a guide for your code and functions, that would be awesome. But of course you don't have to do this all by yourself.

I would love to do it for the software. We could use the GitHub wiki of the repo for this, like I did here

So far, everything about the new code that I have been able to test works. One small thing is disabling Bluetooth. If "ENABLE_BLUETOOTH" and "ENABLE_KEYBOARD_BLE" are no longer defined, the compilation fails. Maybe I'm doing something wrong, but it would be nice to be able to turn Bluetooth off when it's not needed, since that increases the boot time considerably.

You did nothing wrong. As soon as a device is disabled or completely removed from code, it can happen that the commandHandler, the GUI or the keys are still using the commands from that device. This is what happened here. Normally, if you change something in the devices, you have to change the code of the different frontends (gui, keys, commandHandler) accordingly.

But I understand, as it is likely that people want to deactivate WiFi, BLE or both (and there are already #defines for that), deactivating should work out of the box. I changed the code, so now you can simply deactivate it without further changes.

We could also deactivate the BLE keyboard by default.

[CoretechR]

The new Chromecast seems to be like a regular Android TV, comes with a remote and should also work with the already available BLE keyboard.

Oh, so the BLE keyboard can already control lots of different devices as long as they recognize a keyboard. That's perfect, I didn't know that it's this simple. It even shows up on my PC and iPhone and is recognized right away. Being able to connect to multiple devices would be even better, but you're right, we don't have to solve this immediately.

Sure. I reorganized everything into folders. Please have a look if you like it. Devices and corresponding GUI (if there even is one) are in the same folder.

What I originally had in mind was a separate folder for devices and another for GUIs. But one folder for each device seems perfectly fine to me. That way each device has its commands and gui in one place. Thanks!

But I understand, as it is likely that people want to deactivate WiFi, BLE or both (and there are already #defines for that), deactivating should work out of the box. I changed the code, so now you can simply deactivate it without further changes.

Bluetooth is probably a useful features for a lot of people, especially with the keyboard, so we can leave it enabled by default.

In any case, I hope I will have time tomorrow to try and adapt the modular code so that I can use it to control my devices. Simply so that I can better understand how everything works. One thing is that I would like the hardware keyboard to change commands based on the current device and I am not sure what's the best way to achieve this. Does this require an extra keyCommands array for each device? The current software switches the hardware keyboard assignment based on the device that is selected on the screen. At least that is what I am used to with universal IR remotes, but maybe it is not so intuitive to have the button change their assignment like that.

[KlausMu]

One thing is that I would like the hardware keyboard to change commands based on the current device and I am not sure what's the best way to achieve this. Does this require an extra keyCommands array for each device? The current software switches the hardware keyboard assignment based on the device that is selected on the screen. At least that is what I am used to with universal IR remotes, but maybe it is not so intuitive to have the button change their assignment like that.

Well, I think, first we should clarify some terms.

A device, as the term is currently used in the software, can be different things:

• a media player like Apple TV, Fire TV, Chromecast, DVD player, or the TV itself
• a TV or generally spoken a monitor
• an AV receiver
• something like the BLE keyboard or MQTT keyboard, used to control e.g. a media player
• something like the "smarthome" or LED stripes, which can be controlled via MQTT
• or even something else we currently don't think of

In most cases you want to use scenes, e.g.

• watching TV
• watching FireTV
• watching Chromecast
• watching AppleTV

Each scene normally uses exactly one media player and some of the other devices.

So I think what we need is a keypad layout for each scene, right?

Scenes are a concept already available in the software. I would extend this so that each scene:

• defines a start sequence (which devices should be turned on, which input should be selected and so on)
• defines a stop sequence
• defines a keypad layout

Switching the scene is currently done with one of the four lower buttons of the keypad. If someone needs more scenes, HOLD could also be used, so overall it is easily to support up to eight scenes.

Maybe you have seen that the variable "currentDevice" is no longer available in the software.
Instead now we have "currentScene" and "currentScreen". I think this reflects much better what is going on.

Some or even all screens may be independent in terms that they are not related to a scene at all

• smarthome
• IR receiver
• settings
• numpad

Only the "AppleTV" screen is depending on the scene, or to be more precise, it should only be available if the scene "AppleTV" is active.

In the orginal software, all the screens were treated as devices (at least currentDevice was set), which does not really fit in several ways, I think. That's why I removed "currentDevice".

I hope you like the idea of scenes. BTW this is exactly how Logitech Harmony is doing it. I think this concept is proven. I used a Harmony for years and am still using it.

So my proposal is:

• I would extend the use of scenes: start sequence, stop sequence, definition keypad layout
• new folder where all the scenes go into
• for the screens, I think there is no need for a scene dependant command definition. You can handle all this in the screen. Currently I think none of the screens is really dependant of the scene. If someone ever needs this, you can get the current scene via the variable "currentScene" and react corresponding in the GUI code (e.g. sending commands corresponding to the scene we are currently in)

[KlausMu]

As I have seen, you are using RC5 IR commands. I added this to the commandHandler.
So now you can define commands in your Apple TV device like this:

  commands[APPLETV_KEYPAD_EVENT_1]           = makeCommandData(IR_SONY, {"0x70"});
  commands[APPLETV_GUI_EVENT_USER_DATA]      = makeCommandData(IR_SONY, {});  // payload must be set when calling commandHandler

or for your Technisat like this

  commands[TECHNISAT_KEYPAD_EVENT_1]         = makeCommandData(IR_RC5, {"0x69"});
  commands[TECHNISAT_GUI_EVENT_USER_DATA]    = makeCommandData(IR_RC5, {});  // payload must be set when calling commandHandler

and use the command in a screen like this

  executeCommand(APPLETV_GUI_EVENT_USER_DATA, std::to_string(50 + (int)e->user_data));
  executeCommand(TECHNISAT_GUI_EVENT_USER_DATA, std::to_string(virtualKeyMapTechnisat[(int)target->user_data])));

For use in the keypad, you have to explicitely define commands for every key (as with APPLETV_KEYPAD_EVENT_1).
You cannot use the dynamic payload here, because the keypad implementation does not generate any dynamic payload.
But I think, once you have defined the commands, this is even more understandable than the dynamic payloads.

If you need more support, you can send me a list of the commands in keyMapTechnisat and virtualKeyMapTechnisat and what they mean (e.g. names for the commands), then I can create a Technisat device for you.

Same for Apple/Sony, With

IrSender.sendSony((keyCode/ROWS)*(keyCode%ROWS), 15)

you simply don't know what is going on.

[KlausMu]

And I did some more movements of files and folders. Keys are now in folder "gui_general_and_keys".
So overall we have:

• "device_*": the devices, defining capabilities and available commands of the devices. The GUIs of the devices (also in these folders, if there is a specific gui for a device) can send the commands
• "gui_general_and_keys": general purpose guis like settings, IR receiver..., and also the keypad. They all can send commands
• "hardware": the low level part like WiFi, MQTT, IR, battery, tft, sleep, user_led

Between the "devices_*"and "gui_general_and_keys" is the "commandHandler" as the glue between all. The commandHandler is located in the base directory. Hope this makes things clearer.

[CoretechR]

So my proposal is:

• I would extend the use of scenes: start sequence, stop sequence, definition keypad layout
• new folder where all the scenes go into
• for the screens, I think there is no need for a scene dependant command definition. You can handle all this in the screen. Currently I think none of the screens is really dependant of the scene. If someone ever needs this, you can get the current scene via the variable "currentScene" and react corresponding in the GUI code (e.g. sending commands corresponding to the scene we are currently in)

To be honest, I never actually owned a harmony remote, so thanks for clarifying how it works. From what I read in the manuals, I thought that the physical buttons change their mapping based on the device that it has to control. Logitech mentions activity control buttons, which makes sense since they can control multiple different devices with one layout. The term activity might be better than device for what I had in mind originally.

I like how you implemented scenes and having sequences is probably very useful as well. For BLE and MQTT devices that should not be a problem but I am a little worried about IR devices. The harmony has a hub and even extra IR transmitters but the OMOTE has only its tiny IR LED. How can we make sure that the sequence is received by all the devices?

[KlausMu]

Logitech mentions activity control buttons, which makes sense since they can control multiple different devices with one layout. The term activity might be better than device for what I had in mind originally.

Right, Logitech calls it activity. But I think a more common term now (Logitech introduced this years ago) is scene, so I would use this term.
You can define for each activity (= scene), which key controls which device.
We would do exactly the same.

The harmony has a hub and even extra IR transmitters but the OMOTE has only its tiny IR LED. How can we make sure that the sequence is received by all the devices?

IR commands should be sent repeatedly. Of course this only works for commands like PowerOn, not for PowerToggle.
Harmony is doing the same. You can define for each device how Harmony should behave.

I think we don't need this general approach in OMOTE, since for a scene you will simply programmatically define the sequence of commands and delay between them. This is even more detailed than what Harmony can do.
If someone wants to send commands repeatedly in a scene, he or she can define so.
See in file "commandHandler.cpp" line 170 and following how I did it.

[KlausMu]

To have a more reliable IR connection, we could also create a small device, having only an ESP32 and an IR LED in it, nothing else. This small device could be placed next to the IR receiving devices (TV, amp, ...) The OMOTE sends MQTT messages to it, and the small device sends the IR signals.

A good name for such a device could be "OMOTE IR satellite". Easy to do, one weekend.

The satellite could also be used if the amp is in a cupboard without line of sight to the OMOTE. Harmony also had a small additional IR sender for this purpose.

For me the IR performance of the OMOTE is good. Not as good as the Harmony Hub sitting next to the devices, but ok. I'm 3-4 meters away from the devices. If sometimes an IR signal is not recognized, I repeat it. Not a big thing. Also the start sequence for a scene can be repeated with an additional key hit.

And this also happens sometimes with the Harmony Hub as well ...

And I have seen here a comment about a hardware change for having a more powerful IR LED in the OMOTE. Let's see what future brings to us ...

[MatthewColvin]

Something I did in my branch was spawned up a thread and bumped its priority when the user was expecting to use the IR receiver on device. This allowed me to get IR RX working 100% of the time at the cost of the user having to enable RX. I think I may have done the same thing for TX but I am no sure.

[KlausMu]

Interesting! I will have a look at how you did it.
Was RX working only with this trick? Because for me IR RX works out of the box. I also added a switch to activate it, it seems not necessary to have it running all the time.

[CoretechR]

Right, Logitech calls it activity. But I think a more common term now (Logitech introduced this years ago) is scene, so I would use this term.
You can define for each activity (= scene), which key controls which device.
We would do exactly the same.

Yes, that is something I find very useful. Especially if you have multiple IR devices with lots of buttons that cannot be mapped to a single keyboard layout. Would this mean that we need a keyRepeatModes, keyCommands_short and key_commands_long for each scene? Maybe these three can be combined into one keymap. At least then there is only one keymap per scene. Or how would you do this?

Initially I was against the idea of a hub. The remote was supposed to be completely independent so that you don't need additional hubs and docks. But I can absolutely see the benefits of having some sort of IR repeater. Especially since it would be optional. The remote could check if it is connected to its satellite and decide if it should send the IR command itself or make the hub do it via MQTT. And it would be pretty easy to build one. I guess I will have to put that on my todo list :)
I have a new board revision on this list for some time, too. There are a lot of small things that I would like to change and I was always reluctant to do this because I did not want to break compatibility. But now that the memory issue is out of the way, I can continue with the regular ESP32-PICO-D4.

[KlausMu]

Yes, that is something I find very useful. Especially if you have multiple IR devices with lots of buttons that cannot be mapped to a single keyboard layout. Would this mean that we need a keyRepeatModes, keyCommands_short and key_commands_long for each scene? Maybe these three can be combined into one keymap. At least then there is only one keymap per scene. Or how would you do this?

Maybe I would define a default map, and delta maps for each scene that overwrite only the keys you have in the delta map. I expect that the scenes share a lot of common keys. And as a user you could decide to simply put all the keys into the delta map, which would overwrite all keys from the default map. You could do it as you want.

About combining: currently we have three maps. The layout of the code is so that you can understand or see where each key is. Not neccessary for the compiler of course, but at least for my eyes much easier to read.

std::map<char, repeatModes> keyRepeatModes {
                                                                             {'o', SHORT         },
  {'=', SHORT         },   {'<', SHORTorLONG   },   {'p', SHORT         },   {'>', SHORTorLONG   },
  {'c', SHORT         },                                                     {'i', SHORT         },
                                         {'u', SHORT         },
                {'l', SHORT         },   {'k', SHORT         },   {'r', SHORT         },
                                         {'d', SHORT         },
  {'b', SHORT         },                                                     {'s', SHORT         },
  {'+', SHORT_REPEATED},                 {'m', SHORT         },              {'^', SHORT         },
  {'-', SHORT_REPEATED},                 {'e', SHORT         },              {'v', SHORT         },
  {'1', SHORT         },   {'2', SHORT         },   {'3', SHORT          },  {'4', SHORT         },
};

std::map<char, std::string> keyCommands_short {
                                                                                               {'o', ALLDEVICES_POWER_TOGGLE},
/*{'=', KEYBOARD_PLAYPAUSE},*/    {'<', KEYBOARD_REWIND},      {'p', KEYBOARD_PLAYPAUSE},      {'>', KEYBOARD_FASTFORWARD},
  {'c', KEYBOARD_HOME},                                                                        {'i', KEYBOARD_MENU},
                                                {'u', KEYBOARD_UP},
                    {'l', KEYBOARD_LEFT},       {'k', KEYBOARD_SELECT},     {'r', KEYBOARD_RIGHT},
                                                {'d', KEYBOARD_DOWN},
  /* {'b', }, */                                                                               {'s', KEYBOARD_BACK},
  {'+', YAMAHA_VOL_PLUS},                       {'m', YAMAHA_MUTE_TOGGLE},                     {'^', SAMSUNG_CHANNEL_UP},
  {'-', YAMAHA_VOL_MINUS},                      /* {'e', }, */                                 {'v', SAMSUNG_CHANNEL_DOWN},
  {'1', SCENE_TV},                {'2', SCENE_FIRETV},         {'3', SCENE_CHROMECAST},        {'4', YAMAHA_STANDARD},
};

std::map<char, std::string> key_commands_long {
  {'<', KEYBOARD_REWIND_LONG},
  {'>', KEYBOARD_FASTFORWARD_LONG},
};

If you don't like three maps (or delta maps) for each scene, these three maps could be combined into a single map which would become a litte bit wider

std::map<char, std::list<std::string>> keyCommands {
                                                                                                                                      {'o', SHORT,       ALLDEVICES_POWER_TOGGLE},
/*{'=', SHORT, KEYBOARD_PLAYPAUSE},*/  {'<', SHORTorLONG, KEYBOARD_REWIND, KEYBOARD_REWIND_LONG},  {'p', SHORT, KEYBOARD_PLAYPAUSE},  {'>', SHORTorLONG, KEYBOARD_FASTFORWARD, KEYBOARD_FASTFORWARD_LONG},
  {'c', SHORT, KEYBOARD_HOME},                                                                                                        {'i', SHORT,       KEYBOARD_MENU          },
                                                                                    {'u', SHORT, KEYBOARD_UP    },
                                                      {'l', SHORT, KEYBOARD_LEFT},  {'k', SHORT, KEYBOARD_SELECT},     {'r', SHORT, KEYBOARD_RIGHT},
                                                                                    {'d', SHORT, KEYBOARD_DOWN  },
/*{'b', SHORT, }, */                                                                                                                  {'s', SHORT, KEYBOARD_BACK          },
  {'+', SHORT_REPEATED, YAMAHA_VOL_PLUS },                                          {'m', SHORT, YAMAHA_MUTE_TOGGLE},                 {'^', SHORT, SAMSUNG_CHANNEL_UP     },
  {'-', SHORT_REPEATED, YAMAHA_VOL_MINUS},                                       /* {'e', SHORT, }, */                                {'v', SHORT, SAMSUNG_CHANNEL_DOWN   },
  {'1', SHORT, SCENE_TV        },      {'2', SHORT, SCENE_FIRETV   },                              {'3', SHORT, SCENE_CHROMECAST  },  {'4', SHORT, YAMAHA_STANDARD        },
};

Do you like this more? To me this would be fine.
The third value in the list (the long press command) needs only to be there when using SHORTorLONG as repeatMode.

[CoretechR]

I think that depends on where the three maps are in the code. If they placed are in the device file, I don't mind. But the key file would get convoluted really quickly if there are three maps for each device.

[KlausMu]

They would not go into the key file, and not into the device file.
For each scene there will be a separate file. The keymaps will be in the scene files, each keymap in the corresponding scene file, so only one per file.
All scene files are in a new folder "scenes".
And the start and stop sequences would go from the commandHandler to the scene files.
So key file and commandHandler will become smaller.

[CoretechR]

My Harmony has no display, so I'm used to switch scenes with keys. This is done with the three keys at the top.

OK, this explains why it makes sense for you to put the scene switching on dedicated buttons.

Is this correct for your setup:
Screen "Numpad" is for your TV, using keypad (hardware keys)
Screen "Apple TV" for Apple TV, also using keypad, but with a different setting for the hardware keys
Screen "SmartHome" is special, not using the keypad. Hardware keys are doing nothing at all.
Screen "Settings" is also special, not using keypad. Hardware keys are doing nothing at all.

Yes, exactly. This was meant to be similar to a LCD-Harmony remote like the Elite. The screen has two extra capacitive button to invoke an "activities" and a "devices" menu. After selecting a device, the remote switches to a Numpad or channel list which fits the device type. The only thing I wanted to do differently was to get rid of the selection menu to make the programming easier and the UI clearer. So it is just this:

Now having more than one screen per device either means adding more screens to this side-scrolling menu or we need to another level in form of a menu or dedicated device switching buttons. In any case, it adds complexity. If there are scenes with start and end actions (like switching inputs) the simple menu does not work anymore because you would have to scroll through and unintentionally initialize devices that you do not want to use. So I think with scenes, an extra menu is probably unavoidable.

How do you access this menu? Logitech's approach of having a dedicated button seems good. We could do the same with a UI button. Gestures like swiping from the bottom of the screen could work, too. But it is probably quicker to just press a button, even if that means losing screen real estate. The menu button could be the breadcrumb that is already on the bottom of the screen. It is not really needed since the sideways swiping does not change devices anymore. So it could become both the indicator for the current device/scene as well as access to the menu. Having two separate UI elements for the breadcrumb and the menu could be more intuitive. At least that is the way the YIO Remote 2 works (They have a physical home button below the screen). If that is the way to go, I would prefer the menu as in your first mockup and hide the settings etc in this menu as well. So the regular UI is just for remote control actions.

[KlausMu]

If there are scenes with start and end actions (like switching inputs) the simple menu does not work anymore because you would have to scroll through and unintentionally initialize devices that you do not want to use. So I think with scenes, an extra menu is probably unavoidable.

Exactly. I think we need a dedicated start screen for selecting the scene. Depending on the scene, we have different screens to the right. That seems to be the best way to do it.

So for each scene you could

• define a keymap for the hardware keys
• have one or more additional screens to the right. These are for controlling the involved devices of that scene. Not necessarily one screen for each involved device. You could do whatever you want on a screen. You couls also have only one single screen for each scene (as in your case).
• you could reuse screens in more than one scene. In the callbacks of the screens the code knows in which scene we currently are in and reacts accordingly

With that approach you can achieve all three of my mockups. It would be configurable which scenes you want to have and which screens you want to use in them.

Scene indicator

• I think we should remove the page indicator in the new concept. It's not needed anymore. I would display the name of the scene at the top of the screen, between the WiFi and battery indicator. This is what I already did in the newest commit of my branch (which still does not run for you)
• The breadcrump could at the bottom or the top. Need to think about where it fits best. But this can be changed later easily if we don't like it.

Accessing the first page
Could be done

• with a hardware key (could be configured which one)
• with a GUI element like a home button. Do be defined where this should be placed
• by hitting the "scene indicator" or the breadcrump at the top or the buttom of the screen
• with up or down swiping. I would have to dig deeper into lvgl to know how to do this

For the beginning I would use a hardware key.

Accessing dedicated device pages

• maybe a similar approach as for the scenes could be helpful. First select the device, than you have a screen with all the commands
• could be placed to the left of the scene selection page.
• the available devices could be only the ones used in the scene

To be honest, I never used the device pages in Harmony. I always only needed the most common commands on my remote.
So for now I would focus on the scenes, not on the device pages providing every single available command. What do you think?

Is this concept ok for you? Then I would start on implementing this so that we can see if we are happy with that.

BTW, this concepts means that we have to dynamically create the screens (or tabs in terms of lvgl). But this is needed anyway, because as I saw we are still strugglin with memory. I'm already working on this. The best way to save memory seems to dynamically create the tabs and to delete them as soon as they cannot been seen anymore.

Moving bufA and bufB to the heap allowed me to have 100+ screens. But if BLE and WiFi are actived, it is much less, only 6-7.
In fact we need only three screens to be in memory at the same time: the active one, and one to the left and the right, because they already can be seen during swiping.

So there are still some tricks left we could do. I will start a dedicated discussion about this topic in the near future.

[CoretechR]

Scene indicator

Yes, I have seen where you put it and I think that is alright. I feel a little sorry for my old page indicator, but that will not fit with this approach.

Accessing the first page

The key layout is not really made to have a dedicated menu button like e.g. the YIO Remote. It would have to be recognizable as a home button of some sort. I would prefer a touchscreen based command like swiping up from the bottom of the screen. I can try to implement his for a mockup but I am sure it can be done with LVGL. I imagine it could look and work something like this:

So you have a menu layer with scenes, devices, settings and so on. And when you enter a scene or device, you can have multiple pages. In both the menu and the devices layer you can switch pages by side scrolling. To get back to the menu you could pull up from the bottom of the screen (the page indicator could also indicate that there is a swipe up gesture similar to iOS). A menu layer would also allow the hardware keys to be used for controlling the UI. Buttons can be easily configured as keyboard inputs in LVGL. So you only have to touch the screen once to enter the menu and then use the arrow keys for everything else.

However we do it, it will be a big change from the way the remote worked with my old code. I am still not completely sure if we should add this extra complexity. Personally I would not need scenes at all but former Logitech users like you might miss them. Then again, the previous version of the YIO remote only had a one-layered UI, too: https://yio-remote.github.io/documentation/
I just keep referencing them because there are not so many universal remote with screens, but they seem to have also switched to a more complex UI with their commercial version.

[KlausMu]

Accessing the first page

The key layout is not really made to have a dedicated menu button like e.g. the YIO Remote. It would have to be recognizable as a home button of some sort. I would prefer a touchscreen based command like swiping up from the bottom of the screen. I can try to implement his for a mockup but I am sure it can be done with LVGL. I imagine it could look and work something like this:

Yes, great, looks good. And if someone wants to define a hardware key as home, this can be done very easily. Just put the according command in the keypad map.

An example on how to swipe up would be helpful for me. The simulator does not yet really work for me. Maybe @MatthewColvin can help me in the other discussion about the abstraction branch.

Something else that could really help me is this thread:
https://forum.lvgl.io/t/delete-a-tab-after-the-tabview-scroll-animation-is-complete/3155/5
To do the creation and deletion of the lvgl objects at runtime (and to save a lot of memory), it is necessary to have an event when the animation ended. I did not get it running.

So you have a menu layer with scenes, devices, settings and so on. And when you enter a scene or device, you can have multiple pages. In both the menu and the devices layer you can switch pages by side scrolling. To get back to the menu you could pull up from the bottom of the screen (the page indicator could also indicate that there is a swipe up gesture similar to iOS). A menu layer would also allow the hardware keys to be used for controlling the UI. Buttons can be easily configured as keyboard inputs in LVGL. So you only have to touch the screen once to enter the menu and then use the arrow keys for everything else.

Great idea.

However we do it, it will be a big change from the way the remote worked with my old code. I am still not completely sure if we should add this extra complexity. Personally I would not need scenes at all but former Logitech users like you might miss them. Then again, the previous version of the YIO remote only had a one-layered UI, too: https://yio-remote.github.io/documentation/ I just keep referencing them because there are not so many universal remote with screens, but they seem to have also switched to a more complex UI with their commercial version.

Yeah, this would be a big change.

To be honest, my last published version is also already fine for me. I think scenes are very helpful so that you can define different keyboard layouts binded to them. This is exactly what I needed. For me it is fine for having only two or three screens, and they can already be context sensitive to the scenes they are in. I'm more used to the keys, so the current state would be totally fine for me.

But others might be much more screen focused, and I think for them the screen concept shown by you would be very helpful. I would love to spend some time on this and to implement it. But of course, someone should see a benefit in it.

Is anyone else reading this?

[JustMe-NL]

Great work! and yes, someone else is reading :)

in my opinion switching scenes should only be done through a keypress, either a physical button or a virtual button on screen because of the start scene and end scene commands that need to be send by the remote. (And the sometimes associated pointing of the remote)

But I do like the swiping, we could use that for a scene selection screen and for individual device control without switching scenes, eg. when a IR command was missed.

[KlausMu]

Thanks for your comment :-) I also think that swiping shouldn't be used for selecting a scene. And as far as I understand the mockup of @CoretechR this is exactly what he proposed. You could swipe almost everythere, but not for selecting a scene.

[KlausMu]

BTW @JustMe-NL do you have the possiblity to test my latest version?
git clone -b modular-approach https://github.com/KlausMu/OMOTE
@CoretechR is having trouble with it, but I cannot reproduce it.

[JustMe-NL]

Would love to test, but so far my Omote consists of a ESP32S3 breakout and a Adafruit 2090 screen on a breadboard as a Proof of Concept to see if I can use this can replace my old Harmony One.

[CoretechR]

@KlausMu The crash on my remote can be narrowed down to this line in sceneRegistry.cpp:
if (registered_scenes[sceneName].this_key_repeatModes->count(keyChar) > 0) {
If I comment that out, it crashes at if (registered_scenes[sceneName].this_key_commands_short->count(keyChar) > 0) {
Maybe it has to do with the function pointers. But why does it work on your hardware then?

[KlausMu]

Ok, thanks for narrowing it down. I think I got it. The problem was that if no scene was active the code was too optimistic ...
Sorry for this. Please try it again.

[CoretechR]

That fixed it, thank you!

[CoretechR]

An example on how to swipe up would be helpful for me. The simulator does not yet really work for me.

I just tried out the LVGL gesture event, which would be a good start for what we want to achieve. You can set it up with this:

lv_obj_add_event_cb(lv_scr_act(), Gesture_event_cb, LV_EVENT_GESTURE, NULL);

Inside the event you can check to make sure it was a swipe up gesture (LV_DIR_TOP)

static void Gesture_event_cb(lv_event_t* e) {
    lv_obj_t* screen = lv_event_get_current_target(e);
    lv_dir_t dir = lv_indev_get_gesture_dir(lv_indev_get_act());
    if (dir == LV_DIR_TOP) {
        // start screen transition to menu
    }    
}

This would allow for a basic gesture detection. If we want a smooth transition, it gets more complicated.

[bittnert]

Dear @KlausMu ,

I really like the changes you did to clean up the code base. My understanding is, that the behavior you describe you image for the remote is very close to my understanding as well.

However, in the current code base, all devices are actually hard coded and I think this is fine for people who feel comfortable changing and re-compiling the software. However, I think it would be better to be able to make this configurable without re-compiling the software.

I think this is possible I already started to document a possible approach. I have the first rough draft written here: https://github.com/bittnert/OMOTE/blob/modular-approach/doc/Omote%20remote%20overview.md

To configure the device I would image (in the long run) a web interface which is stored on the remote control and which can be loaded.

In a first step however, I would change the current approach you have on your branch. Currently there are devices like "yamahaAmp" which seems to be an amplifier from Yamaha which uses the NEC IR Protocol. I would change this so the firmware does not know what a Yamaha AMP is and does not know how to command it (has no idea what the power on command is for example). Instead, the firmware know what the NEC IR Protocol is and how to talk to a device using this protocol. The configuration structure would then contain the information to know how to command a specific device.

I think as a first proof on concept, it should be possible to define such a config structure and then fill it within the C code. The change to know would only be, that such a config structure needs to be filled to define how to communicate with a device, the firmware would still need to be re-compiled to change this. But in the future, a configuration file in Flash could be used which can be uploaded over wifi (e.g. from a web interface) so the firmware does not need to be re-compiled.

I pulled your branch into my repo and will try to make the changes as a proof of concept so you can see what I mean exactly and we can discuss further.

BR
Thomas

[KlausMu]

Thanks @bittnert for pointing me to the document. It is very similar to what I already have in my branch.

• "Device Types" are called "commandHandlers" in my branch (MQTT, several IR protocols, BLE keyboard for Fire TV and others, ...)
• "Devices" are more or less the same in my branch as you described them, but they could use different kind of "commandHandlers" at the same time, if needed. And it absolutely doesn't matter if a hardware key, a screen interaction or whatever wants to control a device. And it doesn't matter if you have a device centric approach or a scene centric approach. In software, you can send at any time the command the device has registered, and everything else automatically happens.
• The "Actions" are more or less the start sequences and end sequences you can define for a scene in my branch. And you can have "special commands" which also could be used for an arbitrary action. So this is also already available.
• The "Scenes" are exactly how I implemented them.

Regarding the configuration structure: this would not only to be read in by the OMOTE on a daily base, but with every wake up, since the ESP32 does not really have a sleep where it continues where it went to sleep. It is always a complete reboot.

I think your idea on using configuration files in code as a first step goes into the right direction. YAML is of course widely known. Home Assistant is using it for example. Can sometimes be a pain, but currently I think it would be the best. As you said, everything would still be in code, but at least in some way as you would do it with files.

Of course, an external software for defining everything on a full blown computer would be the best. But I would like to say that this was really a huge step and would need a lot of time to create it.

The software for it could be e.g. a web page provided by the ESP32 (so that it could be used by any device with a big screen) or a completely platform specific software. But as said, a really huge thing to do. And most likely a big challenge for the ESP32 when it should act as a web server. I guess at least BLE has to be turned off while doing this.

I think the biggest hurdle to come in touch with OMOTE currently is not the software, but the hardware. At least for me it was. I am not so familiar with PCBs and how to order them at JCLPCB and to answer all the questions they have. I was glad that someone had a spare PCB which he sold me. And getting a screen is also an odyssey. And you need to have a really good 3D printer for printing the TPU buttons.

Let me give you an example: while doing my research about the memory issues, I found the project EPSuino . They had or still have more or less the same challenges as the OMOTE has with memory and other stuff. They already switched to a wroom with PSRAM. And lot of other interesting things could be learned from them. So I directly had the wish to have one. Now. Immediately.

They always have some ready made PCBs in stock. Two days ago I asked them for sending me a complete set, and tomorrow it will arive. Everything that is needed. Awesome.

I think having something similiar for the OMOTE pcb and maybe for a silicon keypad (as someone proposed on hackaday.io) would open the OMOTE to so many more people.

And I think providing the hardware is much easier to achieve than having a full blown software for a gui based configuration of the OMOTE. So much time would be needed for creating such a software. We would need several experienced software developers to achieve this. And for getting them into the OMOTE project, hurdles need to be lowered so that a lot of people can easily get the hardware. This is what I think.

@CoretechR what do you think about it?

[bittnert]

Dear @KlausMu,

yes the firmware would need to read the config file on every boot in the current configuration. Although this is not a hardware limitation. The ESP32 has some RTC memory which can be kept powered on while the device is sleeping with very little power impact. Specifically it has 2x 8k of RTC memory which is retained when the device is in deep sleep.

Additionally, I would not use yaml or any other human readable form within the firmware. For now I would use structs where the firmware can directly access the information it needs. When this configuration structure is changed, a binary blob would be uploaded which can be read by the firmware. I think it is unrealistic to let the firmware parse any human readable format.

I will try to adapt the current firmware to show the intention on this at least for IR configurations.

I also think, that additional memory will be needed in the future and also would vote for additional external memory.

I would even vote for 8 MB of RAM and an additional SD card. The ESP32 has SD/MMC interfaces (for up to two cards) and by using a micro SD card, we can change the memory size as needed or only add a memory card if needed. This should not increase the cost of the BOM that much but still provide future extensibility.

I also think having a GUI based configuration system is a lot of work, but I think this should be possible and we should at least prepare the architecture for this. My idea is, that the ESP32 web server would only be active when someone wants to configure the device (we could then also disable BLE and other things as long as the configuration is on going). The main calculations should then be done on the host machine and the ESP32 would only be accessed where needed (e.g. for reading some IR signals or uploading a new configuration).

[CoretechR]

So the question is, should we make the code easier to modify or should we go straight to a config file based customization? Not needing to tweak the code seems intriguing, but it is probably also going to be a lot of work. And I can't say how much I can support you in terms of time and programming skills. This kind of abstract programming is not really my strongest suit, so I would much rather focus on the hardware. I just started planning a new PCB revision, which will hopefully cover a lot of issues and feature requests from the community. When it comes to deciding between the current ESP32-Pico-D4 and newer variants with PSRAM and more flash, I am tending towards the D4. Klaus showed that the memory for LVGL is manageable and I really want to keep the compatibility for those people who already built a remote. With this new version I can also try to make the hardware more easily available, maybe even though something like Tindie.

[KlausMu]

I would first try to significantly raise the number of people using the OMOTE. I believe that with only a few programmers like us it is not possible to create a GUI based, non code approach to customize the OMOTE. At least if noone is willing to work more or less fulltime on it. Currently I can't see this person.

So my proposal would be

• make the software easy to use and easy to customize for people with average or even low programming skills
• implement all the concepts we discussed into the software (scenes, gui supporting the scenes etc.)
• write a good documentation on how to understand and customize the software
• make the hardware very easy to obtain
• let people play around with the OMOTE and find enough people willing to contribute to a GUI based approach
• for now the current hardware is hopefully powerful enough. I'm not 100% sure, but I hope so. If not, we still have the option with the bluetooth/IR satellite which would be cheap and easy to setup. For makers absolutely acceptable, and makers are the target group in this first phase. If we used something like the satellite, we instantly would have plenty of ressources in the OMOTE.

I would be willing to contribute to the software as described above and to the documentation. I think, almost 50% of it is already available in my branch (ok, estimations of developers are always too optmiistic).

For starting a GUI based configuration tool now, unfortunately I have to say that I wouldn't be part of the team, because for me this would be much to time consuming.

For the GUI based configuration my guess is that we would need more powerful hardware, mainly RAM. But this could be done later, if really needed and if the community got bigger. If this ever happens, I'm pretty optimistic that it is possible to find some kind of compatibilty to the old hardware. Could be for example that users of the old hardware always have to use the satellite. Or they have to stay on the software version that has been reached until then (which will be fully functional). Or they have to live with a simplified version of the new software. There would be several options.

[phant0mias]

You are discussing if OMOTE is getting an easy to use "product" or a DIY project.

For me the current software is good to understand the usage of the hardware and adapt it for me. I learned a lot about the ESP (I normally use RP2040), lvgl (Thanks @CoretechR to your extra article in the make magazine) and sleep mode. I would keep the current implementation, it is straight forward and understandable.

Now I have a portable user interface with display, buttons and all kind of communication. I was searching a long time for something like this. I use it e.g. currently as a user interface for developing a plant growing project.....

[WizardCM]

A lot of interesting discussions to catch up on, and lots of topics covered.

I think the branch is in a very good place currently, and would be worth submitting as a PR soon to gather more in-depth feedback to the code.

My PCB arrived 2 days ago, and since then I've been tinkering with Klaus's branch as I'm only missing the case & buttons. For reference, I am a programmer with a mid-level experience in C/C++ and zero prior experience with the ESP32 platform or Platformio itself. Even then, I was able to jump in without a lot of hassle.

Without reading any of the above discussion before tinkering:

1. The concepts as they are: near-perfect. The default scenes were clear in their intent, and it's really easy to tweak things for yourself once you've skimmed the code. The folder structure helps significantly in this area, even without code comments.
2. Adding extra button definitions to existing devices was pretty straightforward, but I got quite lost attempting to integrate a device myself until Klaus pushed the initial Denon implementation. The biggest part I was confused by was the separate definitions for the #define (a string) vs the registration of commands. It's unclear what the string should be, even with ample existing examples.
3. It's also a bit confusing for a first-timer that all the different command types are all routed the same way, even if some are to control devices
4. The secrets definition is a good start, and allowed me to get set up very quickly.
5. Building UI is difficult, even for someone with experience. This is a challenge worth tackling at some point, but not as part of this initial implementation. I was able to use a mix of SquareLine Studio and the existing gui_numpad.cpp to get a functional screen I'm happy with.

My thoughts summarised: perfect is the enemy of done. The branch as-is makes it much easier to get started tinkering. It gives users (developers) a much better grasp of how to start writing new screens, device integrations, and scenes than the current repository. It's a very good step in the right direction. There's SO MUCH more to improve, but we don't need to do it all in one go!

Trust me: merge what's in the branch now, and you'll get much more activity with people wanting to add their own integrations based on existing examples.

---

Now, in terms of topics discussed:

• The keyCommands mappings are alright to use, but the keys are a bit unwieldy. I got lost writing mappings for a device, scene, and the default scene - this would only get more difficult the more configurations I have. I'd personally vote for having some global defines for the physical buttons, something like

#define BTN_PWR 'o'
#define BTN_STOP '='

std::map<char, std::string> keyCommands_short {
      {BTN_PWR, ALLDEVICES_POWER_TOGGLE}
};

• I personally have no preference about what the default method for scene switching should be, I think a home page is probably the most user-friendly. But I will personally go with a single button that toggles between 2 scenes, and I'll likely almost exclusively use 1 scene.

• In terms of UI, I think Reworked software - modular approach #58 (comment) is a step in the right direction, I like the 3 dots but I think really large headings should be avoided. Alongside the scene name in the header, having colour coding would be good too - icons and colours are much easier to process at a glance than text. The top status bar could be coloured, as that is closer to a user's eyeline when they're looking at their devices.

• In terms of configurations, I agree having some well-defined machine-readable config files would be best, and separating the relevant "protocols" out (as explained in Reworked software - modular approach #58 (comment)) compared to the devices. Once a good system is devised, having a full folder of ready-to-use-but-disabled configurations for various devices and systems would make getting started significantly easier.

• And finally, I think the discussion around the difficulties of acquiring new users (those of us who had a PCB shipped from another user are incredibly lucky) is important to have, but out of scope for this discussion and just muddies the water.

---

In terms of open points from my perspective, I only had one real question @KlausMu - is switching scenes the only way to run a batch of commands? For example, I'd want one of the physical buttons to

• send an MQTT call (to turn on my PC)
• switch my TV to the HDMI input
• switch my Denon AVR to the PC input

Currently this doesn't seem to be possible? Again, generally I will stick to one scene to lower UX complexity.

Awesome work so far though! Thanks to all the hard work of everyone involved!

[KlausMu]

2. Adding extra button definitions to existing devices was pretty straightforward, but I got quite lost attempting to integrate a device myself until Klaus pushed the initial Denon implementation. The biggest part I was confused by was the separate definitions for the #define (a string) vs the registration of commands. It's unclear what the string should be, even with ample existing examples.

Ok, thanks for your feedback, @WizardCM. I will cover this in the documentation that I will write soon.
Regarding

#define SAMSUNG_POWER_TOGGLE       "Samsung_power_toggle"
commands[SAMSUNG_POWER_TOGGLE]      = makeCommandData(IR_SAMSUNG, {"0xE0E040BF"});

Every command needs to have a unique key, In the example it is Samsung_power_toggle. The key itself has absolutely no meaning. It didn't have to be a #define, also could have been a variable. But I like the defines.

3. It's also a bit confusing for a first-timer that all the different command types are all routed the same way, even if some are to control devices

There are three types of commands:

• almost all commands are to control devices
• very few are to control scenes. Scene commands are received by the commandHandler, but immediately forwarded to the sceneHandler.
• SPECIAL commands, where you can do in the commandHandler whatever you want, e.g. having a sequence of other commands or whatever. Currently only available as example. I do not use them

To me this makes sense. With that you have a single command registry. If handling of a command is complicated (as with scenes), the command will simply be forwarded to another handler,.

5. Building UI is difficult, even for someone with experience. This is a challenge worth tackling at some point, but not as part of this initial implementation. I was able to use a mix of SquareLine Studio and the existing gui_numpad.cpp to get a functional screen I'm happy with.

Plan is to have a simulator for Windows/Linux, so that you can create the UI without using the ESP32. But as you said, not in the first version.

Now, in terms of topics discussed:

• The keyCommands mappings are alright to use, but the keys are a bit unwieldy. I got lost writing mappings for a device, scene, and the default scene - this would only get more difficult the more configurations I have. I'd personally vote for having some global defines for the physical buttons, something like

#define BTN_PWR 'o'
#define BTN_STOP '='

std::map<char, std::string> keyCommands_short {
      {BTN_PWR, ALLDEVICES_POWER_TOGGLE}
};

Yes, good idea. I will do the change. BTW here we have exactly the same as with the commands. A #define for having a unique key, then you can use the #define in a more readable way.

In terms of open points from my perspective, I only had one real question @KlausMu - is switching scenes the only way to run a batch of commands? For example, I'd want one of the physical buttons to

I think, in the meantime you found it. There is an example command named "MY_SPECIAL_COMMAND" in the code.

Thanks a lot for your detailed feedback!

[CoretechR]

Trust me: merge what's in the branch now, and you'll get much more activity with people wanting to add their own integrations based on existing examples.

I agree with @WizardCM 😄. The modular code is a better starting point for people new to this project and not everything has to be implemented and polished from the start.

@KlausMu If that's OK for you I would love to merge your modular code and make this the default branch. Then we can continue from there.

[KlausMu]

Sure, I think it is already good enough. I will create a PR on Monday. There are already some topics on my todo list, and I would like them to be published soon, in that order - [x] documentation on how to understand, to use and to modify the code. Is there anything special you want me to explain? I will write a page than can be published in the github wiki - [x] memory optimization of lvgl. This is indeed a major task, and e.g. for the sliding event I still need a solution. But it is important, because with wifi and ble enabled you will quickly get memory issues - [ ] simulator for creating pages in windows (and Linux?) - [ ] scene selector page as start page - [ ] available gui pages based on the currently active scene. Hide pages not needed in a scene - [ ] make gui actions context sensitive for the currently active scene

[phant0mias]

Hello,
I just tested the new software. It is really a good base for starting with omote. Thanks for the great work you did.

[KlausMu]

I now created a documentation for the new main branch. Please have a look here.
https://github.com/KlausMu/OMOTE/wiki/How-to-understand-and-modify-the-software

If you have any feedback, please let me know.

[CoretechR]

If that's OK for you, I can just copy/paste the text. The workaround with Github actions seems overly complicated for this.

[KlausMu]

Sure. From now on, I will keep the Wiki up to date with the progress of the software. Currently I am working on the memory optimization.

[CoretechR]

Great, and I will the OMOTE Wiki up to date as well. Again, thank you for putting so much time and effort into this project!

[KlausMu]

Ok, the "Architectural overview" seems to be complicated. But in fact it is not necessary to understand and use the software.

I will change the order of the sections. First "Getting started", "Basic concepts" and "How to". This is everything you need to adjust the software to your needs.

And as last section, only for those who are interested, the "Architectural overview" to understand how everything works behind the scenes.

[WizardCM]

It's pretty common for open source projects to mark their Wiki as editable by anyone (by unchecking the "Restrict") checkbox. The Wiki is its own GitHub repo https://github.com/CoretechR/OMOTE.wiki.git so all changes are tracked and can be reverted if necessary (either via the GitHub UI or via Git directly).

This way you wouldn't have to keep the wiki in sync manually.

[KlausMu]

Hi all, I pushed a new version of my branch.
If not yet done, you can get it with

git clone -b modular-approach https://github.com/KlausMu/OMOTE

This new version implements the memory optimization described here

This was really a big change in the software, and also a complex one. To me this was the most important missing feature. Otherwise, we would constantly have had memory issues.

I also created a small memory status text at top of the screen which can be activated or deactivated at compile time, showing both ESP heap and lvgl memory. The numbers turn red in case of resources running out.

Now only three tabs stay in memory at the same time. As long as these three tabs do not become too big, everything is fine. With that you can have 100 tabs and more without any problem.

Swipe through the tabs and watch the memory status. It is very interesting to see how lvgl memory usage is going up and down and what happens if you use WiFi.

Some tricks were necessary so that you cannot recognize that the whole content of the screen is recreated after each swipe. I think the result is very good. Please have a look.

It would be very helpful if you could test it out. And please also read the new section in the Wiki.

I reduced the default memory for lvgl from 48k to 32k, because now this seems to be enough. I think with 48k you might get trouble when using WiFi. @WizardCM maybe this is what you encountered.

I also introduced a #define to enable easy switching between one or two lvgl buffers. Default are two buffers, as in the past. With only one buffer you save 15360 bytes RAM in heap. I can't see any difference when using only one buffer. If you want to have this additional 15360 bytes, comment out #define useTwoBuffersForlvgl in file guiBase.h

Please see this description if 32k does not fit to your needs.

lv_conf.h is no longer used, everything is in platformio.ini now.

Unfortunately you need to change your GUI files, if you already have created some. You have to

• put the name of the gui in the .h file
• rename function init_gui_tab_<name> to create_tab_content_<name>
• very important: delete line lv_obj_t* tab = lv_tabview_add_tab(tabview, <name>; in function init_gui_tab_<name>
• completely delete function init_gui_pageIndicator_<name>
• create new function void notify_tab_before_delete_<name>(void) {
• change line register_gui(& init_gui_tab_<name>, & init_gui_pageIndicator_<name>); to register_gui(std::string(tabName_<name>), & create_tab_content_<name>, & notify_tab_before_delete_<name>);
• if you want to preserve the state of a GUI before it gets deleted (e.g. slider position etc.), use function notify_tab_before_delete_<name>. See gui_smarthome.cpp for an example

[KlausMu]

Memory optimization is now in main branch. You can directly use https://github.com/CoretechR/OMOTE

[KlausMu]

Hi all,
A new version with a significant improvement was released here in this repository from @CoretechR in a feature branch called "lvgl-simulator" https://github.com/CoretechR/OMOTE/tree/lvgl-simulator
If you never cloned the repository before:

git clone -b lvgl-simulator https://github.com/CoretechR/OMOTE

or, if you already have cloned the repository:

git checkout lvgl-simulator

Note: New feature versions will no longer be published in my repository, but directly here in a feature branch.

The main new feature is, as the title of the branch suggests, the simulator directly running on Windows or Linux:

With the simulator, you can run the GUI code directly on Windows or Linux and don't have to flash the firmware to the ESP32 each time you do an iteration while developing the LVGL gui. This can dramatically reduce the time needed for adjusting the software to your needs.

The simulator uses exactly the same GUI code that is used for compiling the ESP32 firmware. So when you are finished with your GUI code, simply compile the code for ESP32 and flash it.

For a comprehensive documentation on how to setup the simulator, please read the wiki Software simulator for fast creating and testing of LVGL GUIs

Memory usage was again optimized and another 25k were saved. Thanks @MatthewColvin for your valuable hints.

There have been a lot of other changes in the code, especially the folder structure has significantly changed. Code is now divided into

• parts you normally adjust to your needs (devices, scenes, guis) and
• application internal parts you normally should not need to change.

Wiki documentaton How to understand and modify the software has been updated as well.

My advise is to not try a merge of the new version, but to start with a fresh installation. If you already changed the code, then

• save your code at a secure place
• do a new clone of the repository
• bring back your customized code into the new version (should be easy to do)

So if you are curious about the new feature branch, feel free to test it. Feedback is very welcome.

After about one week, the feature branch will be brought into main branch.

[KlausMu]

#69
What happens with the PR if I do additional commits to that feature branch?

[WizardCM]

It's kept in sync, any changes to the branch (including force-pushes even) are displayed in realtime.

[KlausMu]

So additional commits after the PR was created will be added to the PR?

[WizardCM]

Yep!

[KlausMu]

Great, thanks

[KlausMu]

Sending MQTT messages is now supported in the simulator, both in Windows and Linux/WSL2.

An example on how to subscribe to MQTT messages and how to propagate them to the GUI is also included.

And it is shown how to send an update request to your home automation software as soon as OMOTE has WiFi connected. With that, your home automation software can send the states of the smart home devices known to OMOTE.

https://github.com/CoretechR/OMOTE/wiki/02-How-to-understand-and-modify-the-software#subscribe-to-mqtt-topics

@WizardCM I think MQTT subscription is something you wanted to use.

Everything is still in feature branch "lvgl-simulator" https://github.com/CoretechR/OMOTE/tree/lvgl-simulator

[KlausMu]

A new feature branch called "scene-selection" is available for test.
If you never cloned the repository before:

git clone -b scene-selection https://github.com/CoretechR/OMOTE

or, if you already have cloned the repository:

git fetch -a
git checkout scene-selection

Feautures in this branch:

• "scene selection gui" as start page
• a so called "main_gui_list" for all guis that are not dependent from a scene (e.g. settings, IR receiver, ...)
• a "scene specific gui list" for guis that are relevant for a scene
• switching back and forth between "main_gui_list" and "scene specific gui list"
• breadcrump in page indicator to see where you are in the list of guis

You can deactivate the "scene selection gui" and scene specific gui lists if you want to keep the old behaviour (only one single list of guis, available in all of the scenes).

Please see the updated Wiki:
https://github.com/CoretechR/OMOTE/wiki/02-How-to-understand-and-modify-the-software#scene-specific-guis
https://github.com/CoretechR/OMOTE/wiki/02-How-to-understand-and-modify-the-software#register-new-gui
https://github.com/CoretechR/OMOTE/wiki/02-How-to-understand-and-modify-the-software#register-a-scene

This time, there are not so many code changes. I also created a PR if you want to comment:
#72

[CoretechR]

Wow, this is awesome! I really like how this adds a second level to the UI for things like settings and the IR receiver. That makes a lot of sense compared to having these all on one level to swipe though. The gesture for bringing up the menu also works great, even in both directions. Fantastic!
Maybe the UI could use some kind of hint on how to open the menu. Generally there are some details about the UI that I have ideas on. I hope I can contribute to that in the future.

[KlausMu]

I also like the second level very much. It improves usabilty a lot.

The gesture is only active for "swipe down"

OMOTE/Platformio/src/applicationInternal/gui/guiBase.cpp

Line 46 in 47c10aa

if (dir == LV_DIR_BOTTOM) {

But you can also get back to the scene selection page by

1. pressing the scene label
2. pressing the page indicator
3. hitting the "back" hardware key

If "swipe up" worked for you, probably you hit the page indicator. But of course we could add "swipe up" as well.

Swipe up and down do not work well if the content is scrollable. That's why I added the clicked event handlers to the scene label and page indicator.

I also thought about having a hint on the UI, but I didn't have a good idea. But at least it is explained in detail in the Wiki.

[KlausMu]

Branch "scene-selection" is now in main.