[Feature Request]: Updated Device Screen UI modes for all devices

[tropho23]

Platform

NRF52, ESP32

Description

Updated Device Screen UI modes

Overview of this proposal:

The current Meshtastic device screen user interface (UI) needs to be updated to display additional node and sensor information, enable enhanced user interaction without relying on BLE-paired mobile devices, and support advanced features introduced by new standalone devices integrating keyboards, rotary encoders, and matrix keypads. Device capabilities and new user interactions have grown beyond the capabilities of the current device screen UI.

Ideally, the current screen UI library should be able to support these proposed navigation flows and nested menus/actions for the new 128x64 screen UI modes; if not a new screen UI library must be considered. If currently supported e-ink displays can also take advantage of this proposal, that compatibility should also be accommodated if feasible.

For the Advanced Screen UI mode, a new screen UI/display library will need to be integrated to not only take advantage of TFT LCD and controllers, but also offer the graphical performance to support higher screen resolutions, interactive map displays, and possibly even touch interface, if feasible. The goal of the Advanced Screen UI mode is to minimize, or even eliminate the need to use BLE-paired mobile devices for any non-configuration use.

Selecting these modes will require a CLI/app/web configuration setting to choose Device Screen UI mode.

List of proposed Device Screen UI modes:

• 1. Boot/splash screen update: (128x64+) See note below
• 2. Unset region screen update: (128x64+) See note below
• 3. Standard Screen UI mode: (128x64) Improved single-screen information display for devices without user buttons
• 4. Carousel Info mode: (128x64) Continually cycle between 'default info' and 'additional info' screens; useful for testing/setup
• 5. Enhanced Screen UI mode: (128x64) Standard mode plus new menu navigation flows and functionality; includes additional menu navigation for sending canned messages
• 6. Standalone Screen UI mode: (128x64) For standalone devices with CardKB, rotary encoder, or matrix keypad input methods; supports existing features (DMs and canned messages) and planned features (group messages, access previously received message store)
• 7. Advanced Screen UI mode: (320x240+) Eventually integrate TFT screens; support all features of UI modes 3-6 plus maps, waypoints, etc. and perhaps even touch-enabled input; likely requires different screen/UI library
• 8. Basic Device Settings for Standalone Screen UI devices and Advanced Screen UI

1. Boot Splash Screen: Shown during boot

1. Add OEM/user configurable splash screen, either in place of Meshtastic default or show it after

2. Unset Region Screen: Shown when devices have region UNSET

1. No changes recommended

3. Standard Screen UI: For devices without a user button or environmental sensors

1. User can toggle displaying received messages on device screen:
   • If enabled: display last received message on device screen for XXX seconds (configurable; 0=always)
   • If disabled: display default screen information
2. Default screen shows local device information:
   • Device short name + last 4 characters of BLE MAC
   • Connected nodes (#/#)
   • GPS status (enabled/disabled, # of satellites)
   • Battery charge/percentage bar

4. Carousel Info Mode: For devices without a user button; continually rotates through device information

1. User can toggle displaying received messages on device screen:
   • If enabled: display last received message on device screen for XXX seconds (configurable; 0=always)
   • If disabled: display default screen information
2. Default screen shows local device info:
   • Device short name + last 4 characters of BLE MAC (‘NODE 3f58’)
   • Connected nodes (#/#)
   • GPS status (enabled/disabled, # of sats)
   • Battery charge and percentage bar
3. Additional info for the local device:
   • Environmental sensor data (if none, detect this and display "No sensors connected")
   • GPS-specific data (lat/long, dead reckoning direction arrow, # of satellites)
   • Battery state (charging/discharging) and voltage
   • Device role (client, router, relay, etc.) and firmware version
4. Carousel mode continually cycles through items 1-3 above

5. Enhanced Screen UI: For devices with a user button

NOTE: Long press duration should be 2 seconds; 5+ second long press already initiates shutdown for nrf52 devices

1. User can toggle displaying received messages on device screen:
   • If enabled: display last received message on device screen for XXX seconds (configurable; 0=always)
   • If disabled: display default screen information
2. Default screen shows local device info:
   • Device short name + last 4 characters of BLE MAC (‘NODE 3f58’)
   • Connected nodes (#/#)
   • GPS status (enabled/disabled, # of sats)
   • Battery charge and percentage bar
3. Each single user button press cycles additional info for the local device:
   • Environmental sensor data (if none, detect this and display "No sensors connected")
   • GPS-specific data (lat/long, dead reckoning direction arrow, # of satellites)
   • Battery state (charging/discharging) and voltage
   • Device role (client, router, relay, etc.) and firmware version
   • 10 second timeout if no user input; returns to default screen (ideally configurable; 0=no timeout)
4. Double user button press displays a list of connected mesh nodes:
   • Single user button press cycles through list of nodes
   • Double user button press selects desired remote node
     • Action 1: Send canned message
       • Press user button to select 'Send Canned Message'
       • Press user button again to cycle through list of canned messages
       • Double press user button to send canned message
       • <Once canned message is sent, automatically return to node list>
       • Long (2 sec) user button press * cancels/exits to node list
     • Action 2: Send waypoint
       • Press user button to select ‘Send Waypoint’
       • <Once waypoint is sent, automatically return to node list>
       • Long (2 sec) user button press cancels/exits to node list
     • Action 3: Request position update
       • Press user button to select 'Request position update'
       • <Once position update request is sent, automatically return to node list>
       • Long (2 sec) user button press cancels/exits to node list
     • Action 4: Display remote info
       • Press user button to select 'Display remote info'
       • Show default screen information described above (2a-2d) for selected remote node
       • Press user button again to display additional information described above (3a-3d) for selected remote node
       • Long (2 sec) user button press * cancels/exits to node list
     • Action 5: Admin actions
       • Press user button to select 'Admin actions'
       • Press user button again to cycle through list of admin actions (reboot, shutdown)
       • Double press user button to execute desired admin action
       • Long (2 sec) user button press exits to default local device screen at any time
       • 10 second timeout if no user input; returns to default screen
5. Triple user button press disables GPS (if supported by device)

6. Standalone Screen UI: For devices with CardKB, rotary encoder, or matrix keypad

NOTE 1: For matrix keypads assign keys as desired for user button, CW, CCW, and preconfigured canned messages
NOTE 2: Long press duration should be 2 seconds; 5+ second long press already initiates shutdown for nrf52 devices

1. User can toggle displaying received messages on device screen:
   • If enabled: display last received message on device screen for XXX seconds (configurable; 0=always)
   • If disabled: display default screen information
2. Default screen shows local device info:
   • Device short name + last 4 characters of BLE MAC (‘NODE 3f58’)
   • Connected nodes (#/#)
   • GPS status (enabled/disabled, # of sats)
   • Battery charge and percentage bar
3. Each single user button press cycles additional info for the local device:
   • Environmental sensor data (if none, detect this and display "No sensors connected")
   • GPS-specific data (lat/long, dead reckoning direction arrow, # of sats)
   • Battery state (charging/discharging) and voltage
   • Device role (client, router, relay, etc.) and firmware version
   • 10 second timeout if no user input; returns to default screen (ideally configurable; 0=no timeout)
4. Double user button/rotary press (or CardKB Tab key):
   • Show 'Received Messages'
     • Press Enter or user button/rotary press to select 'Received Messages'
     • Subsequent rows show received messages, in descending chronological order (newest first)
       • Each row shows short name, date/time stamp, first few characters of message
       • Use Up/Down arrow keys (CardKB) or CCW/CW (rotary encoder/matrix keypad) to choose the desired message
       • Press Enter or user button/rotary press to display selected message
       • Esc (CardKB) or long (2 sec) user button/rotary press to exit to Received Messages list
       • Esc (CardKB) or long (2 sec) user button/rotary press to exit to default screen
   • Show Node List
     • Navigating the node list:
       • Use Up/Down arrow keys (CardKB) or CCW/CW (rotary encoder/matrix keypad) to cycle among remote nodes
       • Press Enter or user button/rotary press to select a remote node:
         • Action 1: Send a direct message (DM) (CardKB only)
           • Press Enter key to select 'Send DM'
           • Type message
           • Press Enter to send message
           • <Once DM is sent, automatically return to node list>
           • Esc (CardKB) or long (2 sec) user button/rotary press exits to node list
         • Action 2: Send a canned message
           • Press Enter key (CardKB) or user button/rotary press to select 'Send Canned Message'
           • Use Up/Down arrow keys (CardKB) or CCW/CW (rotary encoder/matrix keypad) to choose the desired canned message
           • Press Enter or user button/rotary press to send canned message
           • <Once canned message is sent, automatically return to node list>
           • Esc (CardKB) or long (2 sec) user button/rotary press exits to node list
         • Action 3: Request position update
           • Press Enter key (CardKB) or user button/rotary press to select Request position update'
           • <Once position update request is sent, automatically return to node list>
           • Esc (CardKB) or long (2 sec) user button/rotary press exits to node list
         • Action 4: Display remote node info
           • Press Enter key (CardKB) or user button/rotary press to select 'Display Info'
           • Screen will show default screen information described above (2a-2d) for selected remote node
           • Enter key or user button/rotary press displays additional information described above (3a-3d) for selected remote node
           • Esc (CardKB) or long (2 sec) user button/rotary press exits to node list
         • Action 5: Admin actions
           • Press Enter key (CardKB) or user button/rotary press to select ‘Admin actions’
           • Use Up/Down arrow keys (CardKB) or CCW/CW (rotary encoder/matrix keypad) to choose the desired admin action (reboot, shutdown)
           • Esc (CardKB) or long (2 sec) user button/rotary press exits to node list
         • To exit node list at any time:
           • Esc (CardKB) or long (2 sec) user button/rotary press
           • 10 second timeout if no user input; exits node list and returns to default screen
     • 10 second timeout if no user input; returns to default screen
5. Quick method to send broadcast messages from main/default screen (CardKB only):
   • Type message
   • Press Enter to send message
6. Quick method to send DMs from main/default screen (CardKB only):
   • Press Tab to show node list
   • Press Enter to select desired node
   • Type message
   • Press Enter to send DM
7. Quick method to send canned messages to a selected node from main/default screen (CardKB only):
   • Press Tab to show node list
   • Press Up/Down arrow key to highlight desired node
   • Press Enter to select desired node
   • Press Up/Down arrow key to highlight desired canned message
   • Press Enter to send canned message
8. Quick method to send canned messages to Broadcast (all nodes) from main/default screen (CardKB only):
   • Press Up/Down arrow key to highlight desired canned message
   • Press Enter to send canned message
9. Triple user button press disables GPS (if supported by device)

7. Advanced Screen UI: For devices with TFT LCD screen (TBD)

8. Basic Device Settings: For Standalone Screen UI and Advanced Screen UI only; show essential settings user can change without requiring a paired phone

1. Name: set/change long name and short name
2. BLE: enable/disable, set/change pairing PIN
3. Notifications: enable/disable external notifications (LED, buzzer, vibramotor; each independently toggled)
4. GPS: enable/disable (same as triple user button press), toggle GPS smart position updates, change GPS update interval

[mverch67]

Great proposal!

Another nice option for the ultimate meshtastic device experience:

9. Enhanced Device Settings: For Advanced UI with touch screen only; full device configuration without ever requiring a paired phone or web/cli.

BTW: I started working on (7) driver configuration (lovyangGFX library), I have the following color TFT screens available for configuration and tests:

• Heltec-tracker with 160x80 0.96" TFT (ST7735), ESP32-S3 + LoRa SX1262
• soon LILYGO T-Deck with 320x240 2.8" TFT (ST7789), GT911 touch controller, ESP32-S3 + LoRa SX1262
• LILYGO T-PicoC3 with 135x240" 1.14" TFT (ST7789V), RP2040
• WT32-SC01 Plus with 480x320 3.5" TFT (ST7796), FT6336U touch controller, ESP32-S3
• PicoW with Waveshare ResTouch 480x320 3.5" IPS (ILI9488), XPT2046 touch controller, RP2040
• PicoW with Waveshare LCD+Joystick hat 135x240 1.14" TFT (ST7789VW), RP2040

[tropho23]

Great suggestion! I will add that to my list of proposed device screen UI modes, it's great.

Also thank you for the LCD/TFT development, it's super exciting to imagine what we can do with that.

[GUVWAF]

I would also like to propose having a way to show whether a sent message was acknowledged. I don't think it's worth to store transmitted messages as well; a small pop-up when the last message was acknowledged should be enough.

[geeksville]

This issue has been mentioned on Meshtastic. There might be relevant details there:

https://meshtastic.discourse.group/t/canned-messaging-see-list-of-recent-messages/8060/2

[Nikguy321]

Yeah I think being able to see a small list of like the last 30-40 text messages sent (instead of just the most recent) would be really nice. Would make it a lot more viable to use canned messaging instead of having to have a phone attached.

[YAMLcase]

I'm glad this issue was proposed. I've been tinkering with the T-deck and thought about how we might replicate the android meshtastic client onto the device

[geeksville]

This issue has been mentioned on Meshtastic. There might be relevant details there:

https://meshtastic.discourse.group/t/t-deck-from-lilygo/7679/74

[TheCommsChannel]

I would love to see this as I have a T-Deck as well.

[ko4whd]

Best part about the T-Deck is the TF Card slot. I know this could play a huge role in future updates and options either for additional storage or a full custom OS (ex... ESP32Berry or Raspberry Pi OS Lite ...etc)

[ko4whd]

@tropho23 There is a guy on Youtube (@ThatProject) stated he would be able to help the Meshtastic Team in the development of the Meshtastic app on the T-Deck with FreeRTOS if any team members could reach out to him and let him know what he needs to do to assist.

[Nikguy321]

There is a guy on Youtube (@ThatProject) stated he would be able to help the Meshtastic Team in the development of the Meshtastic app on the T-Deck with FreeRTOS if any team members could reach out to him and let him know what he needs to do to assist.

Is he the guy that does espberry? That's a nice software.

[ko4whd]

@Nikguy321 that's the guy

[danumbaman]

I would love to see a way to load openstreet maps saved to Sd-card on T-Deck.

[thatSFguy]

This is a very well written epic, since it is way larger than a feature. I'm not sure how the meshtastic developers decide what they work on, however, I think it would be more likely taken up if this is split up into individual features. I'm mostly interested in #6 at this point, but I know having the complete vision helps a developer to avoid rework. Again, a very well written request. 👏

[KyleMaas]

I'd love to be able to help with this effort, but not sure where to start. I keep seeing references to the new UI branch, which doesn't look like it's active, and references to 3.0, but again can't find anything current on that. I don't want to go about taking on such a massive architectural change without substantial guidance, and ideally not duplicating someone else's efforts. So what's the best way for me to start helping with this?