Protobuf Config #263

mthiesen · 2023-05-19T11:07:02Z

This is the promised Protobuf refactoring. This PR contains A LOT of changes:

Configuration is now stored with Protobuf serialization using the Nanopb library. This will allow us to change our configuration between versions (within limits) without invalidating the configs stored by users. The two Protobuf source files are proto/config.proto and proto/enums.proto.
Protobuf compilation is automated with CMake. Compilation is only invoked when the .proto file actually changed. The only requirement is a Python3 installation.
Config conversion from / to JSON. This is implemented using the reflection information generated by Nanopb, there is no need for manual adjustment when the Protobuf schema changes. This does not support all Protobuf features right now, but covers what we need currently and more. Can be extended as needed.
api/getConfig and api/setConfig endpoints are implemented in Webconfig to read / write the whole config at once. They use the JSON functionality and will also adjust to schema changes. api/setConfig will correctly return 400 if it gets passed invalid JSON and 500 if saving fails for whatever reason (it should not).
I took the chance to change the structure of config. In particular I split BoardConfig into multiple structs as well as AddonConfig. I used the structure of Webconfig UI and the Webconfig API as a rough guideline.
Migration from 0.7.1. On first boot the legacy config will be converted to Protobuf config. If everything goes well users should not notice any change and will not lose any settings.

The following was tested by me:

Loading / saving
Building with GitHub Actions
Migration from 0.7.1 to Protobuf
Booting after flash nuke
Webconfig API
Changing Input Mode / D-Pad Mode / SOCD Cleaning Mode
Changing Hotkey Settings
Changing Pin Mapping
Changing Keyboard Mapping
Changing Display Configuration
Input Reverse Addon
On-Board LED Addon
BOOTSEL Button Config (disables my OLED, but the Webconfig UI says this can happen)

The following addons should still be tested by those with the required hardware:

Analog Addon
Turbo Addon
Joystick Selection Slider Addon
I2C Analog ADS1219 Addon
Dual Directional Input Addon
Buzzer Speaker Addon
Extra Button Addon
Player Number Addon
SOCD Cleaning Mode Selection Slider Addon
PS4 Mode Addon
Wii Extension Configuration Addon
LED Configuration
Custom LED Theme

I'd also like to hear feedback for the current config schema. If we want to change something it will be much easier to do this now than later.

I will provide proper documentation that explains how to change the config schema with an additional PR.

This approach reduces the binary size by ~200kb

nanopb would not generate code section with enum type defines if the source file does not contain any enums itself. However enums can be included from other source files.

Implemented toJson() and fromJSON()

No more loading / saving of ConfigLegacy structs

It is not part of the 0.7.1 release

henrebotha · 2023-05-19T11:12:10Z

Configuration is not stored with Protobuf serialization

Typo?

Must have gotten lost in a merge

mthiesen · 2023-05-19T11:27:04Z

Configuration is not stored with Protobuf serialization

Typo?

Yes, fixed. Thanks.

src/addons/jslider.cpp

arntsonl

Code looks good! I didn't checkbox the 500+ protobuf library files, but I looked at everything else. Lots of various fixes and improvements, this is a big change so we'll have to do some merging with all the PR features coming in.

arntsonl · 2023-05-19T22:39:01Z

compile_proto.cmake

@@ -0,0 +1,52 @@
+function (compile_proto)
+	find_package(Python3 REQUIRED COMPONENTS Interpreter)


We'll want to make a note that we require Python again for protobuf since we removed that previously.

arntsonl · 2023-05-19T22:40:50Z

headers/addons/ps4mode.h

@@ -20,8 +22,16 @@ class PS4ModeAddon : public GPAddon {
    virtual std::string name() { return PS4ModeName; }
 private:
 	struct mbedtls_rsa_context rsa_context;
-    uint8_t hashed_nonce[32];
-    PS4Options ps4Options;
+    mbedtls_mpi_uint bytesN[64] = {};


Was removing PS4Options for something with zeroing out the structs? All good either way!

arntsonl · 2023-05-25T15:44:35Z

src/config_legacy.cpp

+        bool displayInvert;
+        int displaySaverTimeout;
+        ButtonLayoutCustomOptions buttonLayoutCustomOptions;
+        char boardVersion[32]; // 32-char limit to board name


We might be able to use boardVersion when doing legacy conversions

arntsonl · 2023-05-25T18:55:40Z

Bug #1: Custom LED enable mode does not save

Go to web configurator
Click enable and create a custom LED theme
Click save, and restart back into web configurator
Navigate back to Custom LED theme and observe the option is no longer selected

arntsonl

Fixed!

* Add an option to allow switching Touchpad and Share buttons (#254) Add an option to switch between Touchpad and Share buttons for PS4 Mode * Update development.md (#261) typo embeded to embedded * Update usedPins on save on /pin-mapping page (#262) * Update button labels for button label change and TP-Switch swap (#266) * Fix serialization of switchTpShareForDs4 * Add button labels for PS4 * Setup button labels for PS4 with TP & Share switch * Remove outdated Buttons constants * Update references to GP2040 in www (#264) * Update issue templates * SNES Controller Input Support (#252) * SNES controller support * adds webconfiguration options for SNESpad controller input * fix: corrects NES button mapping * fix[SNESpad]: updates lib to latest with snes mouse to analog working correctly * Fix invalid reference to button label (#279) * Protobuf Config (#263) * Build makefsdata for Linux w/ MUSL * Restored deflate support to makefsdata * Added nanopb library * Initial version of Proto config * Added AnimationOptions_Proto * Storage::toJSON() WIP * Pretty JSON output * Use std::string instead of std::ostringstream This approach reduces the binary size by ~200kb * Simplify ToJSON() and implented repeated support * Default initialization * Fixed issue in nanopb_generator.py nanopb would not generate code section with enum type defines if the source file does not contain any enums itself. However enums can be included from other source files. * Moved Config functions to separate file Implemented toJson() and fromJSON() * Restored StorageManager to its pristine state * Config loading and saving * Make FlashPROM cache public for the time being * Implemented api/getConfig and api/setConfig endpoints * Moved most Gamepad enums to enums.proto * Use DEFAULT_INPUT_MODE and DEFAULT_DPAD_MODE * Renamed generic/short enum values * Moved bulk of config structs to the ConfigLegacy namespace No more loading / saving of ConfigLegacy structs * Moved GamepadOptions and GamepadHotkeyEntry to ConfigLegacy * FlashPROM::start() is no longer needed * Added more settings to config.proto * Support for the latest config changes from main * Store config struct on the heap to avoid stack overflow * Increase LWIP_HTTPD_POST_MAX_PAYLOAD_LEN to 8kb We need more to fit the whole config * Further stack optimizations Only save config if content has changed * Removed template to reduce code bloat * Update config.boardVersion and save after ConfigUtils::load() This makes sure that any migrations performed are preserved in EEPROM * Config instance in StorageManager * Renamed getAddonOptions() -> getLegacyAddonOptions() Renamed setAddonOptions() -> setLegacyAddonOptions() * Added getters for sub-configs * Migrate PinMappings to Protobuf * Fixed wrong name caused by overeager renaming * Migrated reverse addon to Protobuf storage * Mirrored latest version of AnimationOptions in Protobuf * Fixed compatibility with arm-none-eabi-gcc 11.3.1 * Fixed crash issue in Release builds * Fixed board configs * Added commented out PICO_DEOPTIMIZED_DEBUG to CMakeList.txt Use this to toggle between optimized / unoptimized builds * Added support for byte arrays to base64 encoder / decoder * Store binary data as byte arrays, convert to Base64 in JSON * Migrated api/getSplashImage and api/setSplashImage * Change footer magic to invalidate incompatible stored data * Refactor pin assignment from JSON * Add bool to int workaround for web-config * Migrate SOCD Slider addon to ProtoBuf * Migrate On-Board LED addon to ProtoBuf * Migrate Analog Input addon to ProtoBuf * Migrate BOOTSEL button addon to ProtoBuf * Migrate Buzzer Speaker addon to ProtoBuf * Migrate Dual Directional Input addon to ProtoBuf * Migrate Extra Button addon to ProtoBuf * Migrate I2C Analog ADS1219 addon to ProtoBuf * Migrate Joystick Selection Slider addon to ProtoBuf * Migrate Player Number addon to ProtoBuf * Migrate Turbo addon to ProtoBuf * Migrate Wii Extension addon to ProtoBuf * Migrate PS4 Mode addon to ProtoBuf * Check writeCache when determining whether to save Due to the delay in FlashPROM we run the risk of delaying the save indefinitely * We only allow saves from core0. Saves from core1 have to be marshalled to core0. * Migrate AnimationOptions * Check serial and signature for size as well * Removed unncessary cast * Moved PS4Options to config_legacy.cpp * Migrate Display addon to ProtoBuf * Removed old code related to SplashImage * Removed old code related to AddonOptions * Removed stale functions from ConfigManager * Moved OnBoardLedMode to config_legacy.cpp * Fixed issues caused by the merge * Added new player LED properties to LEDOptions * No more usage of ConfigLegacy::GamepadOptions * Added switchTpShareForDs4 to config.proto * Conversion from legacy storage to Protobuf * Return -1 for invalid pins from webconfig * More conversion from legacy storage to Protobuf * Validate enums when converting from legacy config * Remove switchTpShareForDs4 from legacy conversion It is not part of the 0.7.1 release * More conversion from legacy storage to Protobuf * Removed references to ConfigLegacy::BoardOptions * Fixed compilation of KB2040 * Re-added /api/getConfig function entry Must have gotten lost in a merge * Fixed issue where JSLider addon fails to change dpadMode * Migrated SNES options to Protobuf * Fixed saving of AnimationOptions --------- Co-authored-by: deeebug <77402236+deeebug@users.noreply.github.com> * Make danger zone an actual button (#284) * Reference splash bitmap data from BoardConfig.h (#282) * Reference splash bitmap data from BoardConfig.h * Remove unused constant for bitmap * Add documentation on default splash customization * .gitignore a couple files used by vim + tools (#267) * Implement Forced Setup Mode & Hotkey Lock settings (#273) * Implement Forced Setup Mode & Hotkey lock settings * Rename enums * Add warning to web-config for Forced Setup Mode * Update warning check text * Fix BoardConfig.h * Feature: webui color schemes (#295) * WIP * DarkMode is a Component not Service * Give the theme button a right margin * Use Dropdown from `react-bootstrap` and dynamically create dropdown items * Rename to ColorScheme * move colorScheme to AppContext * Stop setting localStorage here * Simplify setTheme logic and ensure data-bs-theme is 'light' not 'auto' when light system scheme * Remove unused imports * More contrast for the navbar * Fixed PS4Options saving (#298) Fixed ps4Options saving * Move invert Y to process() to avoid hotkey issues (#300) * Preparing 0.7.2 for release candidate (#310) * Fix #306 (#307) * Fixed PLEDs not being properly retrieved from protobuf (#311) * Fix #265 (#312) Fix external links * Disabled /api/getConfig (#318) * Add Stress FightPad (#319) * Add Stress FightPad * Add controller photo * New photo, blurred the NSFW text * Updating config for full censor, docs/index.html will be updated when release goes out. * Exclude fields from JSON config export (#322) * Fix button labels and hotkey when Touchpad Share are swapped (#317) * Fix Hotkey when Touchpad and Share are swapped in PS4 Mode When the Touchpad and Share buttons are swapped in PS4 Mode, Hotkey F1 will be the Touchpad button, and Start will not be the Option button, but the Start button. * Fix button label when touchpad and share swapped in ps4 mode When selecting button label is not the PS4 button label, the button label may be different. * Fix button label when touchpad and share swapped in ps4 mode When selecting button label is not the PS4 button label, the button label may be different. * Fix button label when touchpad and share swapped in ps4 mode When selecting button label is not the PS4 button label, the button label may be different. * Fix button label when touchpad and share swapped in ps4 mode When selecting button label is not the PS4 button label, the button label may be different. * Fix WebConfig button label when touchpad and share swapped in ps4 mode When selecting button label is not the PS4 button label, the button label may be different. * This change did not make sense to be part of the web config modification. * Doc updates for v0.7.2 (#323) * Doc updates * Remove unused image * Styling updates --------- Co-authored-by: deeebug <77402236+deeebug@users.noreply.github.com> Co-authored-by: Wren <67722465+wrennnnnn@users.noreply.github.com> Co-authored-by: Pelsin <ian@vidales.se> Co-authored-by: Henré Botha <henrebotha@gmail.com> Co-authored-by: TheTrain <32771064+TheTrainGoes@users.noreply.github.com> Co-authored-by: Robert Dale Smith <2039258+RobertDaleSmith@users.noreply.github.com> Co-authored-by: Malte Thiesen <mail@maltethiesen.net> Co-authored-by: Oliver Sayers <171312+SavageCore@users.noreply.github.com> Co-authored-by: Brian S. Stephan <bss@incorporeal.org> Co-authored-by: FeralAI <13342258+FeralAI@users.noreply.github.com> Co-authored-by: teikai1216 <133628715+teikai1216@users.noreply.github.com>

mthiesen added 30 commits March 12, 2023 14:19

Build makefsdata for Linux w/ MUSL

dc3236e

Restored deflate support to makefsdata

4659a97

Merge remote-tracking branch 'upstream/main' into main

2136666

Added nanopb library

4291b69

Merge remote-tracking branch 'upstream/main' into ProtobufSettings

96f74b6

Initial version of Proto config

aff0e9d

Added AnimationOptions_Proto

ca0b3f0

Storage::toJSON() WIP

05dbb21

Pretty JSON output

8672790

Use std::string instead of std::ostringstream

3e97571

This approach reduces the binary size by ~200kb

Simplify ToJSON() and implented repeated support

ee64ab4

Default initialization

5e84afe

Fixed issue in nanopb_generator.py

69ac593

nanopb would not generate code section with enum type defines if the source file does not contain any enums itself. However enums can be included from other source files.

Moved Config functions to separate file

e25bbbd

Implemented toJson() and fromJSON()

Restored StorageManager to its pristine state

3b8e79e

Merge remote-tracking branch 'upstream/main' into ProtobufSettings

beb3dea

Config loading and saving

c8c8c88

Make FlashPROM cache public for the time being

9051bc8

Merge remote-tracking branch 'upstream/main' into ProtobufSettings

0c195bd

Implemented api/getConfig and api/setConfig endpoints

8b3dc35

Moved most Gamepad enums to enums.proto

db2dd98

Use DEFAULT_INPUT_MODE and DEFAULT_DPAD_MODE

70ca17e

Merge remote-tracking branch 'upstream/main' into ProtobufSettings

058b66d

Renamed generic/short enum values

4f71763

Merge remote-tracking branch 'upstream/main' into ProtobufSettings

7fd2058

Moved bulk of config structs to the ConfigLegacy namespace

eb50f9a

No more loading / saving of ConfigLegacy structs

Moved GamepadOptions and GamepadHotkeyEntry to ConfigLegacy

c2362f0

FlashPROM::start() is no longer needed

0ac40ee

Added more settings to config.proto

dc93eb2

Merge remote-tracking branch 'upstream/main' into ProtobufSettings

4206e2e

mthiesen added 10 commits May 17, 2023 21:35

Added switchTpShareForDs4 to config.proto

85d6398

Conversion from legacy storage to Protobuf

4258fca

Return -1 for invalid pins from webconfig

983281a

More conversion from legacy storage to Protobuf

b806be9

Validate enums when converting from legacy config

1c3b2c0

Remove switchTpShareForDs4 from legacy conversion

949a9cb

It is not part of the 0.7.1 release

More conversion from legacy storage to Protobuf

41f5720

Removed references to ConfigLegacy::BoardOptions

9505d57

Fixed compilation of KB2040

2e0ca64

Merge remote-tracking branch 'upstream/main' into ProtobufSettings

72fa060

mthiesen changed the title ~~Proto buf~~ Protobuf Config May 19, 2023

mthiesen requested review from deeebug, arntsonl, Fortinbra and FeralAI May 19, 2023 11:07

Re-added /api/getConfig function entry

0bbf552

Must have gotten lost in a merge

bsstephan reviewed May 19, 2023

View reviewed changes

src/addons/jslider.cpp Outdated Show resolved Hide resolved

Fixed issue where JSLider addon fails to change dpadMode

f0d92d3

arntsonl previously approved these changes May 25, 2023

View reviewed changes

mthiesen added 2 commits May 25, 2023 19:36

Merge remote-tracking branch 'upstream/main' into ProtobufSettings

baf7832

Migrated SNES options to Protobuf

f87dfb2

mthiesen dismissed arntsonl’s stale review via f87dfb2 May 25, 2023 18:06

Fixed saving of AnimationOptions

05b265a

arntsonl approved these changes May 25, 2023

View reviewed changes

arntsonl merged commit 3987f20 into main May 25, 2023
2 checks passed

arntsonl deleted the ProtoBuf branch August 18, 2023 15:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Protobuf Config #263

Protobuf Config #263

mthiesen commented May 19, 2023 •

edited

henrebotha commented May 19, 2023

mthiesen commented May 19, 2023

arntsonl left a comment

arntsonl May 19, 2023

arntsonl May 19, 2023

arntsonl May 25, 2023

arntsonl commented May 25, 2023

arntsonl left a comment

		@@ -0,0 +1,52 @@
		function (compile_proto)
		find_package(Python3 REQUIRED COMPONENTS Interpreter)

Protobuf Config #263

Protobuf Config #263

Conversation

mthiesen commented May 19, 2023 • edited

henrebotha commented May 19, 2023

mthiesen commented May 19, 2023

arntsonl left a comment

Choose a reason for hiding this comment

arntsonl May 19, 2023

Choose a reason for hiding this comment

arntsonl May 19, 2023

Choose a reason for hiding this comment

arntsonl May 25, 2023

Choose a reason for hiding this comment

arntsonl commented May 25, 2023

arntsonl left a comment

Choose a reason for hiding this comment

mthiesen commented May 19, 2023 •

edited