Landing the norns-converged branch

[Dewb]

As I understand it, the goal of the converged branch was/is to combine the matron and crone processes into a single process, and use shared memory rather than OSC for the communication between the control and audio threads.

• Working branches from 2022-23:
  • norns core: https://github.com/monome/norns/tree/norns-converged
  • maiden: monome/maiden@main...ngwese:maiden:feature/norns-converged
• Issues: https://github.com/monome/norns/issues?q=is%3Aissue%20converged
• Some relevant lines posts:
  • https://llllllll.co/t/low-level-norns/22921/10
  • https://llllllll.co/t/approaching-3-0-universal-engine-support/25255/8
  • https://llllllll.co/t/approaching-3-0-universal-engine-support/25255/31

The first step IMO is getting to a new converged branch that's rebased on main so it contains all of the other improvements made in the last 3-4 years. I've done one pass of git rebase -i and I'm going to do it again on top of the nng-ci branch so I can make sure every commit builds cleanly before proceeding. I'm omitting commits 186fa72 (the merge from main into the converged branch), f7c00c3, and 421ca6e (changes already committed to main as 4fe2521 and 41e0817.) There are a bit of entangled changes but most of it seems fairly straightforward to resolve. Couple areas of concern are main-branch changes to the VU polls and the tape messages; those will presumably require some rewriting for the converged architecture.

Beyond that, I'm not sure what the next step is; what do we know about what work was still planned when development stopped?

[ngwese]

it’s been a bit so my recollection is fuzzy. the nng change was part of an intended step which changed maiden and the repl/ws traffic to using TLS and adding authentication. at the time the work was done the nng runtime did not offer a way of controlling the number of i/o threads it spun up which IMO needlessly created noise when debugging, consumed extra memory, and provided little if any benefit (norns perf needs did not require it). since that time i believe nng eventually added some control over thread count.

IIRC the nng change also addressed a case where getting output back from running a sub-process would would get truncated (due to the original implementation). that said there were weird issues with the converged process where spawning sub processes would trigger xruns in the audio thread. I think there was some hacky thing we did to work around that but i would have to go back to the code to remember. not sure the root cause was ever identified.

none of the tls and auth work was started.

in terms of the ambitions around crone + matron ipc or simplification i would have to defer to @catfact

i had done some work on lua bindings to lv2 with the intention of possibly making the compressor and reverb pluggable but again time/interest wasn’t there along with compatibility concerns (more variation in cpu load causing additional support for example). In this area the only work done was a set of lua bindings to lilv

[Dewb]

thanks! ah, so nng is not intended for crone-matron communication, but for the network traffic with maiden? Was any of the maiden-side work started? (edit: found the maiden norns-converged branch, added that to the top post.)

yes, it looks like nng did land some control over thread count: nanomsg/nng@a9e98e5?w=1

[catfact]

what do we know about what work was still planned when development stopped?

well as i recall we were trying to get to functional parity with a stable single-process architecture. i think there were still bugs with things like capturing output of child processes spawned by lua. but at a quick glance it seems like a first pass implementation was basically done?

one of the goals is/was performance. so that goal seems acheived already by the rewrite - i remember it being: more responsive under floods of audio engine commands; avoiding some overruns while somehow causing some others.

so next step was bug hunting. if there were systems that remained to be replaced, i can't remember what they are.

steps after that, well, i think its about being able to add certain features more easily. off top of my head?:

• would be easy to give lua scripts direct access to peek/poke softcut buffers, probably first thing i'd do for weird glitch people
• would have facilitated fixing up Tape / adding features, but @colinmcardell tackled that another way already
• would facilitate adding a scriptable LV2/LADSPA plugin host to crone

... i'm probably forgetting some ideas.

[Dewb]

bug hunting and supporting weird glitch people seems like a clear mandate for followup work!

per @ngwese’s comment above, is there anything to do with nng other than explore the thread cap options and tune performance?

[catfact]

I don't think so ; from my perspective was intended to drop in replace the other IPC library we were using for the web sockets.

Oh before I forget - there are couple useful things in that diff that are unrelated to the single process rework and could easily / usefully be merged separately. I'm particularly thinking of the lua API for jack connection management that ngwese added

[catfact]

(having scripts use that would be improvement over running jack_connect in child proc)

[catfact]

oh yes the sidecar process was a bit of a hackaround, so that the combined norns process (now actually being a real grownup jack client) wouldn't be the direct parent of lua-spawned processes.

[ngwese]

i cherry picked the direct jack api over to the main branch some months back since it was isolated. what didn’t get changed on main was how the initial connections are established in matron, those still shell out on the main branch. iirc the converged branch also has functions to reset the jack connections back to normal if people start experimenting and need to cleanup after the fact.

[colinmcardell]

I tend to wrap my head around things faster when they are visualized. How does this look (simplified)?

flowchart TD
    subgraph COA["norns re-architecture"]
        subgraph HW["norns Hardware"]
            subgraph NP["norns JACK client"]
                MT["matron (Control + Lua VM)"]
                CQ["ConcurrentQueue"]
                CT["crone (Audio + JACK)"]
                
                MT -- "Commands::post()" --> CQ
                CQ -- "handleCommand()" --> CT
                CT -- "direct C++ callbacks / event_post()" --> MT
            end
            
            WW["ws-wrapper (NNG WebSocket ↔ stdio)"]
            SD["sidecar (forked child)"]
            SC["SuperCollider (sclang)"]
            
            WW -- "stdio pipes" --> NP
            MT <-- "NNG IPC (req/rep)" --> SD
            SD -- "popen()" --> S["Subprocesses"]
            MT <-- "OSC (UDP)" --> SC
        end

        subgraph CLIENT["Client"]
            MAIDEN["maiden (Web IDE)"]
            MR["maiden-repl (CLI)"]
        end
        
        MAIDEN -- "NNG WebSocket" --> WW
        MR -- "NNG WebSocket" --> WW
    end

Loading

[Dewb]

Diagrams are awesome! I am still getting up to speed on the details here but that seems generally in line with my current fuzzy understanding.

One suggestion I can make is to put ws-wrapper as a block in between maiden and the norns/matron process, since it's on the hardware side of the maiden (running on client) <-> norns (running on hardware) connection.

[colinmcardell]

Ok. I've updated the diagram a bit. Should be reflecting more accurately what's going on.

[Dewb]

I rebased the norns-converged branch against current main, and it builds and starts up both in the dev container and on norns hardware.

main...Dewb:norns:norns-converged-2026

Still a little short of having a working system off this branch, though. After building and installing nng (see Dockerfile for steps used) the normal build process succeeds. I am able to run the new setup with:

$  ./stop.sh
$ systemctl start norns-jack.service
$ systemctl start norns-sclang.service
$ systemctl start norns-maiden.service
$ build/norns/norns

Startup proceeds encouragingly normally but we eventually crash with a std::bad_function_call.

GDB output

Starting program: /home/we/norns/build/norns/norns 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".
[Detaching after fork from child process 9895]
[New Thread 0x75e25280 (LWP 9896)]
[New Thread 0x75624280 (LWP 9898)]
[New Thread 0x74e23280 (LWP 9900)]
[New Thread 0x74622280 (LWP 9905)]
[New Thread 0x73e21280 (LWP 9918)]
[New Thread 0x73620280 (LWP 9919)]
[New Thread 0x72e1f280 (LWP 9920)]
[New Thread 0x7261e280 (LWP 9921)]
[New Thread 0x71e1d280 (LWP 9922)]
[New Thread 0x7161c280 (LWP 9923)]
[New Thread 0x70e1b280 (LWP 9924)]
[New Thread 0x7061a280 (LWP 9925)]
[New Thread 0x6fe19280 (LWP 9926)]
[New Thread 0x6f618280 (LWP 9927)]
[New Thread 0x6ee17280 (LWP 9928)]
[New Thread 0x6e616280 (LWP 9929)]
[New Thread 0x6de15280 (LWP 9930)]
[New Thread 0x6d614280 (LWP 9931)]
[New Thread 0x6ccff280 (LWP 9932)]
constructed Client: crone
constructed Client: softcut
initializing buffer management worker..
[New Thread 0x6314f280 (LWP 9933)]
setting up jack clients..
[New Thread 0x6294e280 (LWP 9934)]
[New Thread 0x628cd280 (LWP 9935)]
engine sample rate: 48000
[New Thread 0x5c09e280 (LWP 9936)]
engine sample rate: 48000
starting jack clients..
[New Thread 0x5beff280 (LWP 9937)]
[New Thread 0x5be7e280 (LWP 9938)]
connecting ports... 
starting OSC interface...
[New Thread 0x5bdfd280 (LWP 9939)]
osc methods: 
/hello []
/goodbye []
/quit []
/poll/start/vu []
/poll/stop/vu []
/poll/start/tape []
/poll/stop/tape []
/set/level/adc [f]
/set/level/dac [f]
/set/level/ext [f]
/set/level/cut_master [f]
/set/level/ext_rev [f]
/set/level/rev_dac [f]
/set/level/monitor [f]
/set/level/monitor_mix [if]
/set/level/monitor_rev [f]
/set/level/compressor_mix [f]
/set/enabled/compressor [f]
/set/enabled/reverb [f]
/set/param/compressor/ratio [f]
/set/param/compressor/threshold [f]
/set/param/compressor/attack [f]
/set/param/compressor/release [f]
/set/param/compressor/gain_pre [f]
/set/param/compressor/gain_post [f]
/set/param/reverb/pre_del [f]
/set/param/reverb/lf_fc [f]
/set/param/reverb/low_rt60 [f]
/set/param/reverb/mid_rt60 [f]
/set/param/reverb/hf_damp [f]
/set/enabled/cut [if]
/set/level/cut [if]
/set/pan/cut [if]
/set/level/adc_cut [f]
/set/level/ext_cut [f]
/set/level/tape_cut [f]
/set/level/cut_rev [f]
/set/level/in_cut [iif]
/set/level/cut_cut [iif]
/set/param/cut/rate [if]
/set/param/cut/loop_start [if]
/set/param/cut/loop_end [if]
/set/param/cut/loop_flag [if]
/set/param/cut/fade_time [if]
/set/param/cut/rec_level [if]
/set/param/cut/pre_level [if]
/set/param/cut/rec_flag [if]
/set/param/cut/play_flag [if]
/set/param/cut/rec_offset [if]
/set/param/cut/position [if]
/set/param/cut/pre_filter_fc [if]
/set/param/cut/pre_filter_fc_mod [if]
/set/param/cut/pre_filter_rq [if]
/set/param/cut/pre_filter_lp [if]
/set/param/cut/pre_filter_hp [if]
/set/param/cut/pre_filter_bp [if]
/set/param/cut/pre_filter_br [if]
/set/param/cut/pre_filter_dry [if]
/set/param/cut/post_filter_fc [if]
/set/param/cut/post_filter_rq [if]
/set/param/cut/post_filter_lp [if]
/set/param/cut/post_filter_hp [if]
/set/param/cut/post_filter_bp [if]
/set/param/cut/post_filter_br [if]
/set/param/cut/post_filter_dry [if]
/set/param/cut/voice_sync [iif]
/set/param/cut/level_slew_time [if]
/set/param/cut/pan_slew_time [if]
/set/param/cut/recpre_slew_time [if]
/set/param/cut/rate_slew_time [if]
/set/param/cut/buffer [ii]
/softcut/buffer/read_mono [sfffiiff]
/softcut/buffer/read_stereo [sfffff]
/softcut/buffer/write_mono [sffi]
/softcut/buffer/write_stereo [sff]
/softcut/buffer/clear []
/softcut/buffer/clear_channel [i]
/softcut/buffer/clear_region [ff]
/softcut/buffer/clear_region_channel [iff]
/softcut/buffer/clear_fade_region [ffff]
/softcut/buffer/clear_fade_region_channel [iffff]
/softcut/buffer/copy_mono [iifffffi]
/softcut/buffer/copy_stereo [fffffi]
/softcut/buffer/render [iffi]
/softcut/query/position [i]
/softcut/reset []
/set/param/cut/phase_quant [if]
/set/param/cut/phase_offset [if]
/poll/start/cut/phase []
/poll/stop/cut/phase []
/tape/record/open [s]
/tape/record/start []
/tape/record/pause [i]
/tape/record/stop []
/tape/play/open [s]
/tape/play/start []
/tape/play/pause [i]
/tape/play/stop []
/tape/play/loop [i]
/set/level/tape [f]
/set/level/tape_rev [f]
MATRON
norns version: 0.0.0
git hash: 59f88b8
platform: cm3 (2)
no user matronrc file (/home/we/matronrc.lua) found, using default
loading matronrc file: /home/we/norns/matronrc.lua
setup IO keys:gpio
[New Thread 0x5b5fc280 (LWP 9940)]
setup IO enc:gpio
[New Thread 0x5adfb280 (LWP 9941)]
setup IO enc:gpio
[New Thread 0x5a5fa280 (LWP 9942)]
setup IO enc:gpio
[New Thread 0x59df9280 (LWP 9943)]
IO setup OK.
font setup OK.
battery init...
[New Thread 0x58718280 (LWP 9944)]
[Detaching after vfork from child process 9945]
[New Thread 0x57dff280 (LWP 9947)]
[New Thread 0x573ff280 (LWP 9948)]
*** WARNING *** The program 'norns' uses the Apple Bonjour compatibility layer of Avahi.
*** WARNING *** Please fix your application to use the native API of Avahi!
*** WARNING *** For more information see 
[New Thread 0x56bfe280 (LWP 9953)]
[New Thread 0x55fff280 (LWP 9959)]
[New Thread 0x575fe280 (LWP 9960)]
[New Thread 0x557fe280 (LWP 9961)]
[New Thread 0x54ffd280 (LWP 9962)]
[New Thread 0x547fc280 (LWP 9963)]
[New Thread 0x53ffb280 (LWP 9964)]
[New Thread 0x537fa280 (LWP 9965)]
init oracle...
OSC rx port: 8888 
OSC crone port: 9999
OSC ext port: 57120
OSC remote port: 10111
[New Thread 0x52ff9280 (LWP 9966)]
init weaver...
starting main lua vm
running lua config file: dofile('/home/we/norns/lua/core/config.lua')
[New Thread 0x527f8280 (LWP 9967)]
init dev_monitor...
[New Thread 0x51ff7280 (LWP 9968)]
[New Thread 0x517f6280 (LWP 9969)]
setting cleanup...
init input...
[New Thread 0x50ff5280 (LWP 9970)]
i2c init...
ERROR (i2c/adc) failed to write [chan]
running startup...
error loading keyboard layout, using old value: us
start_audio(): 
[New Thread 0x507f4280 (LWP 9971)]
scanning devices...
dev_monitor: vendor=monome model=grid
dev_monitor: TTY appears to be a monome
monome device appears to be a grid; rows=8; cols=16; quads=2
[New Thread 0x4fff3280 (LWP 9972)]
handling pending events...
_norns.midi.add: 1, virtual, userdata: 0x597100
grid added:	2	monome 128 m4193185	m4193185
running post-startup...
_norns._post_startup
norns.startup_status.ok
[Thread 0x507f4280 (LWP 9971) exited]
# script clear
setting DAC level (main thread): 0.977679
crone_set_level_dac(0.977679)
&mixerCommands: 0x2268c0
### SCRIPT ERROR: NO SCRIPT
[New Thread 0x5757d280 (LWP 9973)]
Thread 50 "metro_loop" received signal SIG32, Real-time event 32.
[New Thread 0x57557280 (LWP 9974)]
[Thread 0x5757d280 (LWP 9973) exited]
[New Thread 0x4f7f2280 (LWP 9975)]
[New Thread 0x4eff1280 (LWP 9976)]
terminate called after throwing an instance of 'std::bad_function_call'
  what():  bad_function_call
Thread 53 "norns" received signal SIGABRT, Aborted.
[Switching to Thread 0x4eff1280 (LWP 9976)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x767ad804 in __GI_abort () at abort.c:79
#2  0x76e73f88 in __gnu_cxx::__verbose_terminate_handler() () from /lib/arm-linux-gnueabihf/libstdc++.so.6
#3  0x76e71acc in ?? () from /lib/arm-linux-gnueabihf/libstdc++.so.6
#4  0x76e71b58 in std::terminate() () from /lib/arm-linux-gnueabihf/libstdc++.so.6
#5  0x76e71f28 in __cxa_throw () from /lib/arm-linux-gnueabihf/libstdc++.so.6
#6  0x76e6d720 in std::__throw_bad_function_call() () from /lib/arm-linux-gnueabihf/libstdc++.so.6
#7  0x000808e0 in std::thread::_State_impl > >::_M_run() ()
#8  0x76e9d150 in ?? () from /lib/arm-linux-gnueabihf/libstdc++.so.6
#9  0x76f83300 in start_thread (arg=0x4eff1280) at pthread_create.c:477
#10 0x76872208 in ?? () at ../sysdeps/unix/sysv/linux/arm/clone.S:73 from /lib/arm-linux-gnueabihf/libc.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

This is about all the investigating I will have time to do until later next week at the earliest, so it would be great if someone feels like picking up and running with the baton a bit.

Notes:

• Some useful gdb configuration commands: handle SIG32 nostop, handle SIGTTIN nostop, set pagination off
• The commits 186fa72, f7c00c3, and 421ca6e had already been made to main in some form, so they were omitted from the rebase.
• In the process of trying to get each commit to compile independently during the interactive rebase, I ended up manually bringing a lot of the changes from a8d1eff, d58bbd1, and 5fe3dfd ahead into 3e01ffe, so in the end I just squashed those four commits into one.
• I renamed crone/src/Poll.h to crone/src/CronePoll.h. I hated having to do this, but it was necessary to keep the dev container build on macOS from picking up the local Poll.h when asking for the linux <poll.h>, since the container's overlayFS inherits some of the case-insensitivity behaviors from the host platform. There may be a better solution here but this was the most expedient. I did not rename the copy of Poll.h inside the softcut submodule, it didn't seem to be a problem.
• The nng changes built fine in the dev container, but when I built on the norns hardware, nng was missing symbols from libatomic. Adjusted the wscript config for libatomic slightly to bring it in correctly. This may not be necessary if nng is built/configured differently? I ran the same steps on the hardware that I added to the Dockerfile, which is based off the nng readme.
• I probably made some mistakes blending the new TAPE features that were added in both the main and converged branches in different ways, especially the new polls.

[colinmcardell]

Agree that the work should land in main as soon as possible! Thanks for combing through the brain dump/notes and reflecting and steering.

Give me a day or two to finish up the little fiddly bits and get that PR in order. In addition I can sort through the FIXMEs and TODOs to share, and we can triage together. It is a little tough to tell what is converged branch specific and what is from other efforts.

[colinmcardell]

Putting this here instead of in the PR comments for better visibility.

I've spent some time working through what it might look like to use v2.0 of nng as a submodule.

• Requires initing the submodule.
• Requires building the static library as a step within the Waf wscript.
• Requires refactoring the current use of nng within norns to get it up to date on the bleeding edge.

nng v2.0 is actually not an official release yet and a distro package of v2.0 hasn't been released.

I'm going to pause on the idea of getting nng in place as a submodule in general. The wscript changes required to get the static library built seems like a side quest.

For this PR #1883, I've switched the wscript to the check_cc approach exclusively so that it will rely upon the libnng-dev being installed on device. This is the path of least resistance to getting a working environment.

We can follow up with work to address:

1. Init and compile nng as a submodule (ideally v1.x to be code compatible with norns as it stands)
   • This is worth putting effort into to bring nng closer to the latest version of v1.x which has bug fixes. But actually might be unnecessary if the OS for norns is updated in a re-image, because newer versions of the OS have more recent versions of the libnng package available.
2. Update nng submodule to v2.0 and refactor usage throughout norns to use the updated nng API.
   • After reading through the details of nng v2.0 compared to v1.4.0, I think this would have some benefits within norns to use v2.0 regarding threading and memory usage, but the gains are mostly related to API cleanliness and really future-proofing. v2.x should likely be targeted once it has an official release.

Here is the command to install nng on device so that compilation can find it.
sudo apt-get install -y libnng-dev

[tehn]

i suggest we rely on libnng-dev at v1.x as the easiest path. it's not difficult for me to bundle in the .deb packages with the installer, though perhaps just as well the installer calls apt-get since it'll be connected to the internet. (i'd previously attempted to keep installers capable of being run offline, but that facility seems to have left some time ago.)

thanks again!

[Dewb]

why can't we pin the submodule to 1.x?

[colinmcardell]

We absolutely can do this. See number 1 above. I'm suggesting that we can take this on as a follow up.

It's work to get Waf to compile the libnng similarly to the system library. Maybe there is a way to bring in the nng dependency code compiled similar to other third-party deps.

Edit ---

If we are interested in avoiding pushing out a new image of norns for 2.9, we should pursue doing exactly this: pinning the sub-module to v1.x, getting it to compile as a library, linking to it, or getting it to compile directly within the binary.

For this PR which is bringing the code back to life, I suggest that we should take this easy path of linking to the system installed library for the moment. Get it merged in. Continue to test to make sure it's all working and iterate. Then in parallel get nng in as a submodule before making a release.

[tehn]

This is fantastic. I'm returning tomorrow from a stretch of travel and will review and be available to do further testing when required. Thanks for rebooting this effort! b

…

-------- Original Message --------

On Saturday, 02/21/26 at 04:44 Colin McArdell ***@***.***> wrote: Nice work! I have some bandwidth to run with it in the evenings. — Reply to this email directly, [view it on GitHub](#1879 (reply in thread)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AAB4I4H7UWHYFBWHRABAEQL4NBHJVAVCNFSM6AAAAACVUG7C4SVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTKOBYGA2DGMI). You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Dewb]

Should nng stay in the dev container (and future norns-image update), or are things easier in the long term if it becomes a submodule and is built by waf?

[colinmcardell]

Hm. Not sure. I think it should stay in the dev container. What do you think the pros and cons are?

[colinmcardell]

Ah. Ok. I'm trying to get tests running again and running into nng not being present when running the test GH action locally. This makes sense as it is using the workflow container image.

That said, it might make more sense for it to be set up as a submodule as it is a project dependency. This would then propagate to the actions without an image change if I'm understanding it correctly.

[tehn]

It seems we have quite a few reqs which are not submodules already, so while I like submodules, perhaps it's best to follow the existing pattern. (I was just refreshing myself with the wscript)

[catfact]

holy cow, amazing work @colinmcardell. appreciate the attention to detail, great to see multithreading mechanisms get some TLC.

there are various changes we could consider rolling into a 3.0, mostly or entirely non-breaking. concur that its best to keep a narrow focus on parity for this change, it would be well served by integrated testing with a wider user base, before introducing confounding changes to scripts &c.

[catfact]

opened discussion for milestone planning, with a couple suggestions to get the ball rolling:

#1885

[colinmcardell]

Cross-posting this over here for completeness from #1883, and furthering the discussion on next steps. I've got a stable environment at this point using the service definitions below. I made a change in the last commit to make it so that norns will not spawn a jack server as it was causing a race condition with the norns-jack.service on startup. It's possible that the start.sh and stop.sh needs a little tweaking given this change, I need to check.

norns-main.service

[Unit]
After=norns-jack.service
Requires=norns-jack.service

[Service]
Type=simple
User=we
Group=we
LimitRTPRIO=95
LimitMEMLOCK=infinity
ExecStart=/home/we/norns/build/ws-wrapper/ws-wrapper ws://*:5555 /home/we/norns/build/norns/norns

[Install]
WantedBy=norns.target

norns-sclang.service

[Unit]
After=norns-jack.service
Requires=norns-jack.service

[Service]
Type=simple
User=we
Group=we
LimitRTPRIO=95
LimitMEMLOCK=infinity
ExecStart=/home/we/norns/build/ws-wrapper/ws-wrapper ws://*:5556 /usr/local/bin/sclang -i maiden

[Install]
WantedBy=norns.target

---

Any thoughts on the norns update process at this time (update.sh vs re-image)?

These are the changes that need to happen:

1. disable/remove the obsolete norns-matron.service and norns-crone.service
2. deploy norns-main.service (single binary via ws-wrapper:5555)
3. deploy an updated norns-sclang.service (sclang via ws-wrapper:5556 for maiden SC REPL)
4. install libnng1 and libnng-dev - runtime dependency for both the converged binary's sidecar IPC and ws-wrapper

It seems like this is all manageable as an update.sh task. I gravitate towards the following as an idea to provide the changes needed without a re-image. Rather than inlining this logic into update.sh, we might use a lightweight migration hook:

update.sh iterates over numbered scripts in a migrations/ directory, and the converged transition becomes a single self-contained script (001_converged_services.sh). In this situation, migrations are written to be idempotent, where every operation checks state before acting (e.g., only disable a service if it exists, apt-get install is naturally a no-op if already installed). This makes them safe to re-run on dev devices where changes may have already been applied by hand. The existing version.txt mechanism can gate migrations by version if it's truly needed.

[Dewb]

An idempotent numbered migration script strategy is a great idea. Might be good to have rollback scripts as well so migrations can be undone, but I don’t know if it would be worth the effort.

[colinmcardell]

Yeah. I have to investigate more, and see how the update script can update the update script. 🙃

[colinmcardell]

Ok. I'm starting to better understand the update system.

Looks like the packaged .tgz file is downloaded and the extracted update.sh is run and would handle the migration. So it would be a matter of implementing the changes and the rest should be handled.

[tehn]

Yes, the update script I change as needed, then pack the update tgz manually.

I will take care of all of this part of it.

I can separately make a rollback script. It's a little fiddly but not too bad.

I haven't yet re-tested, what's the state of the maiden communication?

[colinmcardell]

browser

matron and supercollider communication is working for me in the browser.

maiden-repl cli

I'm going through the C code of the maiden-repl and checking the details. I had stashed some work a week or so ago to resolve issues with the switch to nng. I have to tidy that work up and commit, but it resolve the issues.

Basically we need to work around an issue with nng prioritizing IPv6 and ws-wrapper only has IPv4 bindings.

When the mDNS resolver resolves norns.local it returns both IPv6 and IPv4 addresses, nng tries IPv6 first, but ws-wrapper only binds IPv4, so the connection is immediately refused and nng doesn't fall back.

[colinmcardell]

Update on the maiden-repl CLI.

I've got the websocket dialing in place and tested working with the update to nng.
The default ws addresses used by maiden-repl when run on norns, which point to 127.0.0.1 were not the issue. It was when compiling and running from my laptop and attempting to connect to norns.local. This has now been fixed.

Once I fixed the IPv4 dialing and tested, I uncovered another issue, which sent me on a long and winding road. I ended up refactoring a number of the bits within maiden-repl trying to solve it, which will show up as threading improvements and UI improvements. But the issue turned out to be rather funny... sclang wants ESC instead of newline. And it looks like that same issue was surfaced on the web maiden repl because there is a nice comment about it. lame, sclang expects commands to be terminated with special command bytes Ha.

Alright. I'm going to get this work buttoned up and committed this weekend.

[colinmcardell]

Here is a guide for testing the work over on #1883

https://gist.github.com/colinmcardell/3b8982327c009c2c74f7e2a0dff22cab