[RFC] Streamdeck/control surface integration

[Julusian]

Describe the feature

As part of moving towards supporting more surfaces, we should think about how they are implemented. Nothing will happen on this until a release after 3.0.

Today we essentially have 2 implementations for most supported surface types. One in this repository, and one as part of companion-satellite.

Someone has suggested that it would be good for the surfaces to be implemented in something like what we have for modules.
Doing something like this would be beneficial, as it would mean that we could make companion-satellite support the same plugin system, and only have to implement each surface type once. It will also make it possible for people to write and contribute their own surface implementations, like we have for modules.
I think this is something we should do, once we are able to make better use of weirder surfaces.

This leads me to an idea:

Bundling satellite

Should we do usb directly in companion? Instead we could do it all via satellite and make the launcher run an instance of satellite too. For headless builds, it would also be another systemd unit to run, but I dont see that as a problem.
The benefit of this is that all usb devices will be done the exact same way. But we would have to figure out some ui complexity, and consider what the devices show when satellite has lost connection to companion. This wont give us any more cpu usage than we have today, as we are already running each usb device as its own child process.

Anyone else have other ideas on how we should simplify/improve the implementation and maintenance of supporting usb devices

[krocheck]

So let's say running all USB through an internal satellite instance is the method, which I'm not convinced is a good idea yet. Does that mean we have a universal installer for Companion where in the UI you select to run core and/or satellite (needs a new name), and thus if running satellite only we provide a dropdown of Companion cores (probably call them servers at that point) on the network to connect to (in addition to a host/ip manual entry option)? That would be pretty cool.

We're already doing threading of the surfaces, so it doesn't bother me too much to drop an IP layer in instead of the child process 'tunnel' we currently use. My question is, only because I haven't looked at the satellite code thoroughly, does satellite use a similar threading setup per surface and do those establish their own IP connections rather then going through the master? That being the case, it would seem logical to roll that in as suggested.

What I think we need to do in satellite is allow an id field to be able to be tagged into the surface when it reaches the UI so that if you have multiple satellites you can tell what surface is coming from where. That way if you're running core+satellite it can be tagged localhost and that can be filtered out so the surfaces display normally.

What if we run a slimmed down UI for satellite that lets you define the default remote Companion for a surface but then you could on a per-surface basis define a host to connect to? Have 3 surfaces connected to 3 different Companions ... that would be pretty cool for some users I'd guess.

[Julusian]

So let's say running all USB through an internal satellite instance is the method, which I'm not convinced is a good idea yet. Does that mean we have a universal installer for Companion where in the UI you select to run core and/or satellite (needs a new name), and thus if running satellite only we provide a dropdown of Companion cores (probably call them servers at that point) on the network to connect to (in addition to a host/ip manual entry option)? That would be pretty cool.

Yeah, I hadnt thought of that but we definitly could consider it. I do wonder if it will become a ui nightmare, as it would be really good to be able to select specific surfaces to use (for things like xkeys so we play nice with other apps), and it would be nice to not have to implement these things twice. But it depends on how we decide they should be done/work.
We could also consider making standalone satellite be a second exe/start menu item that can be run, bundled in the same installer

We're already doing threading of the surfaces, so it doesn't bother me too much to drop an IP layer in instead of the child process 'tunnel' we currently use. My question is, only because I haven't looked at the satellite code thoroughly, does satellite use a similar threading setup per surface and do those establish their own IP connections rather then going through the master? That being the case, it would seem logical to roll that in as suggested.

No, everything is handled in the one process and thread. But it is using a custom build of node-hid to avoid the 4 surface limit that normal node-hid has. Im still not 100% sure if thats the reason that companion is using a process per surface or not (I suspect it is). But I am also in the middle of working on a fairly large rewrite of node-hid to make it asynchronous which will make them all be a lot happier in a single thread. So I am not worried about this for now

What I think we need to do in satellite is allow an id field to be able to be tagged into the surface when it reaches the UI so that if you have multiple satellites you can tell what surface is coming from where. That way if you're running core+satellite it can be tagged localhost and that can be filtered out so the surfaces display normally.

Yeah good idea. I have been thinking that we really need to add a location column to the table, and put in an ip address for the satellite ones, a name would also be good.

I am really considering removing the satellite- prefix from the ids of satellite surfaces. I dont think it makes sense to keep and in some ways is an anti-feature. You can plug the same streamdeck around satellite machines and companion will see it as the same surface and remember settings. But then connect it locally and now its a different surface?
Instead it would make more sense to use a hardware type prefix streamdeck- or xkeys- just to make sure we dont have collisions across types

[krocheck]

Yeah, I hadnt thought of that but we definitly could consider it. I do wonder if it will become a ui nightmare, as it would be really good to be able to select specific surfaces to use (for things like xkeys so we play nice with other apps), and it would be nice to not have to implement these things twice. But it depends on how we decide they should be done/work.
We could also consider making standalone satellite be a second exe/start menu item that can be run, bundled in the same installer

Yeah, I wasn't sure how to navigate that, especially for headless. 2nd exe would certainly do the trick for starts.

Yeah good idea. I have been thinking that we really need to add a location column to the table, and put in an ip address for the satellite ones, a name would also be good.

I am really considering removing the satellite- prefix from the ids of satellite surfaces. I dont think it makes sense to keep and in some ways is an anti-feature. You can plug the same streamdeck around satellite machines and companion will see it as the same surface and remember settings. But then connect it locally and now its a different surface?
Instead it would make more sense to use a hardware type prefix streamdeck- or xkeys- just to make sure we dont have collisions across types

I think the hardware prefix and s/n would be sufficient without the satellite- prefix, especially with a location column. Certainly settings being persistent for a surface whether its local or remote would be more intuitive for users.