audiovm - Argument list too long - no sound for new qubes

[neowutran]

Qubes OS release

R4.2

Brief summary

For audiovm, after some time "qvm-start-daemon.py" print this kind of errors:

Cannot setup watch on /local/domain/154/data/vchan/3/4713/event-channel
libvchan_client_init_async_finish: Argument list too long

Consequence is that new qubes don't have a working audio

Steps to reproduce

No reliable step to reproduce yet.
Starting and stopping qubes maybe have an effect

[DemiMarie]

I wonder if qvm-start-daemon is running into a Xenstore quota problem.

[neowutran]

When I had this bug I ran xenstore-control quota SYS-AUDIO-XID and I got watches : 124 .
This value seems awfully close to the default max value of 128.

I augmented the watch quota per qubes to 300 with xenstore-control quota set watches 300.
Will see how it goes

[neowutran]

I reached 190. When I start a new dispvm, the number increase by 6. The number doesn't go down when the dispvm shutdown.
Biggest issue seems to be that the "watches" are not correctly released when a VM got shutdown.

Also, maybe the default value is a bit low, there are probably others things than starting vm that increase the number of watches. in the context of an audiovm, if the number is only increased when a VM is started, that limit the number of VM with working audio to 128/6=~21 (i agree, that already a big number of vm).

However I am using a custom version of qvm-start-daemon.py (https://github.com/neowutran/qubes-core-admin-client/blob/f00ae868e444244419dae70453bcc3b88ffcf872/qubesadmin/tools/qvm_start_daemon.py ), I maybe created the issue myself, I don't think it is case but I will verify and report here when I have some time.

update:
indeed unrelated to my modification (reverted the file to the original one, same issue)

[neowutran]

If I start a vm, the counter increase by 6.
As soon as the vm start actually using audio, the counter decrease by 6.

If I start a vm, and shutdown the said vm without actually using audio, the counter have increased by 6.

With my version of qvm_start_daemon.py, "sys-audio" start with a watches count of 10, meaning that with the default watches counts limit it can handle maximum ( 10 + x*6 < 128 => ) 19 vm not actually using audio before blocking any new vm from using audio

[marmarek]

Can you check process list in the audiovm? Perhaps some (pacat-simple-vchan) remain running after VM shutdown?

[neowutran]

It was indeed the case, and manually killing pacat-simple-vchan decrease the watches counter

[neowutran]

I tested to add that in qvm_start_daemon.py

       # def on_domain_stopped(self, vm, _event, **_kwargs):
       # ...........

        if xid != -1:
            self.cleanup_guid(xid)
            self.cleanup_pacat_process(xid)
        if stubdom_xid != -1:
            self.cleanup_guid(stubdom_xid)
            self.cleanup_pacat_process(stubdom_xid)

    def cleanup_pacat_process(self, xid):
        try:
            with open(self.pacat_pidfile(xid)) as f:
                pid = int(f.readline())
                os.kill(pid, signal.SIGTERM)
                print(f"Sent SIGTERM signal to pacat-simple-vchan process {pid}")
        except OSError:
            print(f"Failed to send SIGTERM signal for the pacat-simple-vchan with xid of {xid}")

seems to work, maybe there is a better way to do that