Crashing regularly with 2.0.7

[mikael-s-persson]

So far, the issues I'm seeing with v2.0.7 manifest in about 50% of my test suite crashing with errors like this (curated and abridged):

2025-09-03T16:14:28.610382437Z ERROR  : pc                  : Received payload could not be parsed on /status_signals, with error: INTERNAL: Received buffer (size 0) is not big enough for the declared size of 6 (4 + 2)!
2025-09-03T16:14:28.610393123Z ERROR  : platform_sounds_task: Received payload could not be parsed on /status_signals, with error: INTERNAL: Received buffer (size 0) is not big enough for the declared size of 6 (4 + 2)!


2025-09-03T16:14:27.380877954Z ERROR  : data_recorder_task  : Failed to stage new message on subspace source '/status_signals' with error: INTERNAL: Received header could not be parsed on /status_signals, with error:
2025-09-03T16:14:27.380877954Z ERROR  : data_recorder_task  :  from source.h:86
2025-09-03T16:14:27.380877954Z ERROR  : data_recorder_task  :  from subspace_source.h:99
2025-09-03T16:14:27.381049702Z ERROR  : pc                  : Failed to stage new message on subspace source '/status_signals' with error: INTERNAL: Received header could not be parsed on /status_signals, with error: 
2025-09-03T16:14:27.381134522Z ERROR  : platform_sounds_task: Failed to stage new message on subspace source '/status_signals' with error: INTERNAL: Received header could not be parsed on /status_signals, with error:
2025-09-03T16:14:27.381134522Z ERROR  : platform_sounds_task:  from source.h:86
2025-09-03T16:14:27.381134522Z ERROR  : platform_sounds_task:  from subspace_source.h:99


2025-09-03T16:14:29.050879062Z ERROR  : sound_player        : Traceback (most recent call last):
2025-09-03T16:14:29.050942683Z ERROR  : sound_player        :   File "mapless/hmi/sound_player.py", line 102, in <module>
2025-09-03T16:14:29.050964072Z ERROR  : sound_player        :     reader = SubspaceReader(
2025-09-03T16:14:29.050983028Z ERROR  : sound_player        :              ^^^^^^^^^^^^^^^
2025-09-03T16:14:29.051000774Z ERROR  : sound_player        :   File "mapless/subspace_tools/subspace_reader.py", line 49, in __init__
2025-09-03T16:14:29.051166652Z ERROR  : sound_player        :     self.subscriber = self.client.create_subscriber(channel_name=channel_name, type=channel_type, reliable=reliable)
2025-09-03T16:14:29.051350990Z ERROR  : sound_player        :                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-09-03T16:14:29.051383967Z ERROR  : sound_player        : RuntimeError: INTERNAL: Failed to open shared memory subspace_3379586623466085_.sound_request_0: No such file or directory

I'm working on getting coredumps and stuff, but posting this as a start, and maybe you recognize the issue.

[dallison]

Are you running the subspace server as a separate process or as a thread in your test?

[dallison]

I added a call to CleanupFileSystem in the server main function. Maybe there is a timing issue with that. Could you remove it and see if that helps (main.cc line 55)

[mikael-s-persson]

Yeah, that's it. I'm running servers as separate processes, and ~20 of them at the same time (bazel test --runs_per_test=20 ...). So, yeah, that cleanup function wreaks havoc.

[dallison]

So we have two options:

1. Make a command line option to disable the cleanup at startup
2. Clean up the session on exit.

The first one would involve telling the server not to clean up in test environments. The second would require that the server process is shut down cleanly and we would have to catch the signal or something.

Maybe we should do both. Are you ok with a command line option for special cases like your tests? Most of the time there will be one server per computer.

I added this because my /tmp directory on my mac was so full that ls wouldn't work. The same would apply to /dev/shm on linux so I thought this was safe. Apparently not.

What works best for you?

[dallison]

You know, I'll just make a flag with default false. In cruise we have something else to clean up the filesystem so we don't need to do it. Maybe I can set it to true on MacOS since I really doubt anyone is using subspace on a mac.

[mikael-s-persson]

I'd be fine with a command-line option, I already have to substitute the arguments for the tests (because I have to generate a unique "mktemp" unix-socket for each test instance, i.e., we basically have a config file list processes and their arguments, and for those bringup tests, I have to make a bunch of changes to that config to deal with things that can't run or devices that don't exist in the test environment).

I also think that catching signals more cleanly is desirable too. I don't know how related that is, but I think I had spurious failure issues and I might still have a special exception for processes failing on SIGPIPE, which I think was because the subspace server doesn't block signals and poll signalfds, like all my other client processes.

A third solution, which I might consider too (due to other reasons), is to just have my own main, since that's a pretty small piece of the server.

I guess another potential solution is to clean up only things that should be related to the same subspace server. Aren't all the shared-memory files named after the "socket_name"? So, why would the server clean up shm files based on a different socket_name?

Bad news is that whatever the issue is with 2.0.6, that I thought might be that race condition, is still in 2.0.7. A many-publisher channel silently drops a message on the publishing side (5 in 1000 tests, or so).

[dallison]

To get you going, I've added a command line option with default false except for mac. I'll do a better solution later if we need to.

I was unaware of a issue you mentioned. I thought the problem was a hang in your Concurrent test and I've fixed that. Publishers don't drop messages so it must be something else. Do you happen to have a unit test that shows it?

[dallison]

New version 2.0.8 released with new toolbelt and command line option.

[mikael-s-persson]

Thanks!

I was unaware of a issue you mentioned. I thought the problem was a hang in your Concurrent test and I've fixed that. Publishers don't drop messages so it must be something else. Do you happen to have a unit test that shows it?

Yeah, I was hoping this was the issue. In trying to stress-test to reveal the issue is how I got to that failing concurrency test, and was hoping it was also the issue I was seeing in my integration tests, but I guess it wasn't... I still have to figure out a way to reproduce it at a reasonable (and non-proprietary) setting.

[mikael-s-persson]

This might be a reproduction (fingers crossed): #38