The server-manager leaves zombies remote-machines after termination

[gligneul]

Context

What is the bug that you are experiencing?

The server-manager leaves zombies remote-machines after termination.

Why is this bug relevant?

Wasting computer resources.

Expected behavior

The server-manager should kill all child processes during termination.

Actual behavior

The server-manager leaves zombies remote-machines after termination.

Steps to reproduce

Run server manager

./src/server-manager --manager-address=localhost:10000 &

⬇️

2023-12-08 15:27:06.975247 info server-manager pid:3661803 manager version is 0.8.2                                                    

Make sure you have a machine snapshot

ls ./machine_snapshot                                   

⬇️

0000000000001000-f000.bin    0000000060200000-200000.bin  0000000060800000-200000.bin   config.protobuf
0000000000020000-6000.bin    0000000060400000-1000.bin    0000000080000000-8000000.bin  hash
0000000060000000-200000.bin  0000000060600000-200000.bin  0080000000000000-4400000.bin

Start session

grpcurl -plaintext -import-path lib/grpc-interfaces/ -proto lib/grpc-interfaces/server-manager.proto -d '{"session_id": "foo", "machine_directory": "./machine_snapshot", "active_epoch_index": 0, "processed_input_count": 0, "server_deadline": {"checkin": 5000, "advance_state": 180000, "advance_state_increment": 10000, "inspect_state": 180000, "inspect_state_increment": 10000, "machine": 180000, "store": 180000, "fast": 5000}, "server_cycles": {"max_advance_state": 4294967296, "advance_state_increment": 4194304, "max_inspect_state": 4294967296, "inspect_state_increment": 4194304}}' localhost:10000 CartesiServerManager.ServerManager/StartSession

⬇️

2023-12-08 15:28:37.431159 info server-manager pid:3661803 Received StartSession request for session foo
2023-12-08 15-28-37 info remote-cartesi-machine pid:3662073 ppid:3661803 Initializing server on localhost:0
2023-12-08 15:28:37.456928 info server-manager pid:3661803 Received CheckIn for session foo
...

Terminate server manager

pkill -15 server-manager

⬇️

[1]  + 3661803 terminated  ./src/server-manager --manager-address=localhost:10000                                                      

Remote cartesi machine keeps running

ps

⬇️

3639572 pts/4    00:00:01 zsh
3662073 pts/4    00:00:10 remote-cartesi-
3662815 pts/4    00:00:00 ps

Environment

• Server-mangager v0.8.2
• Emulator v0.15.2
• Debian 12 x86_64 Linux
• This bug is not related to Docker

Possible solutions

Handle the termination signal and kill the child processes.

[gligneul]

@vfusco argued that this is a new functionality, not a bug. I don't wholly disagree, so I'm removing the bug tag.