Skip to content

Self-hosted runners launched through systemd should have different defaults #3909

Open
@Nathan-Furnal

Description

@Nathan-Furnal

Describe the bug
When following the steps to configure a self-hosted runner service the recommended way is to use the template, found here.

The file content is,

[Unit]
Description={{Description}}
After=network.target

[Service]
ExecStart={{RunnerRoot}}/runsvc.sh
User={{User}}
WorkingDirectory={{RunnerRoot}}
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=5min

[Install]
WantedBy=multi-user.target

When used in tandem with the Restart=always option or when manually restarting the service, child processes are still hanging around. Which causes multiple processes to pickup a job under the same runner and it will crash.

The flow is:

service -> calls runsvc.sh -> calls RunnerService.js -> spawns children processes

As seen in the RunnerService.js file,

        console.log("Starting Runner listener with startup type: service");
        listener = childProcess.spawn(
          listenerExePath,
          ["run", "--startuptype", "service"],
          { env: process.env }
        );
      }

To Reproduce

  1. Start a runner with a service
  2. Restart the service
  3. Witness multiple children processes still hanging around from the last process.

Expected behavior
No orphan processes hanging around after one of the (grand-)parent processes has been killed.

Since we're using systemd for lifecycle management of the processes, it should be handling all processes spawned by the service.

Runner Version and Platform

v2.325.0

RHEL8

Fix

We found that simply setting KillMode=control-group was enough to fix the issue, it is also what is recommended in the systemd docs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions