fork fails

[drahnr]

Bug Report

I start my worker with a systemd unit file:

After=network.target
Requires=network.target
RequiresMountsFor=/media/concourse

[Service]
Type=simple
Restart=always
RestartSec=20s
LimitNOFILE=50000
Environment=CONCOURSE_WORKER_NAME=%i
Environment=CONCOURSE_TSA_HOST=...
Environment=CONCOURSE_TSA_PORT=...
Environment=CONCOURSE_TSA_PUBLIC_KEY=tsa_host_key.pub
Environment=CONCOURSE_WORKER_PRIVATE_KEY=worker_key
Environment=CONCOURSE_WORK_DIR=...
Environment=CONCOURSE_KEY_DIR=...
ExecStart=/usr/local/bin/concourse \
		worker \
		--name=${CONCOURSE_WORKER_NAME} \
		--work-dir=${CONCOURSE_WORK_DIR} \
		--tsa-host=${CONCOURSE_TSA_HOST}:${CONCOURSE_TSA_PORT} \
		--tsa-public-key=${CONCOURSE_KEY_DIR}/${CONCOURSE_TSA_PUBLIC_KEY} \
		--tsa-worker-private-key=${CONCOURSE_KEY_DIR}/${CONCOURSE_WORKER_PRIVATE_KEY} \
		--garden-network-pool=172.16.0.0/16

ExecStop=-/usr/local/bin/concourse \
		retire-worker \
		--name=${CONCOURSE_WORKER_NAME} \
		--tsa-host=${CONCOURSE_TSA_HOST} \
		--tsa-port=${CONCOURSE_TSA_PORT} \
		--tsa-public-key=${CONCOURSE_KEY_DIR}/${CONCOURSE_TSA_PUBLIC_KEY} \
		--tsa-worker-private-key=${CONCOURSE_KEY_DIR}/${CONCOURSE_WORKER_PRIVATE_KEY}

[Install]
WantedBy=multi-user.target

[40343.164471] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/1d3e89ad-9813-4588-66fd-f972f35bae6f
[40343.165004] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/1388dafa-6a8b-4f26-64cf-5c996adce18d
[40343.165047] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/6e1d623f-0a22-42ff-6f23-e02ca483bfc4
[40343.166958] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/161c8f0f-6c24-4d04-7398-18cbc410ccc2
[40343.168586] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/767e8f11-f387-44a0-4f70-72462ddb2efb
[40343.170652] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/2d630157-dc25-4c33-739a-eaa4d898deb8
[40343.172100] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/dbdf6e1c-e4a2-48e8-6dd2-bd9dcd54f071
[40343.177167] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/cf7ff62e-e435-497f-74ad-c0825d2b1a99
[40343.180030] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/5147a90f-62a4-49e7-663b-aaf24a2be83d
[40343.182424] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/3f0a5653-8f9c-4ca2-6da9-1f3e13ce8952
[40343.183122] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/39efd78d-7f31-4bc7-50f8-53def939d87c
[40343.184871] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/2eceefe8-bbde-4cbe-548a-ddc95041f882
[40343.185343] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/7b5d00d7-1f1b-435d-7d4c-cafcc6da6183
[40343.200227] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/4b18737f-aec7-453e-6fdb-6770d070995a
[40343.200679] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/7f5a07e1-36b7-4ecf-6f47-5b9cfe237fb9
[40343.200853] cgroup: fork rejected by pids controller in /system.slice/system-concourse\x2dworker.slice/concourse-worker@bourbon.service/garden/a147bf56-b1f2-4228-4566-acee0ac89107

The following can also be handy:

• Concourse version: 3.14.1
• Deployment type (BOSH/Docker/binary): binary worker / docker web
• Infrastructure/IaaS: custom / Fedora / CoreOS
• Browser (if applicable): Firefox
• Did this used to work? Yep
• Bagageclaim Driver: btrfs

[drahnr]

Related: systemd/systemd#3211

[drahnr]

This is more of notification that small deployments with many jobs+tasks might run into.

[marco-m]

@drahnr I am not sure I understand, is the workaround to bump DefaultTaskMax ? To what ?

[drahnr]

@marco-m Yes, the TasksMax variable in the systemd unit file has to be added, because the default is 512 which is way too low for a concourse worker with a decent amount of jobs scheduled.
Even before systemd >= 228 the default was around 4096 which occasionally caused build failures for me.

[marco-m]

See also #480 for Concourse and systemd limits / configuration

[vito]

closing in favor of #480, thanks for the info!