Increase the default maximum number of tasks for balenaEngine

[alexgg]

BalenaOS uses the default systemd maximum number of tasks per unit which is 15% of the maximum number of tasks in the system, which currently is around 4915.

This is deemed to be sensible, except in the case it is inherited by balenaEngine and limits the number of tasks in a container application.

This issue will remove the limit for the balenaEngine unit.

[balena-ci]

[mbalamat] This issue has attached support thread https://jel.ly.fish/#/7b3aaf9b-fc3b-48f0-b63e-78931732a11a

[alexgg]

This is related to https://success.docker.com/article/how-to-reserve-resource-temporarily-unavailable-errors-due-to-tasksmax-setting

[alexgg]

For reference, the number of maximum threads spawned by a service unit can be tested by running the following C program from an application container.

// C program to find maximum number of threads
// compile: gcc thread_test.c -lpthread -o thread_test

#include <sys/time.h>
#include <sys/resource.h>
#include <stdio.h>
#include <pthread.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>

static int thread_duration_sec = 5;
static int n_threads = 1e6;

// This function demonstrates the work of thread
void *thread ( void *vargp){
    usleep(thread_duration_sec*1e6);
}

int main(int argc, char *argv[])
{
    // print out limit (relevant?)
    struct rlimit rlim;
    getrlimit(RLIMIT_NPROC, &rlim);
    printf("RLIMIT_NPROC soft: %lu, hard: %lu\n", rlim.rlim_cur, rlim.rlim_max);

    //
    int err = 0, count = 0;
    pthread_t tid;


    if (argc > 1) {
        thread_duration_sec = atoi(argv[1]);
    }
    if (argc > 2) {
        n_threads = atoi(argv[2]);
    }

    // on success, pthread_create returns 0 and
    // on Error, it returns error number
    // So, while loop is iterated until return value is 0
    while (err == 0 && count < n_threads)
    {
        err = pthread_create (&tid, NULL, thread, NULL);
        if (err == 0)
            count++;
    }

    if (err == 0) {
        printf("Threads created %d\n", count);
    }
    else {
        printf("Threads created %d, failed with err=%d, %s\n", count, err, strerror(err));
    }

    usleep(thread_duration_sec*1e6);
}

For example using the following Dockerfile:

FROM balenalib/%%BALENA_MACHINE_NAME%%-ubuntu:latest

MAINTAINER Alex Gonzalez <alexg@balena.io>

# Non-interactive debconf package configuration
ARG DEBIAN_FRONTEND=noninteractive

RUN apt-get update && apt-get install -y locales util-linux build-essential libpthread-stubs0-dev dbus

RUN echo "dash dash/sh boolean false" | debconf-set-selections - && dpkg-reconfigure dash

RUN echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen && \
	locale-gen en_US.utf8 && \
	update-locale LANG=en_US.UTF-8

ENV LANG en_US.UTF-8

RUN mkdir -p /usr/local/src
COPY src/ /usr/local/src

ENTRYPOINT ["/bin/bash"]

Unfortunately BalenaEngine also creates transient systemd units using the d-bus systemd API. For the maximum number of threads to be lifted, they also need to remove the limit by doing:

systemctl set-property docker-<container-id>.scope TasksMax=infinity

Unfortunately instrospecting the systemd dbus it does not seem possible to set this property programmatically.

I have yet to find a solution to do this without using the global DefaultTasksMax variable.

[alexgg]

Related branch https://github.com/balena-os/meta-balena/tree/alexgg/%231844-max-tasks

[alexgg]

Note that #1824 already sets:

TasksMax=infinity

[robertgzr]

@alexgg the commit that changes the service file was something I added to be closer to upstream, it's marked wip to signify it was up for debate if we want / don't want any of the changes

[jellyfish-bot]

[robertgzr] This issue has attached support thread https://jel.ly.fish/e5391b02-bceb-4f8d-994d-bd912b2ae9d7

[robertgzr]

maybe we can use runc annotations to set the required property: https://github.com/opencontainers/runc/blob/master/docs/systemd-properties.md

The problem is that there seems to be no good way to set those from docker. We would have to run an oci-hook and modify the runc config.json to inject those...

[jellyfish-bot]

[saintaardvark] This issue has attached support thread https://jel.ly.fish/b23a8cc0-c4fb-45b0-924c-15b8483cad7e

[kristiannoergaardx3d]

Is this issue actively been worked on? I keep hitting the limit around 4900 threads...

[alexgg]

@robertgzr did you get to running a proof-of-concept with runc's config.json annotations to increase the transit unit's TaskMask? Is this static runc configuration that we can hardcode?

[jellyfish-bot]

[alexgg] This issue has attached support thread https://jel.ly.fish/15b7c512-1c30-4a0e-9101-33a9bbb44913

[jellyfish-bot]

[alexgg] This issue has attached support thread https://jel.ly.fish/cdfeb30b-114f-45cd-a685-94d00fb9e9bf