[BUG] very slow response time

[leonidas-o]

If you have problems with the installation please use our community forum

What set up are you using

• Cloud Hosted Version
• Self Hosted

Describe the bug
Running in a VM (3Gb RAM, 2 Cores), installed docker based leantime 3.0.7 and running behind a reverse proxy (traefik). It takes literally forever (~10s) until the site loads. After using the admin account to log-in, every click inside leantime takes around ~15s. When the dashboard is fully loaded and I click on the notifications thunder icon (top right), the MyToDos section disappears every time/ starts to reload.

The only message I see is in the container logs (podman logs leantime):

WARNING: [pool www] server reached pm.max_children setting (5), consider raising it

To Reproduce
Steps to reproduce the behavior:

1. deploy leantime via podman and podman-compose
2. see it being slow

Expected behavior
Website response time in milliseconds and not seconds.
I think traefik as reverse proxy can be ruled out as it is used for other applications and none of them has any performance issues at all.

Leantime Version
3.0.7

Server
Apache, nginx, IIS etc
docker.io/leantime/leantime:latest
reverse-proxy: traefik 3

PHP / MySQL Version
docker.io/library/mysql:8.0

Additional context
Rocky Linux 9.3 (Blue Onyx)
podman version: 4.6.1
podman-compose version 1.0.6

I'm seeing the following logs:

Creating configuration file!
2024-02-28 13:28:49,890 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2024-02-28 13:28:49,894 INFO supervisord started with pid 3
2024-02-28 13:28:50,897 INFO spawned: 'php-fpm' with pid 4
2024-02-28 13:28:50,900 INFO spawned: 'httpd' with pid 5
[28-Feb-2024 13:28:50] NOTICE: fpm is running, pid 4
[28-Feb-2024 13:28:50] NOTICE: ready to handle connections
2024-02-28 13:28:51,973 INFO success: php-fpm entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2024-02-28 13:28:51,974 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
127.0.0.1 -  28/Feb/2024:13:30:48 +0000 "GET /index.php" 302
127.0.0.1 -  28/Feb/2024:13:30:51 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:30:54 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:31:08 +0000 "GET /index.php" 302
127.0.0.1 -  28/Feb/2024:13:31:08 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:31:11 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:31:14 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:31:24 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:32:17 +0000 "POST /index.php" 303
127.0.0.1 -  28/Feb/2024:13:32:20 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:32:23 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:32:23 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:32:23 +0000 "GET /index.php" 200
[28-Feb-2024 13:32:34] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
127.0.0.1 -  28/Feb/2024:13:32:33 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:32:33 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:32:33 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:32:33 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:32:33 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:32:59 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:33:02 +0000 "GET /index.php" 200
[28-Feb-2024 13:33:08] WARNING: [pool www] server reached pm.max_children setting (5), consider raising it
127.0.0.1 -  28/Feb/2024:13:33:05 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:33:05 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:33:05 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:33:06 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:33:07 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:33:08 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:33:11 +0000 "PATCH /index.php" 200
127.0.0.1 -  28/Feb/2024:13:33:14 +0000 "PATCH /index.php" 200
127.0.0.1 -  28/Feb/2024:13:33:21 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:33:22 +0000 "GET /index.php" 200
127.0.0.1 -  28/Feb/2024:13:33:33 +0000 "GET /index.php" 200

[0xcryp70]

this error is related to php-fpm so check the setting and change it

in /etc/php-fpm.d/www.conf or something like this

pm.max_requests = 500

[leonidas-o]

@0xcryp70 as I said, I'm using docker based leantime, therefore the php-fpm setting you are talking about would then be inside the container. I found it inside the container under /usr/local/etc/php-fpm.d/www.conf, it was commented out.
I don't think that I should modify files inside the container, if there isn't a var in the .env file for that, then this is a shortcoming of the docker-leantime project.

Nevertheless, for testing purposes I quickly changed that and the setting is now active: pm.max_requests = 500, but saw the same warning again in the logs: WARNING: [pool www] server reached pm.max_children setting (5), consider raising it. So I also changed that pm.max_children from "5" to "15". Haven't seen the warning anymore, but the performance is still poor and around 10s for loading a site.

[0xcryp70]

as you see it didn't change in the error it says pm.max_children setting (5) means = 5 ( default )
ok so in docker based meantime there is a file named custom.ini, you can add this config to this file
config/custom.ini
(https://github.com/Leantime/docker-leantime/blob/master/config/custom.ini)
or you can change it inside your container, but you need to restart your php-fpm service after that

[leonidas-o]

I restarted the container, so basically I added pm.max_requests = 500 and restarted, still got the WARNING: [pool www] server reached pm.max_children setting (5), consider raising it warning. Then I updated that to 15 and restarted again, afterwards this warning disappeared but everything was still slow.

Right now, another warning came up: WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 6 total children

[0xcryp70]

anyway, there are a lot of ways to change this config inside the container, you can add a command for search and replace the config or you can change it inside the container and commit a new image or you can add this change to the docker file and generate the new image

[0xcryp70]

as I see you only need to optimize your nginx and PHP and there is no big deal
if you search you can see what is best for you to change

[leonidas-o]

okay, I guess config/custom.ini is the way to go but first I have to find out which values will increase the performance. As of now, I don't get any warnings anymore but it's simply slow, that's strange:

[0xcryp70]

use this configs:

pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 10
pm.max_spare_servers = 20
pm.max_requests = 2000

you can use gzip and cache for nginx too

Enable gzip compression

gzip on;
gzip_comp_level 5;
gzip_min_length 256;
gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

Enable caching

proxy_cache_path /path/to/cache levels=1:2 keys_zone=my_cache:10m max_size=10g inactive=60m;

I hope these help you

[leonidas-o]

I somehow had the feeling that it is not leantime but podman-compose. I still don't know why or how, but somehow I always end up having issues with this one component. So I deleted everything, just using podman I manually created the pod, the two containers and leantime is as fast as I would expect it. So somehow podman-compose must have screwed up something with the network. It just utilises podman itself so nothing really magical about it but still, when doing it manually, it works fine. Oh man, I'm sorry for wasting your time, thanks a lot. I hope your useful optimisations can still help someone out there.

[johncadengo]

@leonidas-o were you able to update this configuration within the docker container? How did you achieve it?

These warnings seem to be very simple to run into on a vanilla docker compose installation. Do you think we should just update the default configuration to address this issue?

[leonidas-o]

@johncadengo I've used the config/custom.ini on the host system (not inside the docker container).
Hmm, I fear I can't make a decision if or if not you should update the default configuration, I'm simply lacking knowledge over leantime and its behaviour with these settings and especially if other user would ran into issues with that new default settings.

[rqi14]

I have the same issue. Is it possible to fix it within the docker container/compose?

[rqi14]

@johncadengo I've used the config/custom.ini on the host system (not inside the docker container). Hmm, I fear I can't make a decision if or if not you should update the default configuration, I'm simply lacking knowledge over leantime and its behaviour with these settings and especially if other user would ran into issues with that new default settings.

Hi. Do you mind sharing how to use this file? I don't find this file in the docker compose. Is it simply to place this file on the host and mount it to config/custom.ini in the docker?

[rqi14]

I created a Dockerfile to solve this problem according to the advices in above comments. I posted it here for futural references.

# Use leantime/leantime:latest as the base image
FROM leantime/leantime:latest

# Specify the working directory
WORKDIR /usr/local/etc/php/conf.d

# Update memory_limit in custom.ini
RUN sed -i 's/memory_limit = 1G/memory_limit = 5G/' custom.ini

# Add additional settings to custom.ini
RUN echo 'pm = dynamic' >> custom.ini \
    && echo 'pm.max_children = 50' >> custom.ini \
    && echo 'pm.start_servers = 10' >> custom.ini \
    && echo 'pm.min_spare_servers = 10' >> custom.ini \
    && echo 'pm.max_spare_servers = 20' >> custom.ini \
    && echo 'post_max_size = 50M' >> custom.ini \
    && echo 'upload_max_filesize = 50M' >> custom.ini

# The above settings in custom.ini does not seem to work. change php-fpm settings directly
RUN sed -i 's/^pm = .*/pm = dynamic/' /usr/local/etc/php-fpm.d/www.conf \
    && sed -i 's/^pm.max_children = .*/pm.max_children = 50/' /usr/local/etc/php-fpm.d/www.conf \
    && sed -i 's/^pm.start_servers = .*/pm.start_servers = 10/' /usr/local/etc/php-fpm.d/www.conf \
    && sed -i 's/^pm.min_spare_servers = .*/pm.min_spare_servers = 10/' /usr/local/etc/php-fpm.d/www.conf \
    && sed -i 's/^pm.max_spare_servers = .*/pm.max_spare_servers = 20/' /usr/local/etc/php-fpm.d/www.conf \
    && sed -i 's/^pm.max_requests = .*/pm.max_requests = 2000/' /usr/local/etc/php-fpm.d/www.conf