Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Docker container restart time #4896

Open
sharky98 opened this issue Jun 5, 2024 · 3 comments
Open

Improve Docker container restart time #4896

sharky98 opened this issue Jun 5, 2024 · 3 comments

Comments

@sharky98
Copy link
Contributor

sharky98 commented Jun 5, 2024

Describe the problem

Problem

When restarting a stack, the handleStartup() and runServer() commands take a long time to complete.

In my last restart:

  • + /opt/kimai/bin/console -n kimai:install took 5 minutes to complete at the clearing cache step.
  • + /opt/kimai/bin/console -n kimai:update took 4 minutes to complete at the clearing cache step.
  • + /opt/kimai/bin/console kimai:reload --env=prod took 5 minutes to complete at the clearing cache step.

Steps

  1. Start a stack.
  2. Use it for some time.
  3. Restart the stack.

Ideas

By-pass the cache during install

The command kimai:install has a no-cache options. It could be used in .docker/service.sh#L35-L44.

Proposed diff
function handleStartup() {
  # These are idempotent, run them anyway
-  /opt/kimai/bin/console -n kimai:install
+  /opt/kimai/bin/console -n kimai:install --no-cache
  /opt/kimai/bin/console -n kimai:update
  if [ ! -z "$ADMINPASS" ] && [ ! -a "$ADMINMAIL" ]; then
    /opt/kimai/bin/console kimai:user:create superadmin "$ADMINMAIL" ROLE_SUPER_ADMIN "$ADMINPASS"
  fi
  echo "$KIMAI" > /opt/kimai/var/installed
  echo "Kimai2 ready"
}

By-pass the cache during update

The command kimai:update does not have a no-cache options. It could be added similar to the kimai:install command.

Proposed diff
    protected function configure(): void
    {
        $this
            ->setDescription('Update your Kimai installation')
            ->setHelp('This command will execute all required steps to update your Kimai installation.')
+           ->addOption('no-cache', null, InputOption::VALUE_NONE, 'Skip cache re-generation')
        ;
    }

    protected function execute(InputInterface $input, OutputInterface $output): int
    {
        $io = new SymfonyStyle($input, $output);


        $io->title('Kimai updates running ...');


        $environment = $this->kernelEnvironment;


        // make sure database is available, Kimai running and installed
        try {
            if (!$this->connection->createSchemaManager()->tablesExist(['kimai2_users', 'kimai2_timesheet'])) {
                $io->error('Tables missing. Did you run the installer already?');


                return Command::FAILURE;
            }


            if (!$this->connection->createSchemaManager()->tablesExist(['migration_versions'])) {
                $io->error('Unknown migration status, aborting database update');


                return Command::FAILURE;
            }
        } catch (ConnectionException $e) {
            $io->error(['Database connection could not be established.', $e->getMessage()]);


            return Command::FAILURE;
        } catch (\Exception $ex) {
            $io->error(['Failed to validate database.', $ex->getMessage()]);


            return Command::FAILURE;
        }


        // execute latest doctrine migrations
        try {
            $command = $this->getApplication()->find('doctrine:migrations:migrate');
            $cmdInput = new ArrayInput(['--allow-no-migration' => true]);
            $cmdInput->setInteractive(false);
            if (0 !== $command->run($cmdInput, $output)) {
                throw new \RuntimeException('CRITICAL: problem when migrating database');
            }


            $io->writeln('');
        } catch (\Exception $ex) {
            $io->error($ex->getMessage());


            return Command::FAILURE;
        }

+        if (!$input->getOption('no-cache')) {
            // flush the cache, in case values from the database are cached
            $cacheResult = $this->rebuildCaches($environment, $io, $input, $output);


            if ($cacheResult !== Command::SUCCESS) {
                $io->warning(
                    [
                        sprintf('Updated %s to version %s but the cache could not be rebuilt.', Constants::SOFTWARE, Constants::VERSION),
                        'Please run the cache commands manually:',
                        'bin/console cache:clear --env=' . $environment . PHP_EOL .
                        'bin/console cache:warmup --env=' . $environment
                    ]
                );
            } else {
                $io->success(
                    sprintf('Congratulations! Successfully updated %s to version %s', Constants::SOFTWARE, Constants::VERSION)
                );
            }
+        }

        return Command::SUCCESS;
    }
Alternative

As an alternative, the kimai:update could store the result of the
doctrine:migrations:migrate command (see where below) and rebuild the cache only if migrations were executed.

if (0 !== $command->run($cmdInput, $output)) {

Improve super admin creation

As shown in the documentation (https://www.kimai.org/documentation/docker-compose.html), I used the ADMINMAIL and ADMINPASS environment variables to create the super user. However, on each restart I get error messages in the log [ERROR] email: This e-mail address is already in use. and [ERROR] username: The username is already used..

kimai/.docker/service.sh

Lines 39 to 41 in 93ca983

if [ ! -z "$ADMINPASS" ] && [ ! -a "$ADMINMAIL" ]; then
/opt/kimai/bin/console kimai:user:create superadmin "$ADMINMAIL" ROLE_SUPER_ADMIN "$ADMINPASS"
fi

Not sure how it could be improved though. However, seeing the [ERROR] tag while I was trying to figure out why Kimai didn't work (due to long restart time) made me think that the container exited at that moment. Maybe adding an ignore-error option to the command to be used in the handleStartup() command?

By-pass the cache during reload

Same idea as the kimai:reload command above. However, that would leave us with no cache rebuild. So maybe we could keep that one as-is as a single cache rebuild?

Describe your setup and add your Docker compose file (redact your credentials)

Environment

  • Host: linux x86_64 Ubuntu 22.04.4 LTS VM hosted on Proxmox
    • CPU: 4x @ 3.33GHz (virtualized as kmv64 on the VM)
    • Memory: 12.5 GB
  • Docker: 26.1.1 (API: 1.45)
  • Docker Compose: TBD (deployed via Portainer CE 2.20.2)

Docker Compose

name: kimai

services:
  app:
    image: kimai/kimai2:apache-2.15.0-prod
    restart: unless-stopped
    networks:
      - nginx-proxy-manager
      - default
    volumes:
      - var:/opt/kimai/var
    depends_on:
      - sqldb
    ports:
      - 8001:8001
    environment:
      - ADMINMAIL=${ADMINMAIL}
      - ADMINPASS=${ADMINPASS}
      - DATABASE_URL=mysql://${MYSQL_USER}:${MYSQL_PASSWORD}@sqldb/${MYSQL_DATABASE}?charset=utf8&serverVersion=${MYSQL_VERSION}
      - TRUSTED_HOSTS=localhost,127.0.0.1,timesheet.brunops.com
      - memory_limit=1024M

  sqldb:
    image: mysql:${MYSQL_VERSION}
    command: --default-storage-engine innodb
    restart: unless-stopped
    volumes:
      - mysql:/var/lib/mysql
    environment:
      - MYSQL_DATABASE=${MYSQL_DATABASE}
      - MYSQL_USER=${MYSQL_USER}
      - MYSQL_PASSWORD=${MYSQL_PASSWORD}
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD}
    healthcheck:
      test: mysqladmin -p$$MYSQL_ROOT_PASSWORD ping -h localhost
      interval: 20s
      start_period: 10s
      timeout: 10s
      retries: 3

networks:
  nginx-proxy-manager:
    external: true

volumes:
  mysql:
    driver: local
    driver_opts:
      type: nfs
      o: [REDACTED]
      device: [REDACTED]
  var:
    driver: local
    driver_opts:
      type: nfs
      o: [REDACTED]
      device: [REDACTED]

Logs

This is from my last restart.

2024-06-05T14:27:06.426989854Z + re='^[0-9]+$'
2024-06-05T14:27:06.427007063Z + [[ kimai?charset=utf8&serverVersion=5.7.44 =~ ^[0-9]+$ ]]
2024-06-05T14:27:06.427027348Z + DB_PORT=[$MYSQL_PORT]
2024-06-05T14:27:06.427042726Z + echo 'Wait for MySQL DB connection ...'
2024-06-05T14:27:06.427059313Z + php /dbtest.php sqldb kimai [$MYSQL_PORT] [$MYSQL_USER] [$MYSQL_PASSWORD]
2024-06-05T14:27:06.427048175Z Wait for MySQL DB connection ...
2024-06-05T14:27:06.457208235Z Testing DB:** new \PDO(mysql:host=sqldb;dbname=kimai;port=[$MYSQL_PORT], [$MYSQL_USER], [$MYSQL_PASSWORD], [ \PDO::ATTR_ERRMODE => \PDO::ERRMODE_EXCEPTION ]);*Connection established
2024-06-05T14:27:06.457225110Z + echo 'Connection established'
2024-06-05T14:27:06.457253549Z + handleStartup
2024-06-05T14:27:06.457262536Z + /opt/kimai/bin/console -n kimai:install
2024-06-05T14:27:06.658752483Z 
2024-06-05T14:27:06.662665574Z Kimai installation running ...
2024-06-05T14:27:06.662684070Z ==============================
2024-06-05T14:27:06.662693243Z 
2024-06-05T14:27:06.690077240Z Database `kimai` for connection named default already exists. Skipped.
2024-06-05T14:27:06.787129801Z 
2024-06-05T14:27:06.790291930Z  [OK] Already at the latest version ("DoctrineMigrations\Version20240326125247")
2024-06-05T14:27:06.790310902Z 
2024-06-05T14:27:06.790319963Z 
2024-06-05T14:27:06.790327893Z  Rebuilding your cache, please be patient ...
2024-06-05T14:27:06.796345067Z 
2024-06-05T14:27:06.798644066Z  // Clearing the cache for the prod environment with debug false                
2024-06-05T14:27:06.798662459Z 
2024-06-05T14:32:06.613574072Z  [OK] Cache for the "prod" environment (debug=false) was successfully cleared.  
2024-06-05T14:32:06.613620736Z 
2024-06-05T14:32:06.628792828Z 
2024-06-05T14:32:06.628955298Z  // Warming up the cache for the prod environment with debug false              
2024-06-05T14:32:06.628972176Z 
2024-06-05T14:32:11.065002712Z  [OK] Cache for the "prod" environment (debug=false) was successfully warmed.   
2024-06-05T14:32:11.065100861Z 
2024-06-05T14:32:11.065112016Z 
2024-06-05T14:32:11.065120096Z  [OK] Congratulations! Successfully installed Kimai version 2.15.0              
2024-06-05T14:32:11.065128158Z 
2024-06-05T14:32:11.101941895Z + /opt/kimai/bin/console -n kimai:update
2024-06-05T14:32:11.312604531Z 
2024-06-05T14:32:11.316507718Z Kimai updates running ...
2024-06-05T14:32:11.316524629Z =========================
2024-06-05T14:32:11.316533753Z 
2024-06-05T14:32:11.426159334Z 
2024-06-05T14:32:11.430052292Z  [OK] Already at the latest version ("DoctrineMigrations\Version20240326125247")
2024-06-05T14:32:11.430079176Z 
2024-06-05T14:32:11.430087794Z 
2024-06-05T14:32:11.430095482Z  Rebuilding your cache, please be patient ...
2024-06-05T14:32:11.438630867Z 
2024-06-05T14:32:11.440944958Z  // Clearing the cache for the prod environment with debug false                
2024-06-05T14:32:11.440961546Z 
2024-06-05T14:36:27.971960412Z  [OK] Cache for the "prod" environment (debug=false) was successfully cleared.  
2024-06-05T14:36:27.972024675Z 
2024-06-05T14:36:27.987796276Z 
2024-06-05T14:36:27.988002296Z  // Warming up the cache for the prod environment with debug false              
2024-06-05T14:36:27.988023114Z 
2024-06-05T14:36:31.791075674Z  [OK] Cache for the "prod" environment (debug=false) was successfully warmed.   
2024-06-05T14:36:31.791143926Z 
2024-06-05T14:36:31.791154789Z 
2024-06-05T14:36:31.791163189Z  [OK] Congratulations! Successfully updated Kimai to version 2.15.0             
2024-06-05T14:36:31.791177341Z 
2024-06-05T14:36:31.826539187Z + '[' '!' -z '[$ADMINPASS]' ']'
2024-06-05T14:36:31.826581330Z + '[' '!' -a [$ADMINMAIL] ']'
2024-06-05T14:36:31.826590789Z + /opt/kimai/bin/console kimai:user:create superadmin [$ADMINMAIL] ROLE_SUPER_ADMIN '[$ADMINPASS]'
2024-06-05T14:36:48.350635896Z 
2024-06-05T14:36:48.356872801Z  [ERROR] email: This e-mail address is already in use.                          
2024-06-05T14:36:48.356892337Z 
2024-06-05T14:36:48.356901512Z  [ERROR] username: The username is already used.                                
2024-06-05T14:36:48.356910012Z 
2024-06-05T14:36:48.371616435Z + echo 2.15.0
2024-06-05T14:36:48.477592160Z + echo 'Kimai2 ready'
2024-06-05T14:36:48.477632251Z Kimai2 ready
2024-06-05T14:36:48.477915903Z + runServer
2024-06-05T14:36:48.477936841Z + /opt/kimai/bin/console kimai:reload --env=prod
2024-06-05T14:36:48.641368818Z  Validating config file syntax ...
2024-06-05T14:36:48.695934555Z 
2024-06-05T14:36:48.702882796Z  [OK] All 25 YAML files contain valid syntax.                                   
2024-06-05T14:36:48.702903577Z 
2024-06-05T14:36:48.702912644Z 
2024-06-05T14:36:51.085371984Z 
2024-06-05T14:36:51.085445110Z  [OK] All 555 XLIFF files contain valid syntax.                                 
2024-06-05T14:36:51.085582681Z 
2024-06-05T14:36:51.085600180Z 
2024-06-05T14:36:51.085608997Z  Rebuilding your cache, please be patient ...
2024-06-05T14:36:51.093468530Z 
2024-06-05T14:36:51.095749560Z  // Clearing the cache for the prod environment with debug false                
2024-06-05T14:36:51.095767551Z 
2024-06-05T14:41:08.011863818Z  [OK] Cache for the "prod" environment (debug=false) was successfully cleared.  
2024-06-05T14:41:08.011926638Z 
2024-06-05T14:41:08.025534034Z 
2024-06-05T14:41:08.025752979Z  // Warming up the cache for the prod environment with debug false              
2024-06-05T14:41:08.025769585Z 
2024-06-05T14:41:12.724661976Z  [OK] Cache for the "prod" environment (debug=false) was successfully warmed.   
2024-06-05T14:41:12.724703049Z 
2024-06-05T14:41:12.724712148Z 
2024-06-05T14:41:12.724785498Z  [OK] Kimai config was reloaded                                                 
2024-06-05T14:41:12.724800195Z 
2024-06-05T14:41:12.757115585Z + chown -R 33:33 /opt/kimai/var
2024-06-05T14:42:02.751955187Z + '[' -e /use_apache ']'
2024-06-05T14:42:02.751994257Z + exec /usr/sbin/apache2 -D FOREGROUND
2024-06-05T14:42:03.274993445Z [Wed Jun 05 15:42:03.274643 2024] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.57 (Debian) configured -- resuming normal operations
2024-06-05T14:42:03.275025269Z [Wed Jun 05 15:42:03.274722 2024] [core:notice] [pid 1] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'
2024-06-05T14:42:09.990695678Z localhost:8001 127.0.0.1 - - [05/Jun/2024:15:42:08 +0100] "GET / HTTP/1.1" 302 581 "-" "curl/7.88.1"

Command used to run the container

No response

@sharky98 sharky98 added the docker label Jun 5, 2024
@kevinpapst
Copy link
Member

kevinpapst commented Jun 5, 2024

nfs shares are known for their terrible performance. That was the case 10+ years ago with Virtualbox and is likely the reason for your performance issues still today. Try to use a local mount and report the start time.
If a new version was deployed it is necessary to clear the cache, it has nothing to do with the migrations.

I did a docker rebuild/restart on a customer Synology today and it took ... don't exactly know, but definitely less than a minute.

@kevinpapst
Copy link
Member

@sharky98 did you try with a different share type?

@sharky98
Copy link
Contributor Author

Sorry, did not have time yet. Life got really busy. You can keep waiting for feedback label. But I am pretty sure you are right about the NFS share. I've seen a lot of slowing down in all apps that I transfered on the NFS lately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants