bug: logto service is getting restarted and not running in self hosted Coolify #6535

capsmagick · 2024-08-31T08:58:57Z

When I pulled the latest release and updated the logto image in coolify the postgress container is working but the logto container is getting restarted continuously

Describe the bug

Expected behaviour

should be able to login to logto admin console.

How to reproduce?

Selfhost Coolify
Deploy Logto

Context

Logto Cloud
[# ] Self-hosted, Logto version =
- [ #] Container (Docker image)
- Raw Node.js

Yousema · 2024-09-04T15:24:34Z

Same here. Here is the log :

2024-09-04T15:23:30.854047018Z > cli
2024-09-04T15:23:30.854050165Z > logto db seed --swe
2024-09-04T15:23:30.854052537Z
2024-09-04T15:23:32.417189457Z info Seeding skipped
2024-09-04T15:23:32.574406855Z
2024-09-04T15:23:32.574421645Z > start
2024-09-04T15:23:32.574424619Z > cd packages/core && NODE_ENV=production node .
2024-09-04T15:23:32.574427163Z
2024-09-04T15:23:33.308972465Z (node:44) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time
2024-09-04T15:23:33.308995432Z (Use node --trace-warnings ... to show where the warning was created)
2024-09-04T15:23:34.790860554Z cache warn No Redis client initialized, skipping
2024-09-04T15:23:35.091098182Z pre error Found undeployed database alterations, you must deploy them first by npm run alteration deploy command.
2024-09-04T15:23:35.091112213Z
2024-09-04T15:23:35.091115142Z See https://docs.logto.io/docs/references/using-cli/database-alteration for reference.
2024-09-04T15:23:35.091117610Z
2024-09-04T15:23:35.091188294Z index error Error while initializing app:
2024-09-04T15:23:35.093866526Z index error Error: Undeployed database alterations found.
2024-09-04T15:23:35.093874957Z at checkAlterationState (file:///etc/logto/packages/core/build/main-HJQVK3RU.js:25850:9)
2024-09-04T15:23:35.093877885Z at async Promise.all (index 0)
2024-09-04T15:23:35.093880309Z at async checkPreconditions (file:///etc/logto/packages/core/build/main-HJQVK3RU.js:25814:3)
2024-09-04T15:23:35.093882653Z at async Promise.all (index 3)
2024-09-04T15:23:35.093884962Z at async file:///etc/logto/packages/core/build/main-HJQVK3RU.js:25886:3

JustMrMendez · 2024-09-12T19:16:37Z

problem continues

charIeszhao · 2024-09-18T04:10:03Z

Make sure you have run the database alteration scripts to upgrade your database too.
https://docs.logto.io/docs/references/using-cli/database-alteration/

JustMrMendez · 2024-09-18T12:54:42Z

Make sure you have run the database alteration scripts to upgrade your database too. https://docs.logto.io/docs/references/using-cli/database-alteration/

no alterations

skyturtleio · 2024-09-18T22:14:04Z

I'm having a similar issue when attempting to upgrade from v1.19.0 to v1.20.0. After Logto is upgraded, it keeps restarting. Reverting back to v1.19.0 fixed the infinite restarts.

To revert, I had to go from the latest to the 1.19 tag:

Previous Docker Compose with latest:

image: svhd/logto:${TAG-latest}

Docker Compose to revert back to v1.19.0:

image: 'svhd/logto:1.19'

I also shelled into the Docker container running Logto to check for database alterations, and there were not any:

/etc/logto # npx @logto/cli db alteration deploy
info Found 0 alteration to deploy

charIeszhao · 2024-09-19T03:59:14Z

@skyturtleio @JustMrMendez Have you guys upgrade your @logto/cli package to v1.20.0 too?
Or can you also post your error logs like @Yousema did? I can clearly see the problem with @Yousema was that there were undeployed database alterations. But I'm not sure if you are having the same issue.

skyturtleio · 2024-09-20T22:11:42Z

@charIeszhao you're right on the database alterations. Sorry about that. When I was shelling into my Docker container, it had already reverted back to v1.19.0 to get back into a healthy state.

Here is the error log when attempting to upgrade to v1.20.0:

2024-09-20T19:12:36.498573264Z 
    2024-09-20T19:12:36.498664206Z > cli
    2024-09-20T19:12:36.498673673Z > logto db seed --swe
    2024-09-20T19:12:36.498677382Z 
    2024-09-20T19:12:37.560505928Z info Seeding skipped
    2024-09-20T19:12:37.587172549Z npm notice
    2024-09-20T19:12:37.587202065Z npm notice New patch version of npm available! 10.8.2 -> 10.8.3
    2024-09-20T19:12:37.587206212Z npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.8.3
    2024-09-20T19:12:37.587208897Z npm notice To update run: npm install -g npm@10.8.3
    2024-09-20T19:12:37.587211233Z npm notice
    2024-09-20T19:12:37.768354301Z 
    2024-09-20T19:12:37.768382305Z > start
    2024-09-20T19:12:37.768388737Z > cd packages/core && NODE_ENV=production node .
    2024-09-20T19:12:37.768393746Z 
    2024-09-20T19:12:38.699741189Z (node:44) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time
    2024-09-20T19:12:38.699778308Z (Use `node --trace-warnings ...` to show where the warning was created)
    2024-09-20T19:12:40.919728328Z cache    warn No Redis client initialized, skipping
    2024-09-20T19:12:41.380557457Z pre      error Found undeployed database alterations, you must deploy them first by npm run alteration deploy command.
    2024-09-20T19:12:41.380576324Z 
    2024-09-20T19:12:41.380579419Z  See https://docs.logto.io/docs/references/using-cli/database-alteration for reference.
    2024-09-20T19:12:41.380581954Z 
    2024-09-20T19:12:41.380611069Z index    error Error while initializing app:
    2024-09-20T19:12:41.384797522Z index    error Error: Undeployed database alterations found.
    2024-09-20T19:12:41.384809636Z     at checkAlterationState (file:///etc/logto/packages/core/build/main-F72FFTZD.js:26444:9)
    2024-09-20T19:12:41.384812421Z     at async Promise.all (index 0)
    2024-09-20T19:12:41.384814526Z     at async checkPreconditions (file:///etc/logto/packages/core/build/main-F72FFTZD.js:26408:3)
    2024-09-20T19:12:41.384817020Z     at async Promise.all (index 3)
    2024-09-20T19:12:41.384819094Z     at async file:///etc/logto/packages/core/build/main-F72FFTZD.js:26480:3

Is there a way to run the CLI command for database alterations, e.g. logto db alteration deploy, from your link above during the deploy? I have included the stock Coolify Docker Compose file below. I noticed the database alteration command is interactive so I wasn't sure that would be possible.

If not, I also saw an example job to run for database alterations in the deployment section of the Logto docs. I'm not too familiar with jobs in a Docker Compose file. Any chance you know where I could place this in the Docker Compose file? I've tried updating my Docker Compose a few times, but the build keeps failing.

Coolify Docker Compose for Logto:

services:
  logto:
    image: 'svhd/logto:${TAG-latest}'
    depends_on:
      postgres:
        condition: service_healthy
    entrypoint:
      - sh
      - '-c'
      - 'npm run cli db seed -- --swe && npm start'
    environment:
      - TRUST_PROXY_HEADER=1
      - 'DB_URL=postgres://${SERVICE_USER_POSTGRES}:${SERVICE_PASSWORD_POSTGRES}@postgres:5432/${POSTGRES_DB:-logto}'
      - ENDPOINT=$LOGTO_ENDPOINT
      - ADMIN_ENDPOINT=$LOGTO_ADMIN_ENDPOINT
    healthcheck:
      test:
        - CMD-SHELL
        - 'exit 0'
      interval: 5s
      timeout: 20s
      retries: 10
  postgres:
    image: 'postgres:14-alpine'
    user: postgres
    environment:
      POSTGRES_USER: '${SERVICE_USER_POSTGRES}'
      POSTGRES_PASSWORD: '${SERVICE_PASSWORD_POSTGRES}'
      POSTGRES_DB: '${POSTGRES_DB:-logto}'
    volumes:
      - 'logto-postgres-data:/var/lib/postgresql/data'
    healthcheck:
      test:
        - CMD
        - pg_isready
        - '-U'
        - $SERVICE_USER_POSTGRES
        - '-d'
        - $POSTGRES_DB
      interval: 5s
      timeout: 20s
      retries: 10

Example database alteration from docs:

apiVersion: batch/v1
kind: Job
metadata:
  name: alteration
spec:
  template:
    spec:
      containers:
        - name: alteration
          image: ghcr.io/logto-io/logto
          imagePullPolicy: Always
          env:
            - name: CI
              value: 'true'
            - name: DB_URL
              value: postgresql://user:password@localhost:5432/logto
          command:
            - /bin/sh
          args:
            - '-c'
            - 'npm run alteration deploy latest'
      restartPolicy: Never

charIeszhao · 2024-09-21T02:52:28Z

@charIeszhao you're right on the database alterations. Sorry about that. When I was shelling into my Docker container, it had already reverted back to v1.19.0 to get back into a healthy state.

Here is the error log when attempting to upgrade to v1.20.0:

2024-09-20T19:12:36.498573264Z 
    2024-09-20T19:12:36.498664206Z > cli
    2024-09-20T19:12:36.498673673Z > logto db seed --swe
    2024-09-20T19:12:36.498677382Z 
    2024-09-20T19:12:37.560505928Z info Seeding skipped
    2024-09-20T19:12:37.587172549Z npm notice
    2024-09-20T19:12:37.587202065Z npm notice New patch version of npm available! 10.8.2 -> 10.8.3
    2024-09-20T19:12:37.587206212Z npm notice Changelog: https://github.com/npm/cli/releases/tag/v10.8.3
    2024-09-20T19:12:37.587208897Z npm notice To update run: npm install -g npm@10.8.3
    2024-09-20T19:12:37.587211233Z npm notice
    2024-09-20T19:12:37.768354301Z 
    2024-09-20T19:12:37.768382305Z > start
    2024-09-20T19:12:37.768388737Z > cd packages/core && NODE_ENV=production node .
    2024-09-20T19:12:37.768393746Z 
    2024-09-20T19:12:38.699741189Z (node:44) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time
    2024-09-20T19:12:38.699778308Z (Use `node --trace-warnings ...` to show where the warning was created)
    2024-09-20T19:12:40.919728328Z cache    warn No Redis client initialized, skipping
    2024-09-20T19:12:41.380557457Z pre      error Found undeployed database alterations, you must deploy them first by npm run alteration deploy command.
    2024-09-20T19:12:41.380576324Z 
    2024-09-20T19:12:41.380579419Z  See https://docs.logto.io/docs/references/using-cli/database-alteration for reference.
    2024-09-20T19:12:41.380581954Z 
    2024-09-20T19:12:41.380611069Z index    error Error while initializing app:
    2024-09-20T19:12:41.384797522Z index    error Error: Undeployed database alterations found.
    2024-09-20T19:12:41.384809636Z     at checkAlterationState (file:///etc/logto/packages/core/build/main-F72FFTZD.js:26444:9)
    2024-09-20T19:12:41.384812421Z     at async Promise.all (index 0)
    2024-09-20T19:12:41.384814526Z     at async checkPreconditions (file:///etc/logto/packages/core/build/main-F72FFTZD.js:26408:3)
    2024-09-20T19:12:41.384817020Z     at async Promise.all (index 3)
    2024-09-20T19:12:41.384819094Z     at async file:///etc/logto/packages/core/build/main-F72FFTZD.js:26480:3

Is there a way to run the CLI command for database alterations, e.g. logto db alteration deploy, from your link above during the deploy? I have included the stock Coolify Docker Compose file below. I noticed the database alteration command is interactive so I wasn't sure that would be possible.

If not, I also saw an example job to run for database alterations in the deployment section of the Logto docs. I'm not too familiar with jobs in a Docker Compose file. Any chance you know where I could place this in the Docker Compose file? I've tried updating my Docker Compose a few times, but the build keeps failing.

Coolify Docker Compose for Logto:

services:
  logto:
    image: 'svhd/logto:${TAG-latest}'
    depends_on:
      postgres:
        condition: service_healthy
    entrypoint:
      - sh
      - '-c'
      - 'npm run cli db seed -- --swe && npm start'
    environment:
      - TRUST_PROXY_HEADER=1
      - 'DB_URL=postgres://${SERVICE_USER_POSTGRES}:${SERVICE_PASSWORD_POSTGRES}@postgres:5432/${POSTGRES_DB:-logto}'
      - ENDPOINT=$LOGTO_ENDPOINT
      - ADMIN_ENDPOINT=$LOGTO_ADMIN_ENDPOINT
    healthcheck:
      test:
        - CMD-SHELL
        - 'exit 0'
      interval: 5s
      timeout: 20s
      retries: 10
  postgres:
    image: 'postgres:14-alpine'
    user: postgres
    environment:
      POSTGRES_USER: '${SERVICE_USER_POSTGRES}'
      POSTGRES_PASSWORD: '${SERVICE_PASSWORD_POSTGRES}'
      POSTGRES_DB: '${POSTGRES_DB:-logto}'
    volumes:
      - 'logto-postgres-data:/var/lib/postgresql/data'
    healthcheck:
      test:
        - CMD
        - pg_isready
        - '-U'
        - $SERVICE_USER_POSTGRES
        - '-d'
        - $POSTGRES_DB
      interval: 5s
      timeout: 20s
      retries: 10

Example database alteration from docs:

apiVersion: batch/v1
kind: Job
metadata:
  name: alteration
spec:
  template:
    spec:
      containers:
        - name: alteration
          image: ghcr.io/logto-io/logto
          imagePullPolicy: Always
          env:
            - name: CI
              value: 'true'
            - name: DB_URL
              value: postgresql://user:password@localhost:5432/logto
          command:
            - /bin/sh
          args:
            - '-c'
            - 'npm run alteration deploy latest'
      restartPolicy: Never

How about shelling into your docker container and upgrading @logto/cli to 1.20.0 first?

skyturtleio · 2024-09-21T04:28:49Z

@charIeszhao how do I upgrade the @logto/cli when I've shelled into the container? I think I'm missing something basic. In the docs and when I run --help, I only see options for init, db, connector, translate, and proxy.

charIeszhao · 2024-09-21T06:18:53Z

@charIeszhao how do I upgrade the @logto/cli when I've shelled into the container? I think I'm missing something basic. In the docs and when I run --help, I only see options for init, db, connector, translate, and proxy.

Can you do npm install -g @logto/cli?

Also if you upgrade the latest docker image, you also have the latest version of CLI in the container shell env, as they are always shipped together.

skyturtleio · 2024-09-21T23:54:15Z

@charIeszhao thanks for the help, really appreciate it. I was able to run npm install -g @logto/cli in my Docker container.

Yeah, I know the latest Docker image includes the CLI, but it was kind of a chicken and egg problem. When upgrading to the latest image (for Coolify at least), the container would keep restarting because there were database alterations that needed to be run. Because the container kept restarting, I wasn't able to shell into a container with v1.20.0 running to actually run logto db alt deploy. In my post above, my Logto container was running v1.19.0 so its CLI was not seeing any new database alterations.

Anyway, here is how I was able to upgrade Logto in Coolify:

On a working Logto v1.19.0 Docker container, shell into your container and run npm install -g @logto/cli to get the latest CLI
After the Logto CLI is updated, run logto db alt deploy and select 1.20.0
After database alterations have been made, then pull the latest image for Logto and deploy

charIeszhao · 2024-10-10T06:47:06Z

@charIeszhao thanks for the help, really appreciate it. I was able to run npm install -g @logto/cli in my Docker container.

Yeah, I know the latest Docker image includes the CLI, but it was kind of a chicken and egg problem. When upgrading to the latest image (for Coolify at least), the container would keep restarting because there were database alterations that needed to be run. Because the container kept restarting, I wasn't able to shell into a container with v1.20.0 running to actually run logto db alt deploy. In my post above, my Logto container was running v1.19.0 so its CLI was not seeing any new database alterations.

Anyway, here is how I was able to upgrade Logto in Coolify:

On a working Logto v1.19.0 Docker container, shell into your container and run npm install -g @logto/cli to get the latest CLI

After the Logto CLI is updated, run logto db alt deploy and select 1.20.0

After database alterations have been made, then pull the latest image for Logto and deploy

@skyturtleio Thanks for the explaination and detailed steps!

@capsmagick It seems we have a solution now, and I'm closing the ticket. Feel free to reopen it if necessary.

capsmagick added the bug Something isn't working label Aug 31, 2024

charIeszhao closed this as completed Oct 10, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bug: logto service is getting restarted and not running in self hosted Coolify #6535

bug: logto service is getting restarted and not running in self hosted Coolify #6535

capsmagick commented Aug 31, 2024

Yousema commented Sep 4, 2024

JustMrMendez commented Sep 12, 2024

charIeszhao commented Sep 18, 2024

JustMrMendez commented Sep 18, 2024

skyturtleio commented Sep 18, 2024

charIeszhao commented Sep 19, 2024 •

edited

Loading

skyturtleio commented Sep 20, 2024

charIeszhao commented Sep 21, 2024

skyturtleio commented Sep 21, 2024

charIeszhao commented Sep 21, 2024 •

edited

Loading

skyturtleio commented Sep 21, 2024

charIeszhao commented Oct 10, 2024

bug: logto service is getting restarted and not running in self hosted Coolify #6535

bug: logto service is getting restarted and not running in self hosted Coolify #6535

Comments

capsmagick commented Aug 31, 2024

Describe the bug

Expected behaviour

How to reproduce?

Context

Yousema commented Sep 4, 2024

JustMrMendez commented Sep 12, 2024

charIeszhao commented Sep 18, 2024

JustMrMendez commented Sep 18, 2024

skyturtleio commented Sep 18, 2024

charIeszhao commented Sep 19, 2024 • edited Loading

skyturtleio commented Sep 20, 2024

charIeszhao commented Sep 21, 2024

skyturtleio commented Sep 21, 2024

charIeszhao commented Sep 21, 2024 • edited Loading

skyturtleio commented Sep 21, 2024

charIeszhao commented Oct 10, 2024

charIeszhao commented Sep 19, 2024 •

edited

Loading

charIeszhao commented Sep 21, 2024 •

edited

Loading