database schema broken since update to portainer/portainer-ce:latest

[tobilektri]

Temporary solution:

Rollback to 2.19.0 also rollbacks the db migration & works then!

Before you start please confirm the following.

• Yes, I've searched similar issues on GitHub.
• Yes, I've checked whether this issue is covered in the Portainer documentation or knowledge base.

Problem Description

Tonight there was a push to the new Docker-Image "latest" - as I am pulling the latest Images automatically via watchtower my
portainer(s) failed to start with the error

2023/12/07 07:30AM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:403 > The database schema version does not align with the server version. Please consider reverting to the previous server version or addressing the database migration issue. |

A rollback to 2.19.3 does not work (although my previous version was 2.19.3). A rollback to 2.19.0 solves the Problem (as stated in: #10638).

Rollback to 2.19.0

portainer                | 2023/12/07 07:44AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:359 > encryption key file not present | filename=portainer
portainer                | 2023/12/07 07:44AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:382 > proceeding without encryption key |
portainer                | 2023/12/07 07:44AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer                | 2023/12/07 07:44AM INF github.com/portainer/portainer/api/datastore/backup.go:109 > creating DB backup |
portainer                | 2023/12/07 07:44AM INF github.com/portainer/portainer/api/datastore/backup.go:45 > copying DB file | from=/data/portainer.db to=/data/backups/common/portainer.db.2.19.4.20231207074436
portainer                | 2023/12/07 07:44AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer                | 2023/12/07 07:44AM INF github.com/portainer/portainer/api/datastore/migrate_data.go:113 > migrating database from version 2.19.4 to 2.19.0 |

then change to
image: portainer/portainer-ce:2.19.3

portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:369 > encryption key file not present | filename=portainer
portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:392 > proceeding without encryption key |
portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/datastore/backup.go:109 > creating DB backup |
portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/datastore/backup.go:45 > copying DB file | from=/data/portainer.db to=/data/backups/common/portainer.db.2.19.0.20231207074554
portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/datastore/migrate_data.go:113 > migrating database from version 2.19.0 to 2.19.3 |
portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/datastore/migrator/migrate_ce.go:79 > db migrated to 2.19.3 |
portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/chisel/service.go:198 > Found Chisel private key file on disk | private-key=/data/chisel/private-key.pem
portainer                | 2023/12/07 07:45:54 server: Reverse tunnelling enabled
portainer                | 2023/12/07 07:45:54 server: Fingerprint 3Zh8zdLJw6He9V6oa74I45qGHc/UrM09M82EvazMoPk=
portainer                | 2023/12/07 07:45:54 server: Listening on http://0.0.0.0:8000
portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:649 > starting Portainer | build_number=35327 go_version=1.20.5 image_tag=linux-amd64-2.19.3 nodejs_version=18.18.2 version=2.19.3 webpack_version=5.88.1 yarn_version=1.22.21
portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/http/server.go:357 > starting HTTPS server | bind_address=:9443
portainer                | 2023/12/07 07:45AM INF github.com/portainer/portainer/api/http/server.go:341 > starting HTTP server | bind_address=:9000

Expected Behavior

Portainer 2.19.4 works fine

Actual Behavior

Portainer 2.19.4 fails with db migration.

Steps to Reproduce

1. Use Portainer 2.19.3
2. Pull container 2.19.4
3. Start Container 2.19.4
4. Receive the error:

Portainer logs or screenshots

Log extract:

2023/12/07 07:30AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:369 > encryption key file not present | filename=portainer
2023/12/02023/12/07 07:30AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:392 > proceeding without encryption key |
2023/12/07 07:30AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/12/07 07:30AM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:403 > The database schema version does not align with the server version. Please consider reverting to the previous server version or addressing the database migration issue. |
2023/12/07 07:30AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:369 > encryption key file not present | filename=portainer
2023/12/07 07:30AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:392 > proceeding without encryption key |
2023/12/07 07:30AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/12/07 07:30AM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:403 > The database schema version does not align with the server version. Please consider reverting to the previous server version or addressing the database migration issue. |
2023/12/07 07:30AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:369 > encryption key file not present | filename=portainer
2023/12/07 07:30AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:392 > proceeding without encryption key |
2023/12/07 07:30AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/12/07 07:30AM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:403 > The database schema version does not align with the server version. Please consider reverting to the previous server version or addressing the database migration issue. |7 07:30AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:392 > proceeding without encryption key |
2023/12/07 07:30AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/12/07 07:30AM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:403 > The database schema version does not align with the server version. Please consider reverting to the previous server version or addressing the database migration issue. |
2023/12/07 07:30AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:369 > encryption key file not present | filename=portainer
2023/12/07 07:30AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:392 > proceeding without encryption key |
2023/12/07 07:30AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/12/07 07:30AM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:403 > The database schema version does not align with the server version. Please consider reverting to the previous server version or addressing the database migration issue. |
2023/12/07 07:30AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:369 > encryption key file not present | filename=portainer
2023/12/07 07:30AM INF github.com/portainer/portainer/api/cmd/portainer/main.go:392 > proceeding without encryption key |
2023/12/07 07:30AM INF github.com/portainer/portainer/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/12/07 07:30AM FTL github.com/portainer/portainer/api/cmd/portainer/main.go:403 > The database schema version does not align with the server version. Please consider reverting to the previous server version or addressing the database migration issue. |

Portainer version

2.19.4

Portainer Edition

Community Edition (CE)

Platform and Version

Docker version 24.0.4, build 3713ee1

OS and Architecture

Debian AMD64

Browser

No response

What command did you use to deploy Portainer?

docker compose up -d


'''
services:
  portainer:
    container_name: portainer
    image: portainer/portainer-ce:2.19.0
# before the error: image: portainer/portainer-ce:latest
    ports:
      - 9443:9443
    volumes:
      - /root/deployment/portainer/data:/data
      - /root/deployment:/root/deployment
      - /root/jmx-monitoring:/root/jmx-monitoring
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      npm_services:
        aliases:
          - portainer
      ipv6npm:
        aliases:
          - portainer
    restart: unless-stopped
  watchtower:
    image: containrrr/watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/config.json
    command: --interval 30

networks:
  npm_services:
    name: npm_services
  ipv6npm:
    name: ipv6npm
    enable_ipv6: true
    ipam:
      driver: default
      config:
        - subnet: 2001:xxxx:0800:0006:0001::/80
          gateway: 2001:xxxx:0800:0006:0001::1
'''

Additional Information

No response

[ErcinDedeoglu]

I hardly hit this too.

[rmi1974]

Same here. Shouldn't this be caught by any sort of basic QA before pushing releases?

[jamescarppe]

Apologies for the confusion today - we pulled the 2.19.4 image and retagged latest to 2.19.3 while we resolved #10780 - this work is now complete with a fix to that issue in 2.19.4, that version re-released, and latest re-tagged to it.