Migration from 2.18.4 to 2.19.1 error (failed rebuilding stack file system with version: failed to copy stack 1 project folder: failed to backup source directory: Target path already exists)

[WorldSeso7]

Before you start please confirm the following.

• Yes, I've searched similar issues on GitHub.
• Yes, I've checked whether this issue is covered in the Portainer documentation or knowledge base.

Problem Description

I'm trying to update portainer-ee from version 2.18.4 to version 2.19.1 on Docker Standalone
When I start the container with the new image I'm getting an error on the log during the migration process.

Expected Behavior

The migration works well and portainer is updated

Actual Behavior

The migration fails and a backup is restored

Steps to Reproduce

docker stop portainer
docker rm portainer
docker pull portainer/portainer-ee:latest
docker run -d -p 8000:8000 -p 9443:9443 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ee:latest

I already updated the portainer agent

Portainer logs or screenshots

2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/datastore/backup.go:164 > creating DB backup |
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/datastore/backup.go:74 > copying DB file | from=/opt/portainer_data/portainer.db to=/opt/portainer_data/backups/EE/portainer.db.2.18.4.20231006100212
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/datastore/migrate_data.go:90 > migrating database from version 2.18.4 to 2.19.1 |
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/datastore/migrator/migrate_ce.go:51 > migrating data to 2.19.0 |
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/datastore/migrator/migrate_dbversion100.go:133 > enable community addon on all MicroK8s master nodes |
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/datastore/migrator/migrate_dbversion100.go:281 > Success to migrate private key seed to private key file |
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/datastore/migrator/migrate_dbversion100.go:287 > update edge stack status to have deployment steps |
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/datastore/migrator/refresh_rbac_roles.go:15 > refreshing RBAC roles |
2023/10/06 10:02AM WRN github.com/portainer/portainer-ee/api/datastore/migrate_data.go:57 > migration failed, restoring database to previous version | error="failed to migrate database: failed in Always migrations returned error: failed rebuilding stack file system with version: failed to copy stack 1 project folder: failed to backup source directory: Target path already exists" stack_trace=[{"func":"(*Store).MigrateData","line":"55","source":"migrate_data.go"},{"func":"initDataStore","line":"183","source":"main.go"},{"func":"buildServer","line":"481","source":"main.go"},{"func":"main","line":"784","source":"main.go"},{"func":"main","line":"250","source":"proc.go"},{"func":"goexit","line":"1598","source":"asm_amd64.s"}]
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/datastore/backup.go:226 > restoring DB backup |
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/datastore/backup.go:74 > copying DB file | from=/opt/portainer_data/backups/EE/portainer.db.2.18.4.20231006100212 to=/opt/portainer_data/portainer.db
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/database/boltdb/db.go:125 > loading PortainerDB | filename=portainer.db
2023/10/06 10:02AM INF github.com/portainer/portainer-ee/api/datastore/migrate_data.go:63 > database restored to previous version |
2023/10/06 10:02AM FTL main.go:185 > failed migration | error="failed to migrate database: failed in Always migrations returned error: failed rebuilding stack file system with version: failed to copy stack 1 project folder: failed to backup source directory: Target path already exists" stack_trace=[{"func":"(*Store).MigrateData","line":"55","source":"migrate_data.go"},{"func":"initDataStore","line":"183","source":"main.go"},{"func":"buildServer","line":"481","source":"main.go"},{"func":"main","line":"784","source":"main.go"},{"func":"main","line":"250","source":"proc.go"},{"func":"goexit","line":"1598","source":"asm_amd64.s"}]

Portainer version

2.18.4

Portainer Edition

Business Edition (BE/EE) with 5NF / 3NF license

Platform and Version

Docker 20.10.21

OS and Architecture

Ubuntu 20.04.4 LTS

Browser

No response

What command did you use to deploy Portainer?

docker run -d -p 8000:8000 -p 9443:9443 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ee:latest

Additional Information

No response

[Nick-Portainer]

Hi @WorldSeso7, Thanks for posting this. I am looking into this now and will come back if I need any further information etc :)

internal: PCT-212

[Nick-Portainer]

I've tested this myself, I have matched your Ubuntu version as well as your Docker version of 20.10.21. I have had no issues following the steps you documented here.

Can you please let me know how many environments you have running as well as what kind of container/stacks you are running? Just so I can try replicate your environment as close as possible.

Thanks!

[samdulam]

@WorldSeso7
Did you create a backup of Portainer before the upgrade?
Can you create a new fresh instance of Portainer (using a different path for portainer_data volume) then restore from backup. It is OK to restore directly to Portainer 2.19.1

https://docs.portainer.io/admin/settings#restoring-from-a-local-file

[JamborJan]

@samdulam Portainer updates are getting more and more a pain. I had the same issue. Its not that funny to always have to create new instances and re-create things from backups. I cannot count anymore how often I have done that and how many times I'm left behind with all stacks not being under my control anymore and having to re-create all webhooks from scratch. If Portainer wants to be a pro-application, than updates must be more stable. Sorry for the rant but I'm really annoyed by all the effort it takes over and over again.

Am I right that it is not enough to have a backup of the DB?

[Nick-Portainer]

@samdulam Portainer updates are getting more and more a pain. I had the same issue. Its not that funny to always have to create new instances and re-create things from backups. I cannot count anymore how often I have done that and how many times I'm left behind with all stacks not being under my control anymore and having to re-create all webhooks from scratch. If Portainer wants to be a pro-application, than updates must be more stable. Sorry for the rant but I'm really annoyed by all the effort it takes over and over again.

Am I right that it is not enough to have a backup of the DB?

You will need more than just the DB. If you go to Settings, then Backup Portainer, this will give you a complete backup of your Portainer config, which includes stacks and webhooks. When we mention to backup your Portainer, this is where we mean.

[JamborJan]

I postponed this cumbersome thing a bit. But now I installed a fresh instance of Portainer and restored a resent backup. But it is the same result: as soon as the backup is finished, the container keeps crashing. Is there any possible solution to this? Or do I need to setup everything from scratch?

Right now I reverted to 2.18.3 and can use my instance again. But I am stuck with this version.