Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve migration path 0.7 -> 0.8 #7933

Open
ligi opened this issue Feb 21, 2021 · 2 comments
Open

Improve migration path 0.7 -> 0.8 #7933

ligi opened this issue Feb 21, 2021 · 2 comments
Labels
kind/enhancement A net-new feature or improvement to an existing feature

Comments

@ligi
Copy link

ligi commented Feb 21, 2021

The following situation. The migration from 0.7 -> 0.8 went smoothly on my desktop. So I did it on the server. Unfortunately there it now hangs in migration for hours:

ligi@komputing:~/bin$ ipfs daemon
Initializing daemon...
go-ipfs version: 0.8.0
Repo version: 11
System version: amd64/linux
Golang version: go1.15.8
Found outdated fs-repo, migrations need to be run.
Run migrations now? [y/N] y
  => Looking for suitable fs-repo-migrations binary.
  => None found, downloading.
  => Running: /tmp/go-ipfs-migrate964068256/fs-repo-migrations -to 11 -y
Found fs-repo version 10 at /home/ligi/.ipfs
===> Running migration 10 to 11...
applying 10-to-11 repo migration
  - opening datastore at "/home/ligi/.ipfs"
> Upgrading pinning to use datastore
  - importing from ipld pinner

I think this should print a warning before start that this will take so long. Especially as it is the first time it takes so long (did quite a lot of migrations) - maybe also offer a way to migrate in the background to avoid such long downtimes.
Yea the .ipfs folder is quite big in this case:

ligi@komputing:~$ du -hs .ipfs/
95G     .ipfs/

And it is on a spinning disc. But still a warning would be nice I think.

@ligi ligi added the kind/enhancement A net-new feature or improvement to an existing feature label Feb 21, 2021
@ligi
Copy link
Author

ligi commented Feb 21, 2021

just when writing the issue it finished:

converted 210055 pins from ipld storage into datastore updated version file

so at least it terminates after hours - happy the service is back up again ;-)

@aschmahmann
Copy link
Contributor

aschmahmann commented Feb 25, 2021

Yep, this migration can take a while depending on your setup. Mostly the people that the migration takes a long time for are the same ones who will be pretty happy about the migration (i.e. if you have 100k pins then adding/removing a single pin used to take a while, but will be much faster now).

If/when we do large migrations in the future it seems like a good idea to give better progress indicators as well as a heads up, and we should probably put a disclaimer in the release notes. Unfortunately background migrations are pretty difficult to pull off, especially for the ones that would take time and need to access large parts of the datastore, so IMO that seems unlikely to happen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/enhancement A net-new feature or improvement to an existing feature
Projects
None yet
Development

No branches or pull requests

2 participants