Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ipfs: 0.4.23 -> 0.5.0 #86242

Merged
merged 6 commits into from May 8, 2020
Merged

ipfs: 0.4.23 -> 0.5.0 #86242

merged 6 commits into from May 8, 2020

Conversation

@lordcirth
Copy link
Contributor

lordcirth commented Apr 28, 2020

Motivation for this change

Major go-ipfs update to 0.5.0

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

Perhaps some commits should be squashed. I don't know why I the preload.sh script failed, some weird path problem. Seems to work fine without it.

@sorki
Copy link
Member

sorki commented Apr 29, 2020

Please rebase and squash the commits into one with commit message according to guidelines https://github.com/NixOS/nixpkgs/blob/master/.github/CONTRIBUTING.md#submitting-changes

Feel free to ping me if you need help.

@lordcirth lordcirth force-pushed the lordcirth:ipfs05 branch from af8555a to 8106ffb Apr 29, 2020
@lordcirth lordcirth force-pushed the lordcirth:ipfs05 branch from 8106ffb to 445d554 Apr 29, 2020
@lordcirth
Copy link
Contributor Author

lordcirth commented Apr 29, 2020

Ok, I think this is correct?

@vojta001
Copy link
Contributor

vojta001 commented Apr 29, 2020

According to the release notes on GitHub, the ipfs repo fsck command no longer exists as it is no longer necessary. We should remove it from nixos/modules/services/network-filesystems/ipfs.nix.

ipfs repo fsck # workaround for BUG #4212 (https://github.com/ipfs/go-ipfs/issues/4214)

@vojta001 vojta001 mentioned this pull request Apr 29, 2020
3 of 8 tasks complete
lordcirth added a commit to lordcirth/nixpkgs that referenced this pull request Apr 30, 2020
@mat8913

This comment was marked as resolved.

@lordcirth lordcirth force-pushed the lordcirth:ipfs05 branch from 65fbbcf to 4968990 May 5, 2020
@Mic92 Mic92 changed the title Ipfs05 ipfs: 0.4.23 -> 0.5.0 May 7, 2020
Mic92 added 4 commits May 7, 2020
@Mic92
Copy link
Contributor

Mic92 commented May 8, 2020

@GrahamcOfBorg test ipfs

@Mic92 Mic92 merged commit ddef887 into NixOS:master May 8, 2020
2 of 3 checks passed
2 of 3 checks passed
grahamcofborg-eval Checking original out paths
Details
tests.ipfs on aarch64-linux Success
Details
tests.ipfs on x86_64-linux Success
Details
@mweinelt
Copy link
Member

mweinelt commented May 8, 2020

Since this upgrade my local IPFS instance tells me it requires repo migrations and we don't seem to handle that yet.

● ipfs-init.service - IPFS Initializer
     Loaded: loaded (/nix/store/z6bg2vgg8q4qp3j72bw3ykjscmygcpd0-unit-ipfs-init.service/ipfs-init.service; linked; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2020-05-08 13:51:18 CEST; 7min ago
    Process: 19520 ExecStart=/nix/store/hgw4ll6kfpw8cwdxs0flcyd22mars268-unit-script-ipfs-init-start (code=exited, status=1/FAILURE)
   Main PID: 19520 (code=exited, status=1/FAILURE)
         IP: 0B in, 0B out
        CPU: 15ms

May 08 13:51:18 gaia systemd[1]: Starting IPFS Initializer...
May 08 13:51:18 gaia hgw4ll6kfpw8cwdxs0flcyd22mars268-unit-script-ipfs-init-start[19521]: Error: ipfs repo needs migration
May 08 13:51:18 gaia systemd[1]: ipfs-init.service: Main process exited, code=exited, status=1/FAILURE
May 08 13:51:18 gaia systemd[1]: ipfs-init.service: Failed with result 'exit-code'.
May 08 13:51:18 gaia systemd[1]: Failed to start IPFS Initializer.
May 08 13:51:19 gaia systemd[1]: ipfs-init.service: Start request repeated too quickly.
May 08 13:51:19 gaia systemd[1]: ipfs-init.service: Failed with result 'exit-code'.
May 08 13:51:19 gaia systemd[1]: Failed to start IPFS Initializer.

The 0.5.0 release note say the following:

Repo Migrations

IPFS uses repo migrations to make structural changes to the "repo" (the config, data storage, etc.) on upgrade.

This release includes two very simple repo migrations: a config migration to ensure that the config contains working bootstrap nodes and a keystore migration to base32 encode all key filenames.

In general, migrations should not require significant manual intervention. However, you should be aware of migrations and plan for them.

    If you update go-ipfs with ipfs update, ipfs update will run the migration for you. Note: ipfs update will refuse to run the migrations while ipfs itself is running.
    If you start the ipfs daemon with ipfs daemon --migrate, ipfs will migrate your repo for you on start.

Otherwise, if you want more control over the repo migration process, you can manually install and run the repo migration tool.

Also it seems somewhat tricky:

$ ipfs daemon --migrate
Initializing daemon...
go-ipfs version: 0.5.0
Repo version: 9
System version: amd64/linux
Golang version: go1.14.2
Found outdated fs-repo, migrations need to be run.
  => Looking for suitable fs-repo-migrations binary.
  => None found, downloading.
The migrations of fs-repo failed:
  no fs-repo-migration binary found for version 9: failed to check migrations version: fork/exec /tmp/go-ipfs-migrate435794195/fs-repo-migrations: no such file or directory
If you think this is a bug, please file an issue and include this whole log output.
  https://github.com/ipfs/fs-repo-migrations

Error: no fs-repo-migration binary found for version 9: failed to check migrations version: fork/exec /tmp/go-ipfs-migrate435794195/fs-repo-migrations: no such file or directory

The ipfs-migrator package is also not quite intuitive:

$ ipfs-update 
NAME:
   ipfs-update - update ipfs

USAGE:
   ipfs-update [global options] command [command options] [arguments...]
   
VERSION:
   0.1.0
   
COMMANDS:
   versions	print out all available versions
   version	print out currently installed version
   install	install a version of ipfs
   stash	stashes copy of currently installed ipfs binary
   revert	revert to previously installed version of ipfs
   fetch	fetch a given (default: latest) version of ipfs
   help, h	Shows a list of commands or help for one command
   
GLOBAL OPTIONS:
   --verbose		print verbose output
   --help, -h		show help
   --version, -v	print the version

How did y'all solve that?

@priegger
Copy link
Contributor

priegger commented May 8, 2020

What I did so far:

  • Bump ipfs-migrator to the latest released version (1.5.1): priegger@ec70d7c
  • Install it for the ipfs user on the system:
users.users."${config.services.ipfs.user}".packages = let
    rev = "ec70d7cbc3c58c8db8bc45a810968b90a5c4ccc7";
    sha256 = "1qfbjzbdz2idiin8z34gdmac3w65r2641fzgh69q068417smcd1n";
    nixpkgs = import (
      pkgs.fetchzip {
        url = "https://github.com/priegger/nixpkgs/archive/${rev}.tar.gz";
        inherit sha256;
      }) {};
  in
  [ nixpkgs.ipfs-migrator ];
  • Migrate my repo (in the default path) using
sudo -u ipfs env IPFS_PATH=/var/lib/ipfs /nix/store/b1h2fch7s0xc2mh93ks9zi0rx728zx3y-ipfs-migrator-1.5.1/bin/fs-repo-migrations

I can create a PR but I'm not sure if further steps (like documentation, ipfs service adjustments, genaming of the package to fs-repo-migrations) should be done, too.

@Mic92
Copy link
Contributor

Mic92 commented May 8, 2020

To be honest I have used ipfs not before and just made sure our test was running. Should we revert this until the upgrade path becomes clear?

@priegger
Copy link
Contributor

priegger commented May 8, 2020

I created #87265 which updates ipfs-migrator to the latest version.

@mweinelt
Copy link
Member

mweinelt commented May 8, 2020

That did the trick. Thanks!

@Mic92
Copy link
Contributor

Mic92 commented May 9, 2020

Do you think we can add this migrator to our NixOS module somehow? Or should we mention this in the release notes?

@priegger
Copy link
Contributor

priegger commented May 11, 2020

I don't have any background in IPFS or IPFS on NixOS, I just played around with it a bit in the last weeks.

From what I got from the nixpkgs git log and issues and the IPFS Migration Tool README:

  • auto migrate was disabled in the IPFS Service in NixOS about 3 years ago.
  • There seems to have version updated in the NixOS IPFS package during that 3 years. I did not find anything about that in the release notes.

Also there's a ticket that the IPFS test seems to be broken (#74000) which might be nice to have for this kind of feature.

@Mic92
Copy link
Contributor

Mic92 commented May 11, 2020

I have fixed the tests in this pull requests actually.

@priegger
Copy link
Contributor

priegger commented May 11, 2020

Ah, I missed that. Thanks!

@Dietr1ch
Copy link
Contributor

Dietr1ch commented May 13, 2020

I think I might be in a state close to @Mic92

I just started playing with IPFS, and after my last update it stopped starting saying that I need to migrate.

I just tried to do it manually (running as ipfs), but got some errors when doing it.


It seems that the version I have installed right now should be able to do the migration, thanks to #87265 I guess. The -to <target_version> flag defaults to 9, which is the highest version it supports (It doesn't even support --version).

However, when it runs it fails to find a repo,

sudo su ipfs -s (which bash) -c (which fs-repo-migrations)
ipfs migration:  stat /var/lib/ipfs/.ipfs: no such file or directory

From the --help it doesn't seem I can even point it to somewhere else and there's no manual for the tool either.

According to /var/lib/ipfs/version I'm on version 7, but upgrading to 8 didn't worked either.
I seem to be missing the repo,

sudo ls -lsa /var/lib/ipfs/
total 64
4 drwxr-xr-x   5 ipfs ipfs  4096 May 12 23:02 .
4 drwxr-xr-x  25 root root  4096 May  9 15:15 ..
24 drwxr-xr-x 936 ipfs ipfs 20480 May 12 22:32 blocks
8 -rw-rw----   1 ipfs ipfs  5403 May 12 22:32 config
8 -rw-------   1 ipfs ipfs  5405 May 12 22:32 config-pre-local-discovery-440372136
4 drwxr-xr-x   2 ipfs ipfs  4096 May 12 22:32 datastore
4 -rw-------   1 ipfs ipfs   190 May  9 15:15 datastore_spec
4 drwx------   2 ipfs ipfs  4096 May  9 15:15 keystore
4 -rw-r--r--   1 ipfs ipfs     2 May  9 15:15 version

UPDATE: After adding ipfs-migrator to environment.systemPackages I was able to upgrade my (non-existing?) repo with,

sudo su ipfs -s (which bash) -c "ipfs daemon"

I just killed the daemon after is succeeded migrating and restarted ipfs-initand ipfs, which started normally.

sudo systemctl restart ipfs-init
sudo systemctl restart ipfs
@priegger
Copy link
Contributor

priegger commented May 13, 2020

You can also override the IPFS_PATH for fs-repo-migrations, see #86242 (comment).

@Dietr1ch
Copy link
Contributor

Dietr1ch commented May 13, 2020

Oh, thanks for the tip. However I think that /var/lib/ipfs was already the default as that's where I saw that I was on v7 and after the upgrade I got to v9.

I think that the issue is that by running the ipfs service we should just have the ipfs-migrator installed just in case we need an upgrade.

@priegger
Copy link
Contributor

priegger commented May 13, 2020

I think " ${HOME}/.ipfs" is the default for IPFS_PATH, but the NixOS Module uses IPFS_PATH=HOME. So in your error message, the additional ".ipfs" was the problem.

But yes, making ipfs-migrator available might be the right thing to do as long as we make sure, we don't destroy other people's fs repos by auto-upgrading, make auto-upgrade non-default or at least allow people to opt-out. That's the tricky part, I think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

9 participants
You can’t perform that action at this time.