ipfs not starting

[matthiasbeyer]

Issue description

Okt 31 12:56:30 lucy systemd[1]: Starting IPFS Daemon...
Okt 31 12:56:30 lucy ipfs-pre-start[6162]: Lockfiles have been removed.
Okt 31 12:56:30 lucy ipfs-pre-start[6162]: Error: leveldb: manifest corrupted (field 'comparer'): missing [file=MANIFEST-000321]
Okt 31 12:56:30 lucy systemd[1]: ipfs.service: Control process exited, code=exited status=1
Okt 31 12:56:30 lucy systemd[1]: Failed to start IPFS Daemon.
Okt 31 12:56:30 lucy systemd[1]: ipfs.service: Unit entered failed state.
Okt 31 12:56:30 lucy systemd[1]: ipfs.service: Failed with result 'exit-code'.
Okt 31 12:56:31 lucy systemd[1]: ipfs.service: Service hold-off time over, scheduling restart.
Okt 31 12:56:31 lucy systemd[1]: Stopped IPFS Daemon.
Okt 31 12:56:31 lucy systemd[1]: ipfs.service: Start request repeated too quickly.
Okt 31 12:56:31 lucy systemd[1]: Failed to start IPFS Daemon.
Okt 31 12:56:31 lucy systemd[1]: ipfs.service: Unit entered failed state.
Okt 31 12:56:31 lucy systemd[1]: ipfs.service: Failed with result 'exit-code'.

Steps to reproduce

Not sure.

Technical details

• System: 18.03pre118631.66b63d2f5a (Impala)
• Nix version: nix-env (Nix) 1.11.15
• Nixpkgs version: "18.03pre118631.66b63d2f5a"
• Sandboxing enabled: build-use-sandbox = true

[globin]

cc @mguentner

[matthiasbeyer]

Maybe this is not a nixpkgs issue but an IPFS issue @whyrusleeping @jbenet (pinging some IPFS people I know by name)?

[whyrusleeping]

Hey! Thats very odd, it looks like the leveldb instance in the ipfs repo is somehow corrupted, I'm not exactly sure what to do to recover from this. ping @kevina and @magik6k for input

[magik6k]

Can you try running ls -l ~/.ipfs/datastore/ and pasting the output here?

[matthiasbeyer]

$ ls -l /var/lib/ipfs/.ipfs
insgesamt 40
drwxr-xr-x 1026 ipfs ipfs 20480  7. Okt 16:21 blocks
-rw-rw----    1 ipfs ipfs  4894  2. Sep 16:19 config
drwxr-xr-x    2 ipfs ipfs  4096 16. Okt 23:34 datastore
drwx------    2 ipfs ipfs  4096  1. Apr 2017  keystore
-rw-r--r--    1 ipfs ipfs     2  1. Apr 2017  version

[magik6k]

This looks fine, can you send ls of the datastore subdirectory?

[matthiasbeyer]

$ ls -l /var/lib/ipfs/.ipfs/datastore
insgesamt 476
-rw-r--r-- 1 ipfs ipfs 204859 16. Okt 23:34 000356.log
-rw-r--r-- 1 ipfs ipfs 163413 13. Okt 18:03 000358.ldb
-rw-r--r-- 1 ipfs ipfs     16 13. Okt 15:40 CURRENT
-rw-r--r-- 1 ipfs ipfs  94452 16. Okt 23:34 LOG
-rw-r--r-- 1 ipfs ipfs   1183 13. Okt 18:03 MANIFEST-000357

[whyrusleeping]

Yeah, the error message in the issue says the manifest file is actually corrupted. Not sure how to recover from this, maybe some leveldb utility could help

[matthiasbeyer]

I have to add, that the same errors happens on two independent hosts. So this is not an issue that only affects one of my ipfs repositories!

[matthiasbeyer]

After running the repo-migration tool ipfs-5-to-6 (the repo was at version 6 at this time already, as far as I could tell) and chown ipfs:ipfs the config and datastore_spec files, the daemon successfully started.

I guess this can be closed now.

[matthiasbeyer]

I am sorry, I have to reopen this. On my second host the above fix does not work.

It still complains that the MANIFEST-0000321 file is corrupted... I tried to cat that file - it is empty!

---

For testing, I moved it out of the datastore (to /tmp) - then it complains that MANIFEST-000000 does not exist. So that isn't a fix apparently.

---

ipfs version 0.4.11

[whyrusleeping]

@matthiasbeyer What are you doing to reproduce this issue?

[matthiasbeyer]

systemctl start ipfs

[matthiasbeyer]

Cannot reproduce anymore.

[matthiasbeyer]

aaaaand can reproduce.

● ipfs.service - IPFS Daemon
   Loaded: loaded (/nix/store/3cbnw5pwyzz3817x4cjhk54f7lzjx0kx-unit-ipfs.service/ipfs.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2018-10-18 22:27:12 CEST; 2min 19s ago
  Process: 30303 ExecStartPre=/nix/store/gwgjpzjxay7sajppass4jjcwfiq1bfvy-unit-script-ipfs-pre-start (code=exited, status=1/FAILURE)

Okt 18 22:27:12 lucy systemd[1]: ipfs.service: Service RestartSec=1s expired, scheduling restart.
Okt 18 22:27:12 lucy systemd[1]: ipfs.service: Scheduled restart job, restart counter is at 5.
Okt 18 22:27:12 lucy systemd[1]: Stopped IPFS Daemon.
Okt 18 22:27:12 lucy systemd[1]: ipfs.service: Start request repeated too quickly.
Okt 18 22:27:12 lucy systemd[1]: ipfs.service: Failed with result 'exit-code'.
Okt 18 22:27:12 lucy systemd[1]: Failed to start IPFS Daemon.

after nixos-rebuild switch with the generation from ~2hours ago.

ipfs version 0.4.17

[matthiasbeyer]

Some more error output:

Okt 19 14:36:42 lucy gwgjpzjxay7sajppass4jjcwfiq1bfvy-unit-script-ipfs-pre-start[28697]: Lockfiles have been removed.
Okt 19 14:36:42 lucy gwgjpzjxay7sajppass4jjcwfiq1bfvy-unit-script-ipfs-pre-start[28697]: Error: ipfs repo needs migration
Okt 19 14:36:42 lucy systemd[1]: ipfs.service: Control process exited, code=exited status=1
Okt 19 14:36:42 lucy systemd[1]: ipfs.service: Failed with result 'exit-code'.
Okt 19 14:36:42 lucy systemd[1]: Failed to start IPFS Daemon.
-- Subject: Unit ipfs.service has failed
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit ipfs.service has failed.
-- 
-- The result is RESULT.

from journalctl -xe

How to migrate the repository?

[magik6k]

Ipfs should auto-run migration with ipfs daemon --migrate

[matthiasbeyer]

Hmmm...

$ sudo -u ipfs ipfs daemon --migrate
[sudo] Passwort für m: 
Initializing daemon...
Successfully raised file descriptor limit to 2048.
Found outdated fs-repo, migrations need to be run.
  => Looking for suitable fs-repo-migrations binary.
  => None found, downloading.
The migrations of fs-repo failed:
  no fs-repo-migration binary found for version 7: failed to check migrations version: fork/exec /tmp/go-ipfs-migrate591262889/fs-repo-migrations: no such file or directory
If you think this is a bug, please file an issue and include this whole log output.
  https://github.com/ipfs/fs-repo-migrations
Error: no fs-repo-migration binary found for version 7: failed to check migrations version: fork/exec /tmp/go-ipfs-migrate591262889/fs-repo-migrations: no such file or directory

I guess NixOS should do the migration automatically for me...

[magik6k]

Can you try running ldd on migrations binary? (/tmp/go-ipfs-migrate591262889/fs-repo-migrations, assuming it's there)

If it's not there, can you try getting it manually from https://dist.ipfs.io/#fs-repo-migrations?

[matthiasbeyer]

Nope, No such file or directory. The binary from the website wont work as this is NixOS. I used the migration tools with nix-shell and it worked:

$ nix-shell -p ipfs-migrator
[nix-shell:~]$ sudo -u ipfs ipfs-6-to-7 -path /var/lib/ipfs/.ipfs
[nix-shell:~]$ sudo systemctl start ipfs # works

Thanks for the help.