IPFS: cannot change owner and permissions of ‘/ipns’

[davidak]

Issue description

nixos-rebuild switch --upgrade ended with:

Job for ipfs-init.service failed because the control process exited with error code.

warning: the following units failed: ipfs-init.service

● ipfs-init.service - IPFS Initializer
   Loaded: loaded (/nix/store/v2m1iki7x7jsr5ahmw3zz1fhp5hj6bw2-unit-ipfs-init.service/ipfs-init.service; linked; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2018-06-27 21:46:52 CEST; 1s ago
  Process: 24718 ExecStartPre=/nix/store/qmfd1sr96f58lvssjs49j6pz4xd028fz-unit-script/bin/ipfs-init-pre-start (code=exited, status=1/FAILURE)
 Main PID: 928 (code=exited, status=0/SUCCESS)
      CPU: 11ms

Jun 27 21:46:52 nas systemd[1]: Starting IPFS Initializer...
Jun 27 21:46:52 nas ipfs-init-pre-start[24718]: install: cannot change owner and permissions of ‘/ipns’: Transport endpoint is not connected
Jun 27 21:46:52 nas systemd[1]: ipfs-init.service: Control process exited, code=exited status=1
Jun 27 21:46:52 nas systemd[1]: ipfs-init.service: Failed with result 'exit-code'.
Jun 27 21:46:52 nas systemd[1]: Failed to start IPFS Initializer.
Jun 27 21:46:52 nas systemd[1]: ipfs-init.service: Consumed 11ms CPU time
warning: error(s) occurred while switching to the new configuration

[root@nas:~]# ll /
ls: cannot access '/ipns': Transport endpoint is not connected
total 68
...
d--x--x--x   1 root    root      0 Jun 27 22:00 ipfs
d?????????   ? ?       ?         ?            ? ipns

(ipfs.service was sucessfully restarted and runs)

I stopped ipfs.service and tried to delete /ipns

[root@nas:~]# rm -rf /ipns 
rm: cannot remove '/ipns': Is a directory

[root@nas:~]# rmdir /ipns 
rmdir: failed to remove '/ipns': Device or resource busy

umount -l /ipns

that worked in the end. permissions back normal:

[root@nas:~]# ll /
total 76
drwxr-xr-x   2 root    root   4096 Jun 27 21:46 bin
drwx------   3 root    root   4096 Dec 13  2017 boot
drwxr-xr-x   1 root    root     74 Jun 19  2017 data
drwxr-xr-x  19 root    root   3760 May  8 06:32 dev
drwxr-xr-x  20 root    root   4096 Jun 27 21:46 etc
drwxr-xr-x   3 root    root   4096 Jun 18  2017 home
drwxr-xr-x   2 ipfs    ipfs   4096 Nov 26  2017 ipfs
drwxr-xr-x   2 ipfs    ipfs   4096 Nov 26  2017 ipns
...

I can start IPFS again:

systemctl start ipfs-init.service
systemctl start ipfs.service

maybe related to ipfs/kubo#5111

had it on all (2) systems where i have ipfs running.

cc @elitak @mguentner

Steps to reproduce

1. have ipfs service configured
2. nixos-rebuild switch --upgrade

Technical details

• system: "x86_64-linux"
• host os: Linux 4.14.38, NixOS, 18.03.132768.94d80eb7247 (Impala)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.0.4
• channels(root): "nixos-18.03.132768.94d80eb7247"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

[elitak]

I see 2 fixes for this:

1. Move or otherwise make conditional the install lines in pre-start
2. Change the systemd unit dependencies to have ipfs stop before ipfs-init runs, the start up again after during a rebuild.

I'll have to review the docs on systemd unit deps.

[elitak]

I believe ipfs is already supposed to be dead when this runs, so I just added some umount -l commands before the install ones to hopefully avoid the zombie fuse mounts interfering with things.

[stale]

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

1. Search for maintainers and people that previously touched the related code and @ mention them in a comment.
2. Ask on the NixOS Discourse.
3. Ask on the #nixos channel on irc.freenode.net.

[matthewbauer]

Does this still happen in ipfs 0.5.1? I am hoping it was an upstream bug that was fixed at some point.

[stale]

I marked this as stale due to inactivity. → More info