bug: export IPFS_PATH to file server throws error during ipfs init

[kpeterstech]

I made this post on the discussion page at ipfs.io. I was told this is most likely a bug and to post this over here.

Version information:

ipfs version --all

go-ipfs version: 0.4.14-
Repo version: 6
System version: amd64/linux
Golang version: go1.10

Type: Bug

Description:

Desktop OS: Ubuntu 16.04 on physical SSD
File server OS: Debian 9.4 on VM in Proxmox 5.1-46

I am trying to find a way to make IPFS more practical for me. My main OS is ubuntu 16.04 and it is on a 120 gb SSD. In my home network I have a server that I use for storage that has 21 TB on it. I would like to configure IPFS to use my network storage to save the files I add with ipfs add <file> so I don't fill up my SSD. Reading into the documentation it says the following towards the bottom after running ipfs --help from the command line.

Use 'ipfs <command> --help' to learn more about each command.
  
  ipfs uses a repository in the local file system. By default, the repo is
  located at ~/.ipfs. To change the repo location, set the $IPFS_PATH
  environment variable:
  
    export IPFS_PATH=/path/to/ipfsrepo

In zsh I have set the IPFS_PATH variable in my .zshrc file with the following entry

export IPFS_PATH=/media/network_storage/FTP/ipfs

after saving and opening up a new shell, when I run

echo $IPFS_PATH

it returns

/media/network_storage/FTP/ipfs

Now, when I go to initialize ipfs (using both ipfs init and ipfs init --profile=server) the following happens (there is no ~/.ipfs directory and the /media/network_storage/FTP/ipfs directory is empty as well)

initializing IPFS node at /media/network_storage/FTP/ipfs
generating 2048-bit RSA keypair...done
peer identity: Qma2soquUqaxZ3RmJp5QpC3GrbTtykywuhmbkcT8tW41F7
Error: open /media/network_storage/FTP/ipfs/blocks/_README: permission denied

when I run tree on my $IPFS_PATH I get the following:

.
├── blocks
│   ├── _README
│   └── SHARDING
├── config
├── datastore_spec
└── version

if I remove the $IPFS_PATH from my .zshrc and start a new shell and run ipfs init there is no issue initializing ipfs. The ipfs folder is created at "~/.ipfs" and it contains the following

.
├── blocks
│   ├── 2H
│   │   └── CIQN5PVU4ECEKNEVXWRYKCPTVMIPHP5AWWKOCGCPXVNMF7ZWU6UG2HI.data
│   ├── 7J
│   │   └── CIQKKLBWAIBQZOIS5X7E32LQAL6236OUKZTMHPQSFIXPWXNZHQOV7JQ.data
│   ├── 7R
│   │   └── CIQIVZGLOUFY5L4TEYD5WLSCRDDEAW2TNVZSH3OBM5UKNOREHYSY7RA.data
│   ├── B5
│   │   └── CIQDOZU3EAGXWK3PLVFOFOZOAE5USX3XM6I5CSHSQGTML2BAGN7MB5I.data
│   ├── IL
│   │   └── CIQJFGRQHQ45VCQLM7AJNF2GF5UHUAGGHC6LLAH6VYDEKLQMD4QLILY.data
│   ├── JN
│   │   └── CIQPHMHGQLLZXC32FQQW2YVM4KGFORVFJAQYY55VK3WJGLZ2MS4RJNQ.data
│   ├── L5
│   │   └── CIQBIQXZ4NWWDXUSIYSCX7RE6EBXHMGENZNMUDEMGNKMGT2K6LLUL5Y.data
│   ├── LG
│   │   └── CIQJBQD2O6K4CGJVCCTJNUP57QHR4SKHZ74OIITBBGLOMCO3ZOLWLGA.data
│   ├── N2
│   │   └── CIQDWKPBHXLJ3XVELRJZA2SYY7OGCSX6FRSIZS2VQQPVKOA2Z4VXN2I.data
│   ├── OO
│   │   └── CIQBT4N7PS5IZ5IG2ZOUGKFK27IE33WKGJNDW2TY3LSBNQ34R6OVOOQ.data
│   ├── PM
│   │   └── CIQKNNRB2NFYXUZDJ2UWNMSKYLGTKUYDRQTJCDI7JTUDFH6YOYNUPMA.data
│   ├── QV
│   │   └── CIQOHMGEIKMPYHAUTL57JSEZN64SIJ5OIHSGJG4TJSSJLGI3PBJLQVI.data
│   ├── R3
│   │   └── CIQBED3K6YA5I3QQWLJOCHWXDRK5EXZQILBCKAPEDUJENZ5B5HJ5R3A.data
│   ├── _README
│   ├── SH
│   │   └── CIQL3XIOKVDAW5KQF6NNWGFFYAHEQP63TJOVZHAEO7XZBD7KQOCSSHY.data
│   ├── SHARDING
│   └── X3
│       └── CIQFTFEEHEDF6KLBT32BFAGLXEZL4UWFNWM4LFTLMXQBCERZ6CMLX3Y.data
├── config
├── datastore
│   ├── 000002.ldb
│   ├── 000003.log
│   ├── CURRENT
│   ├── LOCK
│   ├── LOG
│   └── MANIFEST-000004
├── datastore_spec
├── keystore
└── version

The way I mount my network storage is with this line in my /etc/fstab

//192.168.1.253/storage_pool /media/network_storage cifs uid=1000,gid=1000,credentials=/home/beard/.network_pw,iocharset=utf8 0 0

It looks as if when trying to initialize IPFS over my network something happens where it is not able to create the datastore directory and the directories and files under $IPFS_PATH/blocks

when I run id I get the following output

uid=1000(beard) gid=1000(beard) groups=1000(beard),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),113(lpadmin),128(sambashare),129(docker)

When I go to /media/network_storage/FTP/ and run ls -ln I get the following output for my ipfs directory

drwxrwxrwx+ 3 1000 1000 0 Apr 11 20:03 ipfs

I have also tried to use symoblic links with no luck either.

I have never submitted and issue on github before so I hope the above is useful/correct and hasn't violated any rules. If I have made any error in submitting this issue please let me know and I will change what needs to be changed. If there is any additional information I can provide that can be of use please let me know what is needed and the quickest way to get it and I will do my best to provide it.

Thank you for your help

[magik6k]

It seems to be more of an issue with the cifs fs. Can you record an strace (strace -ff ipfs init > init.trace) and send it here? (upload it to gist or somewhere like that, it's probably going to be quite lengthy)

[kpeterstech]

I ran the command you pasted strace -ff ipfs init > init.trace but all it gave me was the following output:

initializing IPFS node at /media/network_storage/FTP/ipfs
generating 2048-bit RSA keypair...done
peer identity: QmQq3imtqKNcaQ7PyBvTFBZ4LrMtS7Ru6CC5KTyGK6de7z

So I tried redirecting with 2> and this is what I got. I hope this is what you were looking for. This is what I saw when I ran your command above. It just didn't redirect it to the init.trace file.

https://gist.github.com/kpeterstech/02ee81b9f4848617765dc7d31e29ecde

[Stebalien]

That definitely looks like a bug in samba. Possibly: https://bugzilla.samba.org/show_bug.cgi?id=10762

Basically, we're creating the file with the mode set to read-only. This will prevent future attempts to open the file for writing to fail but shouldn't prevent the initial creator from writing to it. It looks like CIFS doesn't follow this part of the Posix spec.

Could you test the following program?

package main

import (
	"bytes"
	"io/ioutil"
	"os"
)

func main() {
	inp := []byte("data\n")

	fname := os.Args[1]
	err := ioutil.WriteFile(fname, inp, 0444)
	if err != nil {
		panic(err)
	}
	outp, err := ioutil.ReadFile(fname)
	if err != nil {
		panic(err)
	}
	if !bytes.Equal(inp, outp) {
		panic("read/write round-trip didn't work")
	}
}

Invoke it as: go run the_program.go /media/network_storage/FTP/test_file.

[kpeterstech]

:~$ go version
go version go1.6.2 linux/amd64

I copied the code pasted above into a file named "the_program.go". I then ran it with the command listed above from my home directory ~/.

The first time I ran the_program.go I got the following result

panic: open /media/network_storage/FTP/test_file: permission denied

goroutine 1 [running]:
panic(0x4996c0, 0xc82008a000)
        /usr/lib/go-1.6/src/runtime/panic.go:481 +0x3e6
main.main()
        /home/beard/the_program.go:15 +0xfa
exit status 2

I looked in /media/network_storage/FTP and saw that the file test_file had been created. It was not there before I ran the_program.go and it contained no contents and only had read permission (444).

I then removed the file rm /media/network_storage/FTP/test_file and then created it myself with touch /media/network_storage/FTP/test_file and it had permissions of 644 and no contents.

I then ran the_program.go again and got no errors. Inside of test_file was the word data

[Stebalien]

Got it. This is definitely a bug in CIFS Linux kernel module. We could work around it in go-ipfs but I should warn you, storing the repo on a network storage device will give you pretty poor performance.

[kpeterstech]

That's ok. I actually figured out a workout from the thread on the dicuss.ipfs.io that is linked above. I just thought I would do my part trying to make a bug known, and to document this problem for someone else that comes along later. Thank you for taking the time to help though!

[Stebalien]

I wonder if it would be a good idea to try to detect the filesystem on init and warn about filesystems with known issues...