Implement basic filestore 'no-copy' functionality #3629

whyrusleeping · 2017-01-24T18:38:48Z

As per this design doc: https://gist.github.com/whyrusleeping/5565651011b49ddc9ddeec9ffd169050

License: MIT
Signed-off-by: Jeromy why@ipfs.io

kevina · 2017-01-24T23:48:02Z

filestore/filestore.go

+	if has {
+		return nil
+	}
+
 	switch b := b.(type) {


@whyrusleeping I don't know what you have in mind for the filestore. But I very specially designed my filestore so that writes go though. This way if a file is moved it can simply be readded and any invalid blocks are automatically fixed.

That becomes problematic, What then happens if i add dataset A, which contains some subset C. Then later, i add dataset B that also contains some subset C. Then i change my mind and remove dataset B. By your description, the original dataset references would now be lost, with no indication to the user that this has happened.

I think this scenario is worse UX than the reverse (deleting A causing bad refs in B). In either case, repairs are going to have to be manual, and some things will get sticky.

cc @jbenet for his thoughts on this scenario.

kevina · 2017-01-25T18:42:42Z

@whyrusleeping

From an inline comment above:

I don't know what you have in mind for the filestore. But I very specially designed my filestore so that writes go though. This way if a file is moved it can simply be readded and any invalid blocks are automatically fixed.

That becomes problematic, What then happens if i add dataset A, which contains some subset C. Then later, i add dataset B that also contains some subset C. Then i change my mind and remove dataset B. By your description, the original dataset references would now be lost, with no indication to the user that this has happened.

Yes this was originally a problem. In my implementation I fixed this. See ipfs-filestore#12 and ipfs-filestore#23. However, that solution also requires writes to go through.

I think this scenario is worse UX than the reverse (deleting A causing bad refs in B). In either case, repairs are going to have to be manual, and some things will get sticky.

Not if my solution in ipfs-filestore#12 is implemented.

cc @jbenet for his thoughts on this scenario.

kevina · 2017-01-25T22:08:36Z

Yes this was originally a problem. In my implementation I fixed this. See ipfs-filestore#12 and ipfs-filestore#23. However, that solution also requires writes to go through.

Another option is to only replace the block if it is invalid. I considered that, but decided that it was better to support allowing multiple files to point to the same block.

kevina

One major thing I noticed. A few other minor things that I comment on later.

kevina · 2017-01-28T23:35:08Z

filestore/fsrefstore.go

+	out, err := f.readDataObj(&dobj)
+	if err != nil {
+		return nil, err
+	}


I strongly recommend you verify the block.

Ah, yeah. NewBlockWithCid only does that check if u.Debug is true. good catch

kevina · 2017-01-30T18:53:25Z

I see that "--nocopy" is allowed when the daemon is online without any sort of additional checks. If the blocks are ever not verified this could become a security problem. Even without the security problems this could lead to strange results if files are added on another machine with an identical path.

kevina · 2017-01-30T18:55:41Z

For this to be usable outside of ipfs-pack I would strongly recommend that references be allowed "outside the root handler" by simply requiring all paths to be absolute.

kevina · 2017-01-30T19:07:47Z

filestore/fsrefstore.go

+var FilestorePrefix = ds.NewKey("filestore")
+
+type FileManager struct {
+	ds   ds.Batching


Note. In my early implementations I wrote the filestore on top of the datastore also. However, I changed this to directly use the leveldb as I found little value in the abstraction and extra layers. What actually prompted it was ipfs-filestore#10. In particular querying the datastore was very very slow, however I then latter find it very useful to use the features of the LevelDB in order to provide safe verification of the filestore while the deamon was running by using the LevelDB snapshot feature. With my recent improvements to the datastore query the performance may no longer be a problem, but I still see little value in the extra indirection.

kevina · 2017-02-02T01:18:05Z

@whyrusleeping concerning replacing blocks. How about for now we make it an option? There needs to be a way to fix broken blocks due to files moving. One option would be to remove the block and then readd it, but pinning makes that a more complicated operation.

As far as allowing multiple blocks for a hash (ipfs-filestore#12 and ipfs-filestore#23), I agree that we should aim for that but to start with it might be a bit much to review.

kevina · 2017-02-02T19:08:56Z

filestore/fsrefstore.go

+		return nil, err
+	}
+
+	out := make(chan *cid.Cid)


I would suggest you use a buffered channel. Use dsq.KeysOnlyBufSize.

kevina · 2017-02-02T19:09:41Z

filestore/filestore.go

+		return nil, err
+	}
+
+	out := make(chan *cid.Cid)


I would suggest you use a buffered channel. Use dsq.KeysOnlyBufSize.

jefft0 · 2017-02-20T09:12:39Z

Is this branch ready for me to test on my 30 TB of webcam videos or should I wait?

whyrusleeping · 2017-03-02T00:48:07Z

@jefft0 Its almost ready, should be merged in a couple days.

whyrusleeping · 2017-03-02T00:49:53Z

Rebased, waiting on a 👍 from @Kubuxu and CI

kevina · 2017-03-02T01:07:49Z

At @jefft0 I am really not sure how usable this code is going to be for you as it does not allow for files "outside the root handler". For example if you directory layout is:

/home/jeff0/.ipfs
/aux/media

and you want to share files in /aux/media you won't be able to and will get an error.

kevina · 2017-03-02T01:10:01Z

@whyrusleeping I would like this comment, to at least be addressed:

I see that "--nocopy" is allowed when the daemon is online without any sort of additional checks. If the blocks are ever not verified this could become a security problem. Even without the security problems this could lead to strange results if files are added on another machine with an identical path.

whyrusleeping · 2017-03-02T01:23:40Z

@kevina @jefft0 In this case, simply set your IPFS_PATH to be in (given the example) /aux/ipfs or similar, We're pushing for git style security on file references. They need to be within the scope of the ipfs repository.

@kevina Re the comment, blocks are checked now (thanks for pointing that out), and this feature should not be used with client and daemon on separate machines. We can probably implement a check of some sort for this... any ideas on how to do that off the top of your head?

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

jefft0 · 2017-03-06T16:50:44Z

I'm confused. I have several webcam videos totalling 60 GB. With a fresh installation of go-ipfs (master branch), I did ipfs init then ipfs add --nocopy --raw-leaves for each file. But now ~/.ipfs is 60 GB. Is --nocopy working yet in master?

Kubuxu · 2017-03-06T17:03:06Z

@whyrusleeping anything on this?

whyrusleeping · 2017-03-06T17:55:04Z

Ah, theres a check i forgot to add, erroring out if --nocopy is passed without enabling the filestore.

@jefft0 take a look at "How to enable" here: #3397 (comment)

jefft0 · 2017-03-06T18:25:27Z

@whyrusleeping Much better, thanks! Now ~/,ipfs is only 68 MB. I'll try adding a lot more files. Where do you want me to report my results?

whyrusleeping · 2017-03-06T18:27:23Z

@jefft0 You can report them here, or open a new issue to discuss. Either way :)

jefft0 · 2017-03-06T18:31:00Z

@whyrusleeping I want to keep my main repo in ~/.ipfs, so instead of changing IPSF_PATH I put a symbolic link to my external drive in ~/.ipfs and do ipfs add --nocopy with that path. This works. It would also let me use multiple external drives at different mount points. Do you see any downside to this approach?

whyrusleeping · 2017-03-06T18:33:35Z

@jefft0 hrm... unsure about the implications of this. I'll have to think about the security model of using/allowing symlinks like that.

My first thought though is that it should be fine

whyrusleeping · 2017-03-06T18:34:22Z

@jefft0 you can also put the symbolic link in just ~/, it doesnt have to be IN the ipfs dir, just in its parent (like the .git folder for git repos)

whoizit · 2017-03-06T20:29:40Z

every time I get

$ ps aux | grep ipfs
atommixz  3109 36.2  2.5 879840 207656 ?       Ssl  23:20   1:24 /usr/bin/ipfs daemon --manage-fdlimit=true

$ LANG=C type ipfs_add
ipfs_add is aliased to `ipfs add --nocopy --raw-leaves --recursive --progress'

$ ipfs_add download/torrents
added QmcGUHBF2XT8dTblablablazYsNQch6ECGx torrents/file.ext
 3.24 GB / 61.22 GB [==>---------------------------------------------------]   5.30% 15m33s23:23:30.659 ERROR commands/h: open /home/username/.ipfs/blocks/HE/put-789802368: too many open files client.go:247
Error: open /home/username/.ipfs/blocks/HE/put-789802368: too many open files

whyrusleeping · 2017-03-06T20:51:30Z

@atommixz Wanna open a new issue for this?

whoizit · 2017-03-06T21:37:43Z

This helps to me:

$ systemctl --user edit --full ipfs 
[Service]
LimitNOFILE=999999

This is don't make sence to me ipfs daemon --manage-fdlimit=true

whoizit · 2017-03-06T23:58:47Z

I'm trying to download many images from filestore. Everytime it stops at the same place. Last commit on two pc.

$ ipfs get Qmblablabla
Saving file(s) to Qmblablabla
 117.00 MB / 2.36 GB [==>------------------------------------------------------]   4.84% 4s
Error: expected protobuf dag node

...
23:49:27.028 ERROR    bitswap: couldnt open sender again after SendMsg(<peer.ID PRB9SN>) failed: dial attempt failed: <peer.ID Rj75eS> --> <peer.ID PRB9SN> dial attempt failed: i/o timeout wantmanager.go:233
23:50:03.062 ERROR    bitswap: couldnt open sender again after SendMsg(<peer.ID aoqJoJ>) failed: dial attempt failed: <peer.ID Rj75eS> --> <peer.ID aoqJoJ> dial attempt failed: i/o timeout wantmanager.go:233
23:51:15.626 ERROR commands/h: err: expected protobuf dag node handler.go:288
23:52:34.252 ERROR commands/h: err: expected protobuf dag node handler.go:288
23:52:59.859 ERROR commands/h: err: expected protobuf dag node handler.go:288

Kubuxu · 2017-03-07T00:05:41Z

Can you report it in separate issue. It is much easier to track this way.

whoizit · 2017-03-07T00:08:36Z

whyrusleeping said "You can report them here, or open a new issue to discuss. Either way :)"
so I do, because it's about filestore.

Kubuxu · 2017-03-07T00:10:33Z

I think he meant results as it, it worked, performance, small problems.

whyrusleeping · 2017-03-07T00:31:59Z

Yeah, Keeping perf and general 'how it went' things here is fine, but actual problems need to have their own issues so we can address them

jefft0 · 2017-03-07T19:55:23Z

I had a problem with too many open files and opened issue #3763. I don't know if it's related to the issue @Kubuxu has.

whyrusleeping added status/in-progress In progress labels Jan 24, 2017

flyingzumwalt mentioned this pull request Jan 24, 2017

Story: Register a filesystem Directory in IPFS without Duplicating Content ipfs-inactive/archives#92

Closed

whyrusleeping force-pushed the feat/filestore0 branch from 6176525 to f49f52e Compare January 24, 2017 20:34

kevina reviewed Jan 24, 2017

View reviewed changes

kevina suggested changes Jan 30, 2017

View reviewed changes

kevina reviewed Jan 30, 2017

View reviewed changes

kevina mentioned this pull request Feb 2, 2017

Needed Filestore Command For Basic Implementation #3646

Open

kevina suggested changes Feb 2, 2017

View reviewed changes

whyrusleeping force-pushed the feat/filestore0 branch 3 times, most recently from d9c40b5 to 299f5c9 Compare February 12, 2017 20:52

whyrusleeping modified the milestones: Filestore implementation, Ipfs 0.4.7 Feb 15, 2017

This was referenced Feb 20, 2017

filestore-db size ipfs-filestore/go-ipfs#28

Open

When is this expected to ship? ipfs-filestore/go-ipfs#27

Closed

whyrusleeping force-pushed the feat/filestore0 branch from 299f5c9 to 1a94991 Compare March 2, 2017 00:49

whyrusleeping requested a review from Kubuxu March 2, 2017 00:49

whyrusleeping and others added 6 commits March 6, 2017 00:37

add test for deletes

c18ec98

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

validate data read from fsrefstore

8fce0c4

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

Move block verification into readDataObj.

459a4b3

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

Refactor.

8e1a03d

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

Use buffered channels in AllKeysChan to increase performance.

8082af7

License: MIT Signed-off-by: Kevin Atkinson <k@kevina.org>

Address CR feedback

9ec4c25

License: MIT Signed-off-by: Jeromy <jeromyj@gmail.com>

whyrusleeping force-pushed the feat/filestore0 branch from 1aaf351 to 9ec4c25 Compare March 6, 2017 08:37

Kubuxu approved these changes Mar 6, 2017

View reviewed changes

whyrusleeping merged commit ea57c69 into master Mar 6, 2017

whyrusleeping removed the status/in-progress In progress label Mar 6, 2017

whyrusleeping mentioned this pull request Mar 6, 2017

WIP: Basic Filestore implementation. #3368

Closed

3 tasks

whyrusleeping deleted the feat/filestore0 branch March 6, 2017 20:51

kevina mentioned this pull request Sep 21, 2017

ipfs add -r --nocopy Error: Failed to get block for zb2rh... The system cannot find the path specified. #4260

Open

Implement basic filestore 'no-copy' functionality #3629

Implement basic filestore 'no-copy' functionality #3629

Conversation

whyrusleeping commented Jan 24, 2017

kevina Jan 24, 2017

Choose a reason for hiding this comment

whyrusleeping Jan 25, 2017

Choose a reason for hiding this comment

kevina commented Jan 25, 2017 • edited

kevina commented Jan 25, 2017

kevina left a comment

Choose a reason for hiding this comment

kevina Jan 28, 2017

Choose a reason for hiding this comment

whyrusleeping Jan 30, 2017

Choose a reason for hiding this comment

kevina commented Jan 30, 2017 • edited

kevina commented Jan 30, 2017

kevina Jan 30, 2017

Choose a reason for hiding this comment

kevina commented Feb 2, 2017

kevina Feb 2, 2017

Choose a reason for hiding this comment

kevina Feb 2, 2017

Choose a reason for hiding this comment

jefft0 commented Feb 20, 2017

whyrusleeping commented Mar 2, 2017

whyrusleeping commented Mar 2, 2017

kevina commented Mar 2, 2017

kevina commented Mar 2, 2017

whyrusleeping commented Mar 2, 2017

jefft0 commented Mar 6, 2017

Kubuxu commented Mar 6, 2017

whyrusleeping commented Mar 6, 2017

jefft0 commented Mar 6, 2017

whyrusleeping commented Mar 6, 2017

jefft0 commented Mar 6, 2017

whyrusleeping commented Mar 6, 2017

whyrusleeping commented Mar 6, 2017

whoizit commented Mar 6, 2017 • edited

whyrusleeping commented Mar 6, 2017

whoizit commented Mar 6, 2017

whoizit commented Mar 6, 2017 • edited

Kubuxu commented Mar 7, 2017

whoizit commented Mar 7, 2017 • edited

Kubuxu commented Mar 7, 2017

whyrusleeping commented Mar 7, 2017

jefft0 commented Mar 7, 2017

kevina commented Jan 25, 2017 •

edited

kevina commented Jan 30, 2017 •

edited

whoizit commented Mar 6, 2017 •

edited

whoizit commented Mar 6, 2017 •

edited

whoizit commented Mar 7, 2017 •

edited