Add test for ipfs mount #248

chriscool · 2014-10-31T20:32:16Z

This is one more step related to issue #148 (Whole-tool test).

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

whyrusleeping · 2014-10-31T20:35:01Z

Is there a cleanup that deletes the .go-ipfs dir that these tests create?

chriscool · 2014-10-31T20:43:47Z

Yeah, test scripts are executed in a directory named "trash directory.t00X0-name.sh" which is deleted if the test script succeeds.

For example t0030-mount.sh is executed in:

trash directory.t0030-mount.sh/

You can add an "exit 1" into t0030-mount.sh and you will see.

By the way I will add a rule to ignore these test directories into the ".gitignore".

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

whyrusleeping · 2014-10-31T21:34:51Z

Youre awesome. These tests are awesome. everything LGTM

chriscool · 2014-10-31T22:06:29Z

Thanks!

This adds test_launch_ipfs_mount() and test_kill_ipfs_mount() to avoid duplicating tests to launch "ipfs mount" and to kill it. License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

chriscool · 2014-11-02T05:28:19Z

@jbenet could ou have a look at this PR?
Btw the Travis build failure looks weird.

jbenet · 2014-11-02T05:41:51Z

test/.gitignore

@@ -1 +1,2 @@
 test-results/
+trash directory.*.sh/


could we name this trash-directory... ? (replace/ /-)

The space in the directory name was added on purpose during Git development to force people to properly quote arguments which is often a problem in shell scripts.

Ahh, nice! Indeed 👍 😄

Yeah, and if we want to support Windows, it is especially important to check that it works with a path that contains spaces as programs are very often installed in "C:\Program Files\stuff".

jbenet · 2014-11-02T05:46:05Z

@chriscool yep! sorry for delay.

agreed with @whyrusleeping these tests are great and very much needed!

would be great to do a large ipfs add / cat test. i.e. adding ~100MB of data (reasonably high entropy data. could use the sourcecode itself.)
I get failure (unmounting text, see below), but only when running make (if i run the test individually, it passes. weird)

jbenet @ lorien : ~/go/src/github.com/jbenet/go-ipfs/test * chriscool-test_ipfs_mount % make
*** clean ***
rm -r test-results
*** t0010-basic-commands.sh ***
./t0010-basic-commands.sh
ok 1 - current dir is writable
ok 2 - ipfs version succeeds
ok 3 - ipfs version output looks good
ok 4 - ipfs help succeeds
ok 5 - ipfs help output looks good
# passed all 5 test(s)
1..5
*** t0020-init.sh ***
./t0020-init.sh
ok 1 - ipfs init succeeds
ok 2 - .go-ipfs/ has been created
ok 3 - ipfs config succeeds
# passed all 3 test(s)
1..3
*** t0030-mount.sh ***
./t0030-mount.sh
ok 1 - ipfs init succeeds
ok 2 - prepare config
ok 3 - ipfs mount succeeds
ok 4 - ipfs mount output looks good
ok 5 - ipfs mount is still running
ok 6 - ipfs mount can be killed
# passed all 6 test(s)
1..6
*** t0040-add-and-cat.sh ***
./t0040-add-and-cat.sh
ok 1 - ipfs init succeeds
ok 2 - prepare config
ok 3 - ipfs mount succeeds
ok 4 - ipfs mount output looks good
ok 5 - ipfs add succeeds
ok 6 - ipfs add output looks good
ok 7 - ipfs cat succeeds
ok 8 - ipfs cat output looks good
ok 9 - cat ipfs/stuff succeeds
not ok 10 - cat ipfs/stuff looks good
#
#       test_cmp expected actual
#
ok 11 - ipfs mount is still running
ok 12 - ipfs mount can be killed
# failed 1 among 12 test(s)
1..12
make: *** [t0040-add-and-cat.sh] Error 1
jbenet @ lorien : ~/go/src/github.com/jbenet/go-ipfs/test * chriscool-test_ipfs_mount % cat trash\ directory.t0040-add-and-cat.sh/actual
Hello Worlds!
Unmounting /Users/jbenet/go/src/github.com/jbenet/go-ipfs/test/trash directory.t0040-add-and-cat.sh/ipns...
jbenet @ lorien : ~/go/src/github.com/jbenet/go-ipfs/test * chriscool-test_ipfs_mount % cat trash\ directory.t0040-add-and-cat.sh/expected
Hello Worlds!

jbenet · 2014-11-02T05:49:50Z

Also:

dont worry about that travis error, 1.2 is no longer supported (as you saw)
would be great to add these tests to travis (we could have it download sharness, etc)

chriscool · 2014-11-02T06:19:27Z

About the test failure, could you use hexdump -C on the expected and actual files to make sure that there is no difference?
About adding the tests to Travis, yeah that would be great!

chriscool · 2014-11-02T07:09:03Z

About using a 100MB file with entropy, I can generate one with something like:

dd if=/dev/zero bs=1M count=100 | openssl bf-ofb -pass pass:HelloMars

Hopefully it is portable enough.

chriscool · 2014-11-02T09:23:02Z

@jbenet I think it is a bug in ipfs that sometimes "cat ipfs/$HASH" would output not just the content of the $HASH object but also stuff like "Unmounting /Users/jbenet..."

jbenet · 2014-11-02T09:26:25Z

@jbenet I think it is a bug in ipfs that sometimes "cat ipfs/$HASH" would output not just the content of the $HASH object but also stuff like "Unmounting /Users/jbenet..."

Ah, yes, sorry. I didn't even stop to consider that!

Not sure why it's outputting Unmounting... for ipfs cat. It may be that I'm looking at it after all runs, one second.

jbenet · 2014-11-02T09:27:43Z

Speaking of bugs in the system, github's bugging out right now:

jbenet · 2014-11-02T09:35:41Z

@chriscool you can use https://github.com/jbenet/go-random -- it should be portable, and uses io.

chriscool · 2014-11-02T09:48:05Z

@jbenet I think it is a bug in ipfs or fuse (or somewhere in between) that sometimes "cat ipfs/$HASH" would output stuff (like "Unmounting...") that is not in the object referenced by $HASH.

whyrusleeping · 2014-11-02T09:48:38Z

@chriscool sadly that will only work on unix based systems... and even though windows hasnt been in our dev scope for a while, we do want to support it eventually.

chriscool · 2014-11-02T09:49:05Z

@jbenet I think it is a bug in ipfs or fuse (or somewhere in between) that sometimes "cat ipfs/$HASH" would output stuff (like "Unmounting...") that is not in the object referenced by $HASH.

jbenet · 2014-11-02T09:53:28Z

Re: portability of random file, we can build a tiny, portably go cmd that reads and outputs from crypto.Rand:

> rand
Usage: rand <num bytes>
> rand 6
2q���#

jbenet · 2014-11-02T09:55:22Z

@chriscool ah wait, no thats not the problem. That ("Unmounting...") was being added by subsequent parts of the test.

Here's the proper hexdumps:

jbenet @ lorien : ~/go/src/github.com/jbenet/go-ipfs/test * chriscool-test_ipfs_mount % hexdump -C trash\ directory.t0040-add-and-cat.sh/expected
00000000  48 65 6c 6c 6f 20 57 6f  72 6c 64 73 21 0a        |Hello Worlds!.|
0000000e
jbenet @ lorien : ~/go/src/github.com/jbenet/go-ipfs/test * chriscool-test_ipfs_mount % touch LIC
jbenet @ lorien : ~/go/src/github.com/jbenet/go-ipfs/test * chriscool-test_ipfs_mount % hexdump -C trash\ directory.t0040-add-and-cat.sh/actual
00000000  48 65 6c 6c 6f 20 57 6f  72 6c 64 73 21 0a 00 00  |Hello Worlds!...|
00000010  00 00 00 00 00 00                                 |......|
00000016

cat seems to be adding extra stuff at the end. (still an error on our side)

cc @whyrusleeping weren't we seeing this earlier? random trailing zeroes?

jbenet · 2014-11-02T09:55:25Z

@chriscool I meant there is a difference in the files ("Unmounting....") sorry if my wording came off as the opposite.

jbenet @ lorien : ~/go/src/github.com/jbenet/go-ipfs * chriscool-test_ipfs_mount % hexdump -C test/trash\ directory.t0040-add-and-cat.sh/actual
00000000  48 65 6c 6c 6f 20 57 6f  72 6c 64 73 21 0a 00 00  |Hello Worlds!...|
00000010  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
000000d0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 55 6e  |..............Un|
000000e0  6d 6f 75 6e 74 69 6e 67  20 2f 55 73 65 72 73 2f  |mounting /Users/|
000000f0  6a 62 65 6e 65 74 2f 67  6f 2f 73 72 63 2f 67 69  |jbenet/go/src/gi|
00000100  74 68 75 62 2e 63 6f 6d  2f 6a 62 65 6e 65 74 2f  |thub.com/jbenet/|
00000110  67 6f 2d 69 70 66 73 2f  74 65 73 74 2f 74 72 61  |go-ipfs/test/tra|
00000120  73 68 20 64 69 72 65 63  74 6f 72 79 2e 74 30 30  |sh directory.t00|
00000130  34 30 2d 61 64 64 2d 61  6e 64 2d 63 61 74 2e 73  |40-add-and-cat.s|
00000140  68 2f 69 70 6e 73 2e 2e  2e 0a                    |h/ipns....|
0000014a
jbenet @ lorien : ~/go/src/github.com/jbenet/go-ipfs * chriscool-test_ipfs_mount % hexdump -C test/trash\ directory.t0040-add-and-cat.sh/expected
00000000  48 65 6c 6c 6f 20 57 6f  72 6c 64 73 21 0a        |Hello Worlds!.|
0000000e

chriscool · 2014-11-02T09:57:47Z

@jbenet I think it is a bug in ipfs or fuse (or somewhere in between) that sometimes "cat ipfs/$HASH" would output stuff (like "Unmounting...") that is not in the object referenced by $HASH.

chriscool · 2014-11-02T10:03:05Z

What I can do is I can use test_expect_failure instead of test_expect_success in the test script, so that we document that this test might fail.

Also with a 100MB file I get a different hash everytime I add the same file. It looks like a bug to me.

chriscool · 2014-11-02T10:08:29Z

Ooops I don't know what happened to this PR but it loosk like some comments have been duplicated, and I thought that some of my comments were lost, so I reposted them.

jbenet · 2014-11-02T10:08:52Z

What I can do is I can use test_expect_failure instead of test_expect_success in the test script, so that we document that this test might fail.

No, it's definitely a failure, the test is correct (i.e. leave as is). We should fix this problem in ipfs.

Also with a 100MB file I get a different hash everytime I add the same file. It looks like a bug to me.

Yep, that's certainly a bug. But-- is your ipfs up to date? your tests don't build the code beforehand, so may not be. I recently (before your fork) fixed a serious bug on this that got introduced recently. I just rebuilt (with this branch) and passes for me:

% random 134217728 > testfile
% du -sh testfile
128M  testfile
% ipfs add testfile
added QmcAhAJ1s1GDySLEHXXMwLdFTZHqEAHpziXB39yLkRFotU /Users/jbenet/go/src/github.com/jbenet/go-ipfs/cmd/ipfs/testfile
% ipfs add testfile
added QmcAhAJ1s1GDySLEHXXMwLdFTZHqEAHpziXB39yLkRFotU /Users/jbenet/go/src/github.com/jbenet/go-ipfs/cmd/ipfs/testfile
% ipfs add testfile
added QmcAhAJ1s1GDySLEHXXMwLdFTZHqEAHpziXB39yLkRFotU /Users/jbenet/go/src/github.com/jbenet/go-ipfs/cmd/ipfs/testfile

jbenet · 2014-11-02T10:10:48Z

Ooops I don't know what happened to this PR but it loosk like some comments have been duplicated, and I thought that some of my comments were lost, so I reposted them.

Yes, there's some github bugs happening. see the picture in #248 (comment) and compare to reality.

chriscool · 2014-11-02T17:39:16Z

About the bug with trailing zeros and "Unmounting...", just to make sure we agree, it is a bug when running "cat ipfs/$HASH" where "ipfs/" is the directory where ipfs is mounted. What you posted doesn't mean a bug when runnning "ipfs cat $HASH", though maybe there is also the same bug when runnning "ipfs cat $HASH" but it is not triggered for some reason.

Also you are right, I am not running the latest version from master, because I didn't upgrade yet to go 1.3. I am still running go 1.2.1, but I will upgrade soon.

One last point, after thinking about it some more, I am not sure creating the 100MB with pure random stuff is the best thing to do. Maybe using a pseudo random generator is better, because this way we can check that the hash we get back when we "ipfs add" the file is always the same hardcoded value. This way we check that there is no regression in the hash function on a big file.

But I agree that as we want to support Windows, it is better if can rely on a simple go program like go-random.

whyrusleeping · 2014-11-02T18:24:05Z

Hahahaha, you guys had a daylight savings time bug

chriscool · 2014-11-02T23:31:38Z

I created this PR to have a pseudo random generator:

jbenet/go-random#1

jbenet · 2014-11-02T23:51:23Z

jbenet/go-random#1 merged, thanks!

chriscool · 2014-11-03T00:11:53Z

Great thanks! I have to sleep now, so I will finish this later, hopefully before next week end.

jbenet · 2014-11-03T00:33:06Z

@chriscool no worries :) 👍 thanks!

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

chriscool · 2014-11-06T22:16:25Z

@jbenet please tell me if you need something else to merge this PR
About adding the tests to Travis, can I do it or are some special rights needed?

jbenet · 2014-11-06T22:27:42Z

Hey @chriscool -- this is great! The tests fail for me-- presumably because of the extra zeroes at the ends of things. If the tests are correct (confirm?), let's merge it in and fix those bugs in ipfs.

As for travis, I'll add it now. it's just modifying the .travis.yml file

chriscool · 2014-11-06T22:35:04Z

The tests work on my machine. I think they are correct.
Also it will be difficult for me to fix as it works here.
Could it be a fuse bug on MacOS?

whyrusleeping · 2014-11-06T22:38:27Z

MacOSX is known for having fuse bugs, one of which i havent actually gotten around to fixing yet

whyrusleeping · 2014-11-06T22:38:45Z

(but i do know the solution... maybe that should be my next priority)

jbenet · 2014-11-06T22:41:06Z

@chriscool ok great, I'll merge this then

Add test for ipfs mount

jbenet · 2014-11-06T22:42:12Z

Fix a race in dial queue

Add test for ipfs mount

b6a98c1

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

chriscool added 2 commits October 31, 2014 21:44

Add trash directories into test/.gitignore

223d5c3

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

Add test script for add and cat commands

e3da26d

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

chriscool added 2 commits November 1, 2014 10:52

Refactor mount related tests

4e2763a

This adds test_launch_ipfs_mount() and test_kill_ipfs_mount() to avoid duplicating tests to launch "ipfs mount" and to kill it. License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

Speed up checking if mount works

630816c

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

jbenet reviewed Nov 2, 2014
View reviewed changes

test ipfs cat and ipfs add with a 100MB file

22ac599

License: MIT Signed-off-by: Christian Couder <chriscool@tuxfamily.org>

jbenet added a commit that referenced this pull request Nov 6, 2014

Merge pull request #248 from chriscool/test_ipfs_mount

62f1614

Add test for ipfs mount

jbenet merged commit 62f1614 into ipfs:master Nov 6, 2014

aschmahmann mentioned this pull request Aug 23, 2021

Release v0.10 #8176

Closed

62 tasks

github-actions bot mentioned this pull request Oct 2, 2021

Publish New Versions JonasKruckenberg/go-ipfs-sidecar#1

Merged

ariescodescream pushed a commit to ariescodescream/go-ipfs that referenced this pull request Oct 23, 2021

Merge pull request ipfs#248 from raulk/master

ad151c1

Fix a race in dial queue

aschmahmann mentioned this pull request Dec 1, 2021

Release v0.11 #8343

Closed

80 tasks

Add test for ipfs mount #248

Add test for ipfs mount #248

Conversation

chriscool commented Oct 31, 2014

whyrusleeping commented Oct 31, 2014

chriscool commented Oct 31, 2014

whyrusleeping commented Oct 31, 2014

chriscool commented Oct 31, 2014

chriscool commented Nov 2, 2014

jbenet Nov 2, 2014

Choose a reason for hiding this comment

chriscool Nov 2, 2014

Choose a reason for hiding this comment

jbenet Nov 2, 2014

Choose a reason for hiding this comment

chriscool Nov 2, 2014

Choose a reason for hiding this comment

jbenet commented Nov 2, 2014

jbenet commented Nov 2, 2014

chriscool commented Nov 2, 2014

chriscool commented Nov 2, 2014

chriscool commented Nov 2, 2014

jbenet commented Nov 2, 2014

jbenet commented Nov 2, 2014

jbenet commented Nov 2, 2014

chriscool commented Nov 2, 2014

whyrusleeping commented Nov 2, 2014

chriscool commented Nov 2, 2014

jbenet commented Nov 2, 2014

jbenet commented Nov 2, 2014

jbenet commented Nov 2, 2014

chriscool commented Nov 2, 2014

chriscool commented Nov 2, 2014

chriscool commented Nov 2, 2014

jbenet commented Nov 2, 2014

jbenet commented Nov 2, 2014

chriscool commented Nov 2, 2014

whyrusleeping commented Nov 2, 2014

chriscool commented Nov 2, 2014

jbenet commented Nov 2, 2014

chriscool commented Nov 3, 2014

jbenet commented Nov 3, 2014

chriscool commented Nov 6, 2014

jbenet commented Nov 6, 2014

chriscool commented Nov 6, 2014

whyrusleeping commented Nov 6, 2014

whyrusleeping commented Nov 6, 2014

jbenet commented Nov 6, 2014

jbenet commented Nov 6, 2014