several test failures on several non amd64 architectures on Debian

[boutil]

Hi,

Up to 2.9.4, when the Debian package for oj was built, only test/tests.rb was run, and it worked successfully. When trying to update to 2.9.6, I tried to enable more test and run more tests (all test/test_*.rb).
Whereas those tests pass reliably on amd64 where I build the package, they fail on several other architectures: see https://buildd.debian.org/status/package.php?p=ruby-oj&suite=sid

To run the tests, I used something like:

Dir.glob("test/test_*.rb").each { |f| require f}

Running a rake task as the one listed in the Rakefile from line 10 to 14 gives the same results.

For example, on armhf, I get from 6 to 9 test failures, depending on the order of the tests.

 1) Failure:
CompatJuice#test_symbol_keys [/home/boutil/ruby-oj-2.9.6/test/test_compat.rb:229]:
--- expected
+++ actual
@@ -1 +1 @@
-{:x=>true, :y=>58, :z=>[1, 2, 3]}
+nil

  2) Failure:
ObjectJuice#test_io_file [/home/boutil/ruby-oj-2.9.6/test/test_object.rb:295]:
--- expected
+++ actual
@@ -1 +1 @@
-{"x"=>true, "y"=>58, "z"=>[1, 2, 3]}
+nil

  3) Failure:
ObjectJuice#test_io_string [/home/boutil/ruby-oj-2.9.6/test/test_object.rb:281]:
--- expected
+++ actual
@@ -1 +1 @@
-{"x"=>true, "y"=>58, "z"=>[1, 2, 3]}
+nil

  4) Failure:
ObjectJuice#test_symbol_keys [/home/boutil/ruby-oj-2.9.6/test/test_object.rb:307]:
--- expected
+++ actual
@@ -1 +1 @@
-{:x=>true, :y=>58, :z=>[1, 2, 3]}
+nil

  5) Failure:
StrictJuice#test_io_file [/home/boutil/ruby-oj-2.9.6/test/test_strict.rb:174]:
--- expected
+++ actual
@@ -1 +1 @@
-{"x"=>true, "y"=>58, "z"=>[1, 2, 3]}
+nil

  6) Failure:
StrictJuice#test_io_string [/home/boutil/ruby-oj-2.9.6/test/test_strict.rb:160]:
--- expected
+++ actual
@@ -1 +1 @@
-{"x"=>true, "y"=>58, "z"=>[1, 2, 3]}
+nil

  7) Failure:
StrictJuice#test_symbol_keys [/home/boutil/ruby-oj-2.9.6/test/test_strict.rb:186]:
--- expected
+++ actual
@@ -1 +1 @@
-{:x=>true, :y=>58, :z=>[1, 2, 3]}
+nil

  8) Failure:
StrictJuice#test_symbol_keys_safe [/home/boutil/ruby-oj-2.9.6/test/test_strict.rb:197]:
--- expected
+++ actual
@@ -1 +1 @@
-{"x"=>true, "y"=>58, "z"=>[1, 2, 3]}
+nil

  9) Failure:
Juice#test_symbol_keys [/home/boutil/ruby-oj-2.9.6/test/test_various.rb:1065]:
--- expected
+++ actual
@@ -1 +1 @@
-{:x=>true, :y=>58, :z=>[1, 2, 3]}
+nil

307 runs, 535 assertions, 9 failures, 0 errors, 0 skips

I would be happy to help investigate this issue.

Cheers,

Cédric

[ohler55]

I think I might need some help with it as I don't have a debian machine or VM set up. It looks like you have a pretty nice test framework there.

btw, there is a test_all.sh script you can use for running all the tests.

[Asmod4n]

It also doesn't compile with clang(Version: 1:3.0-6.2) from debian stable(wheezy) while using rubinius 2.2.9 compiled from source with the same clang version.

compiling saj.c
clang: error: unable to execute command: Killed
clang: error: clang frontend command failed due to signal 2 (use -v to see invocation)
clang: note: diagnostic msg: Please submit a bug report to http://llvm.org/bugs/ and include command line arguments and all diagnostic information.
clang: note: diagnostic msg: Preprocessed source(s) are located at:
clang: note: diagnostic msg: /tmp/saj-djMD86.i
make: *** [saj.o] Error 254

make failed, exit code 2

[ohler55]

Not much do go on with that error. I'll have to put together a debian install in looks like.

[ohler55]

I've installed debian release 7.5, rbenv, ruby 2.1.2 and pulled the oj repo. It compiles and all the tests pass. What do I need to do to reproduce your errors?

[Asmod4n]

aptitude install clang
export CC=clang
export CXX=clang++
export CPP=clang -E
rbenv install rbx-2.2.9
rbx -S gem install oj

[Asmod4n]

It might also be necessary to

sudo update-alternatives --config c++
sudo update-alternatives --config cc

and set them to clang

[ohler55]

Thanks trying now.

[ohler55]

export CPP=clang -E
fails with
-bash: export: `-E': not a valid identifier

[Asmod4n]

export CPP="clang -E"

[ohler55]

rbenv install rbx-2.2.9
fails with
rbenv install rbx-2.2.9
Downloading yaml-0.1.6.tar.gz...
-> http://dqw8nmjcqpjn7.cloudfront.net/7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749
Installing yaml-0.1.6...

BUILD FAILED

Inspect or clean up the working tree at /tmp/ruby-build.20140629211518.26783
Results logged to /tmp/ruby-build.20140629211518.26783.log

Last 10 log lines:
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether clang accepts -g... yes
checking for clang option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of clang... gcc3
checking how to run the C preprocessor... clang
configure: error: in /tmp/ruby-build.20140629211518.26783/yaml-0.1.6': configure: error: C preprocessor "clang" fails sanity check Seeconfig.log' for more details

BUILD FAILED

Inspect or clean up the working tree at /tmp/ruby-build.20140629211518.26783
Results logged to /tmp/ruby-build.20140629211518.26783.log

Last 10 log lines:
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether clang accepts -g... yes
checking for clang option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of clang... gcc3
checking how to run the C preprocessor... clang
configure: error: in /tmp/ruby-build.20140629211518.26783/yaml-0.1.6': configure: error: C preprocessor "clang" fails sanity check Seeconfig.log' for more details

Don't know where the config.log file is.

[Asmod4n]

you sure its set to "clang -E" ? works fine here.

[ohler55]

quotes worked

[Asmod4n]

Don't have the time to debug rbenv now, but grabbing it from http://rubini.us/downloads/ and doing what they say in http://rubini.us/doc/en/getting-started/requirements/ and http://rubini.us/doc/en/getting-started/building/ at Installing Rubinius works for me.

[ohler55]

Similar err to before.

rbenv install rbx-2.2.9
Downloading yaml-0.1.6.tar.gz...
-> http://dqw8nmjcqpjn7.cloudfront.net/7da6971b4bd08a986dd2a61353bc422362bd0edcc67d7ebaac68c95f74182749
Installing yaml-0.1.6...
Installed yaml-0.1.6 to /home/ohler/.rbenv/versions/rbx-2.2.9

Downloading rubinius-2.2.9.tar.bz2...
-> http://dqw8nmjcqpjn7.cloudfront.net/7b01a7f2508167e73b5273b4e55e6616fc7fd975e79c84c4d2e3ef83d849d2ce
Installing rubinius-2.2.9...

BUILD FAILED

Inspect or clean up the working tree at /tmp/ruby-build.20140629212023.27160
Results logged to /tmp/ruby-build.20140629212023.27160.log

Last 10 log lines:
Rubinius::Loader#script at kernel/loader.rb:649
Rubinius::Loader#main at kernel/loader.rb:825
rake aborted!
Command failed with status (1): [/tmp/ruby-build.20140629212023.27160/rubin...]
/tmp/ruby-build.20140629212023.27160/rubinius-2.2.9/rakelib/gems.rake:5:in bootstrap_rubinius' /tmp/ruby-build.20140629212023.27160/rubinius-2.2.9/rakelib/gems.rake:40:inblock (3 levels) in <top (required)>'
/tmp/ruby-build.20140629212023.27160/rubinius-2.2.9/rakelib/gems.rake:39:in chdir' /tmp/ruby-build.20140629212023.27160/rubinius-2.2.9/rakelib/gems.rake:39:inblock (2 levels) in <top (required)>'
Tasks: TOP => install => gems:install
(See full trace by running task with --trace)

[ohler55]

ok, will give that a try... tomorrow.

[Asmod4n]

maybe

sudo apt-get install clang

? if you have just typed aptitude install clang as a normal user its probably not installed.

[ohler55]

Got it installed. Missing some requirements. It just was not obvious from the error message. I was able to reproduce the clang error as well. Pretty deep error. Crash the compile.

[Asmod4n]

guess the next steps would be to install rbx 2.2.10 and master afterwards, seeing how it compiles without a issue with ruby 2.1.2 and the same clang.

[ohler55]

It might be a macro conflict in the RBX headers. I'll try witting away at the header file and figure it out.

[boutil]

Hi!
I just tried to reproduce the problem with test failures on non amd64 architecture with the code from github (as opposed to the gem). I note that the test/test_all.sh file is not present in the gem.
I ran rake on an armhf machine, and got 8 test failures (all in the list of the first entry).

This is Debian unstable, with gcc, so it is unrelated to what @Asmod4n describes.

[ohler55]

I give the rake target a try here then. See what happens.

[ohler55]

Will work on this tonight. With Ruby 2.1.2, debian 7.5 and using rake after doing a bundle install all tests pass. Sadly I don't have an ARM platform to run the tests on. Anyone want to help with debugging that has a ARM platform?

[ohler55]

Narrow the problem down to either calling to not calling a specific function. Clearly a compiler problem. There seems to be enough other people running into the same issue with the clang compiler. Not much I can do about it other than suggest using a different compiler.

[boutil]

Hi,

I did some testing again on armhf. Running the rake test task yields the test failure I described above. Running the test files independently one by one seems to work though. I tried to extract some dfailing tests (test_io_file from test_compat.rb and test_strict.rb) and put them in the same file. Running them together do not yield failures. There might be some hidden interaction between tests causing the failures, but I don't know how exactly.

If you have ideas I can test, I would be happy to run the tests.

Cedric

[boutil]

Some more tests, this time on i386:
ruby -I lib:ext/oj/ test/test_object.rb --seed 50514 passes.

with --seed 39071, I got one failure

  1) Failure:
ObjectJuice#test_symbol_keys [test/test_object.rb:307]:
--- expected
+++ actual
@@ -1 +1 @@
-{:x=>true, :y=>58, :z=>[1, 2, 3]}
+nil

and with --seed 13163, I got two failures

  1) Failure:
ObjectJuice#test_symbol_keys [test/test_object.rb:307]:
--- expected
+++ actual
@@ -1 +1 @@
-{:x=>true, :y=>58, :z=>[1, 2, 3]}
+nil



  2) Failure:
ObjectJuice#test_io_file [test/test_object.rb:295]:
--- expected
+++ actual
@@ -1 +1 @@
-{"x"=>true, "y"=>58, "z"=>[1, 2, 3]}
+nil

[boutil]

running ruby -I lib:ext/oj/ test/test_object.rb fails 2/3 of the time (657 out of 1000) with a random seed. Since two tests are failing, it means that these failures are caused by exactly one other test of the set, when it is run before the other two. (The probability that the faulty test is run after the two sensible ones is 1/3).

[boutil]

For test_object.rb, the culprit is test_io_string. Keeping just test_io_string, test_io_file and test_symbol_keys, the tests fail 66% of the time (when test_io_string is run first). Deactivating test_io_string and keeping all the others make the tests pass 100% of the time.

Actually, deactivating all the test_io_string in test_object.rb, test_compat.rb, test_strict.rb and test_various.rb makes the whole test suite pass reliably, on i386 as well as on armhf, and I guess on all other previously failing architectures.

[ohler55]

Thank you for the discovery and the detective work. I'll look at it tonight.

[ohler55]

Are you using Oj 2.10.2?

[boutil]

I checked with 2.9.6 and 2.10.1.
Le 24 août 2014 20:49, "Peter Ohler" notifications@github.com a écrit :

Are you using Oj 2.10.2?

—
Reply to this email directly or view it on GitHub
#166 (comment).

[boutil]

I couldn't reproduce the problem with 2.10.2. The issue is solved!
My investigation came too late to be useful :/ but I am glad this problem has been sorted out. Thanks!

[ohler55]

2.10.2 did fix a string IO issue. Probably the cause.

[boutil]

I am closing it, since this issue seems resolved now. Don't worry, I'm opening a new one :)