JRuby fails in building from source when using openjdk-7-jdk trusty package #4103

Closed
RochesterinNYC opened this Issue Aug 22, 2016 · 28 comments

Projects

None yet

5 participants

@RochesterinNYC

Environment

Provide at least:

  • Trying to build jruby 9.1.2.0 from source
  • uname -a:
    Linux d64e610fc91e 4.4.16-boot2docker #1 SMP Fri Jul 29 00:13:24 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Expected Behavior

  • JRuby builds from source using version 7u111-2.6.7-0ubuntu0.14.04.3 of the ubuntu trusty openjdk-7-jdk package.

Actual Behavior

  • JRuby fails in building from source using version 7u111-2.6.7-0ubuntu0.14.04.3 of the ubuntu trusty openjdk-7-jdk package.

Reproduction:

git clone git@github.com:cloudfoundry/binary-builder.git
cd binary-builder
docker run -it -v $(pwd):/binary-builder cloudfoundry/cflinuxfs2 bash
cd /binary-builder
./bin/binary-builder --name=jruby --version=9.1.2.0_ruby-2.3.0 --sha256=dfd44f532105725a4d4a2378a3b6a06899bae660be21538a5b36b41e4dd045e6

The full error log after switching maven to try and build jruby with the -X flag enabled is here: https://gist.github.com/RochesterinNYC/fac1f36bb98ded1f5a135f10b2c8741c

Binary builder will try to install openjdk via apt-get, build and compile ant and maven, and then use maven to try to build jruby. It runs the following command to try to build jruby: mvn -Djruby.default.ruby.version=2.3.0

Side Note:
It might be worth noting that for our situation, this build process was working when we were on openjdk-7-jdk 7u111-2.6.7-0ubuntu0.14.04.2. When openjdk-7-jdk 7u111-2.6.7-0ubuntu0.14.04.3 was released on August 11th, our build process automatically started using this (because we use apt-get install) and the build started failing.

@headius
Member
headius commented Aug 22, 2016

It seems like the complete log was truncated, or else the bits I need didn't get in there. Can you do another attempt without -X and gist that too?

We do build with Java 7 periodically, so that shouldn't be the issue. Unfortunately I can't see the actual error.

@mkristian
Member

@RochesterinNYC are you sure it works with 7u111-2.6.7-0ubuntu0.14.04.2 as it is only two days older then 7u111-2.6.7-0ubuntu0.14.04.3 but has some mentioning of 'unsafe' with it - these two unsafe methods could be the cause of the failure. to the see the compilation error it shold help to add -Dmaven.compiler.fork=false to commandline. with the forked compiler most output is hidden.

@RochesterinNYC

I think the full log output is available here: https://gist.githubusercontent.com/RochesterinNYC/fac1f36bb98ded1f5a135f10b2c8741c/raw/ec88989ef61a62fe4aca23030b482113720d2b73/compile.log

We feel that it works with 7u111-2.6.7-0ubuntu0.14.04.2 as our last successful build of jruby was completed August 10th 2016 at ~12 AM. The next build occurred on August 17th 2016 at 12:30 AM and it failed to build jruby. As we use locked versions of Ant and Maven in the build process, but use apt-get install for openjdk-7-jdk, this is the only build input that could have changed. So for the August 10th build, 7u111-2.6.7-0ubuntu0.14.04.2 was used, and for the failed August 17th build, 7u111-2.6.7-0ubuntu0.14.04.3 was used.

@RochesterinNYC
RochesterinNYC commented Aug 23, 2016 edited

Hm, additionally, we're having some trouble trying to build with -Dmaven.compiler.fork=false. We specify this flag/argument but the output logs show that fork is ultimately set to true: https://gist.githubusercontent.com/RochesterinNYC/fac1f36bb98ded1f5a135f10b2c8741c/raw/eb703754f25d7df987c7530ece60e59ab24c7a88/compile-with-attempted-fork-false.log

The specified "fork=false" shows up in the logs if you search for them, but fork is ultimately set as "fork = true".

@mkristian
Member

@RochesterinNYC ok I can reproduce the error with your Docker setup. can you give me some hints on how I can find the output and how to maybe run maven manually ? bit lost in the filestructure in /binary-builder

@headius
Member
headius commented Aug 23, 2016

It shouldn't be too difficult to take that last javac command line and run it. It's the line starting with [DEBUG] -d /tmp/x86_64-linux-gnu (the second one in your output) and ending with -XDignore.symbol.file=true. Everything from -d and on should be passed to javac, and the whole command probably needs to go in a shell script to be runnable (way too long for command line).

Ideally, that will show us why the javac step is blowing up. I sure hate Maven for hiding the actual error.

@mkristian
Member

@headius the error I see is:

checking org.jruby.util.unsafe.UnsafeHolder]
/tmp/x86_64-linux-gnu/ports/jruby/9.1.2.0_ruby-2.3.0/jruby-9.1.2.0/core/src/main/java/org/jruby/util/unsafe/UnsafeHolder.java:96: error: cannot find symbol
        U.fullFence();
         ^
  symbol:   method fullFence()
  location: variable U of type Unsafe
/tmp/x86_64-linux-gnu/ports/jruby/9.1.2.0_ruby-2.3.0/jruby-9.1.2.0/core/src/main/java/org/jruby/util/unsafe/UnsafeHolder.java:100: error: cannot find symbol
        U.loadFence();
         ^
  symbol:   method loadFence()
  location: variable U of type Unsafe
/tmp/x86_64-linux-gnu/ports/jruby/9.1.2.0_ruby-2.3.0/jruby-9.1.2.0/core/src/main/java/org/jruby/util/unsafe/UnsafeHolder.java:104: error: cannot find symbol
        U.storeFence();
         ^
  symbol:   method storeFence()
  location: variable U of type Unsafe
@mkristian
Member

so I missed the bootclasspath and this line could be the problem:

    <compilerArg>-J-Xbootclasspath/p:${settings.localRepository}/com/headius/unsafe-mock/8.92.1/unsafe-mock-8.92.1.jar</compilerArg>

i.e. maven can not resolve ${settings.localRepository} - I do remember having problems with this before but I am guessing here.

why is thie -J-Xbootclasspath and not -Xbootclasspath ?

@headius
Member
headius commented Aug 23, 2016

@mkristian So it's not picking up our bootstrap unsafe mock then I guess?

@headius
Member
headius commented Aug 23, 2016

@mkristian The -J is because it's being passed to javac.

@mkristian
Member

@headius so I am confused as if I change the core/pom.rb to use -Xbootclasspath/p:${unsafe.jar} instead of -J-Xbootclasspath/p:${unsafe.jar} then the compile phase passes.

@headius
Member
headius commented Aug 23, 2016

@mkristian I feel like we've had this discussion before...now I'm confused too.

@mkristian
Member

@headius both seems to valid if I look at javac -help and javac -X so I am OK to add both variants. locally on jdk8 on macOS it works with both.

@RochesterinNYC

Is anything needed from us in terms of reproducibility or debug logs?

@mkristian
Member

@RochesterinNYC I can reproduce the error and I have a workaround. not sure why things changed with jdk version. will produce a PR and let the jruby core team decide if they want to go with it.

@headius headius closed this in #4107 Aug 24, 2016
@headius headius added this to the JRuby 9.1.3.0 milestone Aug 24, 2016
@sesmith177

Hello,

We're not sure that this fixes our problem. We tried to build JRuby 9.1.3.0 in the manner described above, however the build still fails. The command we ran in our container:

./bin/binary-builder --name=jruby --version=9.1.3.0_ruby-2.3.1 --sha256=2acde32e0069a2d9a5593c45386545bf66eb08bdb3d514c49e8258980db5fcab

In the logs, we saw that -Xbootclasspath was passed to the java compiler:

-Xbootclasspath/p:/root/.m2/repository/com/headius/unsafe-mock/8.92.1/unsafe-mock-8.92.1.jar

@cf-tm-bot cf-tm-bot referenced this issue in cf-tm-bot/buildpacks Sep 5, 2016
Closed

Fix Binary Builder for jruby - Story Id: 128679225 #107

@headius
Member
headius commented Sep 6, 2016

@sesmith177 Well that's frustrating :-( I'm not sure what else would be causing a problem.

Can you try removing all bootclasspath lines from the compile plugins (use @mkristian's patch as a guide) and see if we at least get a reasonable error?

For background: the unsafe-mock library was added so we could compile code that uses Unsafe.fullFence -- a Java 8 feature -- even when building using Java 7. fullFence is used mostly to guarantee the integrity of array-based instance variable tables when mutating across threads. Java 7 does this through other means.

There must be a way to get more information out of that blasted compiler plugin!

@headius headius reopened this Sep 6, 2016
@headius headius modified the milestone: JRuby 9.1.5.0, JRuby 9.1.3.0 Sep 6, 2016
@headius
Member
headius commented Sep 6, 2016

Reopened and targeted to 9.1.5.0.

@sesmith177

Hello @headius,

We removed the bootclasspath lines from core/pom.rb and core/pom.xml. The build still failed; logs are here.

Thanks

@headius
Member
headius commented Sep 6, 2016

Ok, so this appears to be something other than the bootclasspath failing. One more thing to try: just remove the -J version of that flag. I'm wondering if forcing our Unsafe into the underlying JVM is breaking javac, where the plain -X version would not have that effect.

@enebo enebo modified the milestone: JRuby 9.1.5.0, JRuby 9.1.6.0 Sep 7, 2016
@RochesterinNYC

@headius Removing the '-J-Xbootclasspath/p:${unsafe.jar}' line caused it to compile correctly with no errors.

@mkristian
Member

I was sure I got it to work before I suggested a patch for our build. well, did not test the patch on the docker container :(

still think we can use maven profiles to the bootclasspath for the different jdks the right way. have to play around with this.

@RochesterinNYC

For context, the way I've been making these jruby source edits and playing around is once you boot up the container and execute a failed build of jruby, the jruby source tarball will be at path_to/binary-builder/ports/archives/jruby-src-9.1.5.0.tar.gz. I untar that and make whatever edits necessary, tar it back up, and ensure the modified tarball is at that path with that filename. Then, when you execute binary-builder again, it won't download jruby source again but rather just use that tarball for building.

@headius
Member
headius commented Sep 8, 2016

@headius Removing the '-J-Xbootclasspath/p:${unsafe.jar}' line caused it to compile correctly with no errors.

Of course it did! 😞 I wonder if we could fudge this with method handles and eliminate the mocks. What a pain.

@headius
Member
headius commented Sep 8, 2016

One other thing did occur to me...we are backing off from forcing instance variables to be volatile in favor of an explicit API. So that would reduce our need for the Java 8+ fence methods on Unsafe.

I also did a quick benchmark of a MethodHandle-based fullFence that doesn't require a mock, and it ends up with basically the same performance as calling Unsafe directly. Pretty impressive.

@RochesterinNYC
RochesterinNYC commented Sep 15, 2016 edited

Are there any other testing or logs or things you'd want us to try to provide feedback/info?

@mkristian mkristian added a commit that referenced this issue Oct 8, 2016
@mkristian mkristian first step to use maven profiles to set bootclasspath
jdk9 on macos can not handle both configurations. is related to #4103
f2c9a07
@headius
Member
headius commented Nov 8, 2016

I think this requires heavier lifting on our side, like moving both the unsafe mock usage AND the utility library wrapped around it to a separate library, so we can build it however we please without affecting the JRuby codebase. That's not going to happen for 9.1.6.0 but it could happen for 9.1.7.0.

Then we could just drop the pesky bootclasspath flag entirely.

@headius headius modified the milestone: JRuby 9.1.7.0, JRuby 9.1.6.0 Nov 8, 2016
@RochesterinNYC

@headius thanks for the update. Please keep us involved and let us know if there's anything the CF Buildpacks Team can do.

@chrisseaton chrisseaton added a commit that referenced this issue Nov 17, 2016
@mkristian @chrisseaton mkristian + chrisseaton first step to use maven profiles to set bootclasspath
jdk9 on macos can not handle both configurations. is related to #4103
79eabf4
@chrisseaton chrisseaton added a commit that referenced this issue Nov 27, 2016
@mkristian @chrisseaton mkristian + chrisseaton first step to use maven profiles to set bootclasspath
jdk9 on macos can not handle both configurations. is related to #4103
07f50dd
@headius headius added a commit to headius/jruby that referenced this issue Dec 13, 2016
@headius headius Remove unsafe-mock from our build altogether.
This commit replaces the unsafe-mock dependency -- which caused
various build problems living in boot classpath -- with a new
library that just shims the three Unsafe methods added in Java 8:
fullFence, loadFence, and storeFence. This allows us to eliminate
all calls to those methods from JRuby proper as well as the build
tricks.

Fixes #4103.
81f43ef
@headius headius added a commit to headius/jruby that referenced this issue Dec 13, 2016
@headius headius Remove unsafe-mock from our build altogether.
This commit replaces the unsafe-mock dependency -- which caused
various build problems living in boot classpath -- with a new
library that just shims the three Unsafe methods added in Java 8:
fullFence, loadFence, and storeFence. This allows us to eliminate
all calls to those methods from JRuby proper as well as the build
tricks.

Fixes #4103.
c6a95fb
@headius headius added a commit to headius/jruby that referenced this issue Dec 14, 2016
@headius headius Remove unsafe-mock from our build altogether.
This commit replaces the unsafe-mock dependency -- which caused
various build problems living in boot classpath -- with a new
library that just shims the three Unsafe methods added in Java 8:
fullFence, loadFence, and storeFence. This allows us to eliminate
all calls to those methods from JRuby proper as well as the build
tricks.

Fixes #4103.
64b9553
@headius headius closed this in #4381 Dec 14, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment