New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JRubyFX application not working when running from jar (works in 1.7, doesn't work in 9k) #4000

Closed
oreoshake opened this Issue Jul 7, 2016 · 17 comments

Comments

Projects
None yet
3 participants
@oreoshake

oreoshake commented Jul 7, 2016

Environment

Provide at least:

  • JRuby version (jruby -v) and command line (flags, JRUBY_OPTS, etc)
    • jruby 9.1.2.0 (2.3.0) 2016-05-26 7357c8f Java HotSpot(TM) 64-Bit Server VM 25.60-b23 on 1.8.0_60-b27 +jit [darwin-x86_64]
  • Operating system and platform (e.g. uname -a)
    • Darwin Neils-MacBook-Pro-4.local 15.5.0 Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64

Other relevant info you may wish to add:

Expected Behavior

The test application at https://github.com/oreoshake/jrubyfx-9k-jar-problems should build and run just fine. $ bundle && rake && java -jar BrakemanPro.jar should launch the app.

Actual Behavior

builds and runs just fine under jruby 1.7.21, but fails under 9k:

$ bundle && rake && java -jar BrakemanPro.jar
Errno::ENOENT: No such file or directory - classpath:/Users/neilmatatall/workspace/jrubyfail/brakeman-pro/jar-bootstrap.rb
          realpath at org/jruby/RubyFile.java:810
  require_relative at uri:classloader:/jruby/kernel/kernel.rb:11
             <top> at classpath:jar-bootstrap.rb:1

But if I unzip the jar and run the file directly, everything is happy:

$ mkdir tmp && cp BrakemanPro.jar tmp/ && cd tmp && unzip BrakemanPro.jar 
... accept the file overwrites ... 
$ ruby jar-bootstrap.rb

There's also a flag for compiling code and using that in the jar, but I ran into errors that I can no longer reproduce so it's disabled by default.

Since we're not using much of JRubyFX to build the jar, I assumed the problem lies with some difference between jruby 1.7 and 9k instead of a jrubyfx problem but I could of course be wrong.

EDIT:

I lied, this could also be a jrubyfx thing since I do call the jarify command:

system 'bundle exec jrubyfx-jarify . --main bin/brakeman_pro_jar.rb BrakemanPro.jar --include-jars --verbose --name "BrakemanPro"'

I will file an issue there too.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Jul 11, 2016

Member

My first guess would be that the require_relative impl in 9k uses realpath where 1.7 doesn't, and realpath may not work right with a jar URL. Maybe @mkristian has some thoughts. I'll try a reduced repro with just realpath

Member

headius commented Jul 11, 2016

My first guess would be that the require_relative impl in 9k uses realpath where 1.7 doesn't, and realpath may not work right with a jar URL. Maybe @mkristian has some thoughts. I'll try a reduced repro with just realpath

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Jul 11, 2016

Member

7e7292e appears to be the commit that introduced File.realpath use in require_relative. This change does appear in JRuby 1.7, in e88a594, but JRuby 1.7.21 was released about a month prior so you wouldn't see it.

Perhaps you can try to reproduce it with 1.7.22 or higher?

Definitely seems likely that this is an issue with File.realpath inside a jar.

Member

headius commented Jul 11, 2016

7e7292e appears to be the commit that introduced File.realpath use in require_relative. This change does appear in JRuby 1.7, in e88a594, but JRuby 1.7.21 was released about a month prior so you wouldn't see it.

Perhaps you can try to reproduce it with 1.7.22 or higher?

Definitely seems likely that this is an issue with File.realpath inside a jar.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Jul 11, 2016

Member

Ok, it does seem like File.realpath has support for classpath URLs, so the bug must be deeper than my quick read-through.

If you could reproduce with -Xbacktrace.style=full passed to JRuby, it would help us pinpoint where inside JRuby it blows up.

Ping @mkristian @kares for suggestions.

Member

headius commented Jul 11, 2016

Ok, it does seem like File.realpath has support for classpath URLs, so the bug must be deeper than my quick read-through.

If you could reproduce with -Xbacktrace.style=full passed to JRuby, it would help us pinpoint where inside JRuby it blows up.

Ping @mkristian @kares for suggestions.

@oreoshake

This comment has been minimized.

Show comment
Hide comment
@oreoshake

oreoshake Jul 12, 2016

Aha! So it's not a jruby 9k issue. Doing the same thing with jruby-1.7.25 produces a similar, but different error:

Errno::ENOENT: No such file or directory - classpath:/Users/neilmatatall/workspace/jrubyfail/brakeman-pro/jar-bootstrap.rb
          realpath at org/jruby/RubyFile.java:820
  require_relative at file:/Users/neilmatatall/workspace/jrubyfail/brakeman-pro/BrakemanPro.jar!/jruby/kernel19/kernel.rb:11
            (root) at classpath:jar-bootstrap.rb:1

If you could reproduce with -Xbacktrace.style=full passed to JRuby, it would help us pinpoint where inside JRuby it blows up.

Any tips on where this would be passed in? I'm afraid the layers of indirection have me at a loss on how to accomplish this. JRUBY_OPTS="-Xbacktrace.style=full" java -jar BrakemanPro.jar doesn't seem to do anything.

oreoshake commented Jul 12, 2016

Aha! So it's not a jruby 9k issue. Doing the same thing with jruby-1.7.25 produces a similar, but different error:

Errno::ENOENT: No such file or directory - classpath:/Users/neilmatatall/workspace/jrubyfail/brakeman-pro/jar-bootstrap.rb
          realpath at org/jruby/RubyFile.java:820
  require_relative at file:/Users/neilmatatall/workspace/jrubyfail/brakeman-pro/BrakemanPro.jar!/jruby/kernel19/kernel.rb:11
            (root) at classpath:jar-bootstrap.rb:1

If you could reproduce with -Xbacktrace.style=full passed to JRuby, it would help us pinpoint where inside JRuby it blows up.

Any tips on where this would be passed in? I'm afraid the layers of indirection have me at a loss on how to accomplish this. JRUBY_OPTS="-Xbacktrace.style=full" java -jar BrakemanPro.jar doesn't seem to do anything.

@oreoshake

This comment has been minimized.

Show comment
Hide comment
@oreoshake

oreoshake Jul 17, 2016

OK, after doing some RTFM'ing, I got more detailed output @headius (9.1.2.0):

Errno::ENOENT: No such file or directory - classpath:/Users/neil/workspace/jrubyfx-9k-jar-problems/.tmp/jar-bootstrap.rb
        getStackTrace at java/lang/Thread.java:1552
     getBacktraceData at org/jruby/runtime/backtrace/TraceType.java:246
         getBacktrace at org/jruby/runtime/backtrace/TraceType.java:47
     prepareBacktrace at org/jruby/RubyException.java:236
             preRaise at org/jruby/exceptions/RaiseException.java:214
             preRaise at org/jruby/exceptions/RaiseException.java:181
               <init> at org/jruby/exceptions/RaiseException.java:111
    newRaiseException at org/jruby/Ruby.java:4175
  newErrnoENOENTError at org/jruby/Ruby.java:3787
             realpath at org/jruby/RubyFile.java:810
                 call at org/jruby/internal/runtime/methods/JavaMethod.java:720
                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:197
         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:313
                 call at org/jruby/runtime/callsite/CachingCallSite.java:163
          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:316
            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:77
            interpret at org/jruby/ir/interpreter/InterpreterEngine.java:86
     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:179
     require_relative at uri:classloader:/jruby/kernel/kernel.rb:11
                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:165
                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:197
         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:313
                 call at org/jruby/runtime/callsite/CachingCallSite.java:163
                <top> at classpath:jar-bootstrap.rb:1
  invokeWithArguments at java/lang/invoke/MethodHandle.java:627
                 load at org/jruby/ir/Compiler.java:111
            runScript at org/jruby/Ruby.java:833
            runScript at org/jruby/Ruby.java:825
          runNormally at org/jruby/Ruby.java:760
          runFromMain at org/jruby/Ruby.java:579
        doRunFromMain at org/jruby/Main.java:425
          internalRun at org/jruby/Main.java:313
                  run at org/jruby/Main.java:242
                 main at org/jruby/Main.java:204
                 main at org/jruby/JarBootstrapMain.java:57

I'm hopping on an ✈️ soon, so I'll drop the output from 1.7.22 "soon".

oreoshake commented Jul 17, 2016

OK, after doing some RTFM'ing, I got more detailed output @headius (9.1.2.0):

Errno::ENOENT: No such file or directory - classpath:/Users/neil/workspace/jrubyfx-9k-jar-problems/.tmp/jar-bootstrap.rb
        getStackTrace at java/lang/Thread.java:1552
     getBacktraceData at org/jruby/runtime/backtrace/TraceType.java:246
         getBacktrace at org/jruby/runtime/backtrace/TraceType.java:47
     prepareBacktrace at org/jruby/RubyException.java:236
             preRaise at org/jruby/exceptions/RaiseException.java:214
             preRaise at org/jruby/exceptions/RaiseException.java:181
               <init> at org/jruby/exceptions/RaiseException.java:111
    newRaiseException at org/jruby/Ruby.java:4175
  newErrnoENOENTError at org/jruby/Ruby.java:3787
             realpath at org/jruby/RubyFile.java:810
                 call at org/jruby/internal/runtime/methods/JavaMethod.java:720
                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:197
         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:313
                 call at org/jruby/runtime/callsite/CachingCallSite.java:163
          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:316
            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:77
            interpret at org/jruby/ir/interpreter/InterpreterEngine.java:86
     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:179
     require_relative at uri:classloader:/jruby/kernel/kernel.rb:11
                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:165
                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:197
         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:313
                 call at org/jruby/runtime/callsite/CachingCallSite.java:163
                <top> at classpath:jar-bootstrap.rb:1
  invokeWithArguments at java/lang/invoke/MethodHandle.java:627
                 load at org/jruby/ir/Compiler.java:111
            runScript at org/jruby/Ruby.java:833
            runScript at org/jruby/Ruby.java:825
          runNormally at org/jruby/Ruby.java:760
          runFromMain at org/jruby/Ruby.java:579
        doRunFromMain at org/jruby/Main.java:425
          internalRun at org/jruby/Main.java:313
                  run at org/jruby/Main.java:242
                 main at org/jruby/Main.java:204
                 main at org/jruby/JarBootstrapMain.java:57

I'm hopping on an ✈️ soon, so I'll drop the output from 1.7.22 "soon".

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Aug 11, 2016

Member

Very strange. Your case seems to be erroring against a classpath URL that contains your full system path. I'm not sure how that would happen. require_relative uses the caller's frame, from Kernel#caller, to determine the source location in the calling method. In my attempt to reproduce the issue, that path appeared to be correct. require_relative then takes that path and uses it to require the file in question. I get to that point in my local test of a hand-made jar. I never get an ENOENT.

I am trying to reproduce with your repo now.

Member

headius commented Aug 11, 2016

Very strange. Your case seems to be erroring against a classpath URL that contains your full system path. I'm not sure how that would happen. require_relative uses the caller's frame, from Kernel#caller, to determine the source location in the calling method. In my attempt to reproduce the issue, that path appeared to be correct. require_relative then takes that path and uses it to require the file in question. I get to that point in my local test of a hand-made jar. I never get an ENOENT.

I am trying to reproduce with your repo now.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Aug 11, 2016

Member

I can reproduce it! Very strange.

Member

headius commented Aug 11, 2016

I can reproduce it! Very strange.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Aug 11, 2016

Member

Oh ho, I think I may have a clue!

My hand-reproduction was still not failing, so I looked at how JarBootstrapMain boots up. It tries to load the jar-bootstrap.rb file from classpath, but uses a URL without a slash in it:

public class JarBootstrapMain {
    public static final String JAR_BOOTSTRAP = "classpath:jar-bootstrap.rb";
    public static void main(String[] args) {
        String[] newArgs = new String[args.length + 1];
        newArgs[0] = JAR_BOOTSTRAP;
        System.arraycopy(args, 0, newArgs, 1, args.length);
        Main.main(newArgs);
    }
}

This makes the jar-bootstrap.rb file look like a relative path, and it gets expanded somewhere along the way (probably by realpath) with the only path we know about: the current directory. So we get a classpath URL pointing at an absolute system path.

This exists in both 1.7 and 9k, but probably only became an issue once we started to use realpath for require_relative. Adding a slash to make these be absolute classpath URLs should fix the issue.

Member

headius commented Aug 11, 2016

Oh ho, I think I may have a clue!

My hand-reproduction was still not failing, so I looked at how JarBootstrapMain boots up. It tries to load the jar-bootstrap.rb file from classpath, but uses a URL without a slash in it:

public class JarBootstrapMain {
    public static final String JAR_BOOTSTRAP = "classpath:jar-bootstrap.rb";
    public static void main(String[] args) {
        String[] newArgs = new String[args.length + 1];
        newArgs[0] = JAR_BOOTSTRAP;
        System.arraycopy(args, 0, newArgs, 1, args.length);
        Main.main(newArgs);
    }
}

This makes the jar-bootstrap.rb file look like a relative path, and it gets expanded somewhere along the way (probably by realpath) with the only path we know about: the current directory. So we get a classpath URL pointing at an absolute system path.

This exists in both 1.7 and 9k, but probably only became an issue once we started to use realpath for require_relative. Adding a slash to make these be absolute classpath URLs should fix the issue.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Aug 11, 2016

Member

Well, that's progress...I managed to get past the ENOENT. Unfortunately now I get the other error I was seeing, a LoadError when it actually tries to do the require within the jar. We must go deeper!

Member

headius commented Aug 11, 2016

Well, that's progress...I managed to get past the ENOENT. Unfortunately now I get the other error I was seeing, a LoadError when it actually tries to do the require within the jar. We must go deeper!

headius added a commit that referenced this issue Aug 11, 2016

headius added a commit that referenced this issue Aug 11, 2016

@oreoshake

This comment has been minimized.

Show comment
Hide comment
@oreoshake

oreoshake commented Aug 11, 2016

so much ❤️ @headius

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Aug 11, 2016

Member

Well 1.7 seems to be working ok without any change, but 9k seems to require this patch to handle require "classpath:/somefile" properly now:

diff --git a/core/src/main/java/org/jruby/runtime/load/LibrarySearcher.java b/core/src/main/java/org/jruby/runtime/load/LibrarySearcher.java
index 7ecb6a2..5381fa9 100644
--- a/core/src/main/java/org/jruby/runtime/load/LibrarySearcher.java
+++ b/core/src/main/java/org/jruby/runtime/load/LibrarySearcher.java
@@ -194,6 +194,10 @@ class LibrarySearcher {
             // uri: are absolute
             return true;
         }
+        if (path.startsWith("classpath:/")) {
+            // classpath URLS are absolute if they start with a slash
+            return true;
+        }
         return new File(path).isAbsolute();
     }

@mkristian @kares Do either of you know why this is needed...or on the flip side, why it doesn't appear to be needed in 1.7?

Member

headius commented Aug 11, 2016

Well 1.7 seems to be working ok without any change, but 9k seems to require this patch to handle require "classpath:/somefile" properly now:

diff --git a/core/src/main/java/org/jruby/runtime/load/LibrarySearcher.java b/core/src/main/java/org/jruby/runtime/load/LibrarySearcher.java
index 7ecb6a2..5381fa9 100644
--- a/core/src/main/java/org/jruby/runtime/load/LibrarySearcher.java
+++ b/core/src/main/java/org/jruby/runtime/load/LibrarySearcher.java
@@ -194,6 +194,10 @@ class LibrarySearcher {
             // uri: are absolute
             return true;
         }
+        if (path.startsWith("classpath:/")) {
+            // classpath URLS are absolute if they start with a slash
+            return true;
+        }
         return new File(path).isAbsolute();
     }

@mkristian @kares Do either of you know why this is needed...or on the flip side, why it doesn't appear to be needed in 1.7?

headius added a commit that referenced this issue Aug 11, 2016

Add classpath:/ as an absolute path prefix. See #4000
This does not appear to be necessary in 1.7, but in 9k if I do not
do this then require 'classpath:/somefile' fails to find the file.
Stepping through it, I see it skips the absolute path logic that
that would have handled it properly and proceeds on to load path
logic, which obviously does not find the file.
@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Aug 11, 2016

Member

I went ahead and pushed the additional change. I'm still not certain why it's needed for 9k and not for 1.7.

And with your repo...success! I think!

$ java -jar BrakemanPro.jar 
Please run `rake reflect` to generate the imports

It appears to be booting properly, but I don't know how to use it :-)

Optimistically calling this one fixed.

Member

headius commented Aug 11, 2016

I went ahead and pushed the additional change. I'm still not certain why it's needed for 9k and not for 1.7.

And with your repo...success! I think!

$ java -jar BrakemanPro.jar 
Please run `rake reflect` to generate the imports

It appears to be booting properly, but I don't know how to use it :-)

Optimistically calling this one fixed.

@headius headius closed this Aug 11, 2016

@oreoshake

This comment has been minimized.

Show comment
Hide comment
@oreoshake

oreoshake Aug 11, 2016

Thanks @headius. We're past this error and on to another require-related bug. I'll make sure it repros on my test repo and create a new issue with ⬆️ debug info.

oreoshake commented Aug 11, 2016

Thanks @headius. We're past this error and on to another require-related bug. I'll make sure it repros on my test repo and create a new issue with ⬆️ debug info.

@headius

This comment has been minimized.

Show comment
Hide comment
@headius

headius Aug 11, 2016

Member

Sounds good, thanks!

Member

headius commented Aug 11, 2016

Sounds good, thanks!

camlow325 added a commit to camlow325/jruby that referenced this issue Aug 17, 2016

Merge from upstream 'jruby-1_7' at 0dc6fa9
* jruby-1_7:
  Use absolute classpath URL when loading jar-bootstrap. See #4000.
  Synchronize around getgrgid call. Fixes #4057.
  Only lock Digest mutex in one place.
  Explicitly close input stream in CompoundJarURLStreamHandler
  Write a log message when findClass IOException is caught
@oreoshake

This comment has been minimized.

Show comment
Hide comment
@oreoshake

oreoshake Aug 18, 2016

Just wanted to report back on this. I was able to workaround the newer issues I stumbled upon and our app is up and running in all envs on jruby-head 🎊 🎉 .

I did run across an issue with require_relative and ../../dir/path type calls. I'll try to create a reproducible test case but this task that I've been putting off for at least a year can be declared complete ❗ 🎉

We'll pin to jruby-head until the next release. Much ❤️ @headius.

oreoshake commented Aug 18, 2016

Just wanted to report back on this. I was able to workaround the newer issues I stumbled upon and our app is up and running in all envs on jruby-head 🎊 🎉 .

I did run across an issue with require_relative and ../../dir/path type calls. I'll try to create a reproducible test case but this task that I've been putting off for at least a year can be declared complete ❗ 🎉

We'll pin to jruby-head until the next release. Much ❤️ @headius.

@griest024

This comment has been minimized.

Show comment
Hide comment
@griest024

griest024 Aug 18, 2016

@oreoshake were you able to get past the Please run 'rake reflect' to generate the imports message?

I created an issue at jruby/jrubyfx#101 but haven't been able to resolve it. Is it related to require_relative issues?

griest024 commented Aug 18, 2016

@oreoshake were you able to get past the Please run 'rake reflect' to generate the imports message?

I created an issue at jruby/jrubyfx#101 but haven't been able to resolve it. Is it related to require_relative issues?

@oreoshake

This comment has been minimized.

Show comment
Hide comment
@oreoshake

oreoshake Aug 18, 2016

@oreoshake were you able to get past the Please runrake reflect to generate the imports message?

Yes, I commented over on jruby/jrubyfx#101 (comment), but something about the way jrubyfx is checking things on disk has changed - maybe this is more of a jrubyfx bug however.

The two lines in question:

$LOAD_PATH.inject(false) { |res,i| res || i.include?(".jar!/META-INF/jruby.home/lib/ruby/")} (bypassed in https://github.com/BrakemanSecurityInc/jrubyfx/commit/5e36e097a33ed275741342d92d91b36628c7c68b)

and

File.size? "#{File.dirname(__FILE__)}/jrubyfx/imports.rb" (bypassed in https://github.com/BrakemanSecurityInc/jrubyfx/commit/d988324b9d5bf3b2c4bad00553d6129576de00fd and https://github.com/BrakemanSecurityInc/jrubyfx/commit/ed2279668d62c63ddba084828a8bda1fb8f26f3a)

oreoshake commented Aug 18, 2016

@oreoshake were you able to get past the Please runrake reflect to generate the imports message?

Yes, I commented over on jruby/jrubyfx#101 (comment), but something about the way jrubyfx is checking things on disk has changed - maybe this is more of a jrubyfx bug however.

The two lines in question:

$LOAD_PATH.inject(false) { |res,i| res || i.include?(".jar!/META-INF/jruby.home/lib/ruby/")} (bypassed in https://github.com/BrakemanSecurityInc/jrubyfx/commit/5e36e097a33ed275741342d92d91b36628c7c68b)

and

File.size? "#{File.dirname(__FILE__)}/jrubyfx/imports.rb" (bypassed in https://github.com/BrakemanSecurityInc/jrubyfx/commit/d988324b9d5bf3b2c4bad00553d6129576de00fd and https://github.com/BrakemanSecurityInc/jrubyfx/commit/ed2279668d62c63ddba084828a8bda1fb8f26f3a)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment