InitError: JavaCall.JavaCallError("Class Not Found org/apache/log4j/Level") #118

ai-ml-with-kapil · 2023-08-04T11:43:03Z

I am facing issue :

julia> if Sys.isunix()
ENV["JULIA_COPY_STACKS"] = 1
end
1

julia> using JavaCall

julia> using Spark
ERROR: InitError: JavaCall.JavaCallError("Class Not Found org/apache/log4j/Level")
Stacktrace:
[1] _metaclass(class::Symbol)
@ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:383
[2] metaclass(class::Symbol)
@ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:389
[3] jfield(typ::Type{JavaObject{Symbol("org.apache.log4j.Level")}}, field::String, fieldType::Type)
@ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:263
[4] set_log_level(log_level::String)
@ Spark ~/.julia/packages/Spark/89BUd/src/init.jl:8
[5] init(; log_level::String)
@ Spark ~/.julia/packages/Spark/89BUd/src/init.jl:60
[6] init
@ ~/.julia/packages/Spark/89BUd/src/init.jl:16 [inlined]
[7] init()
@ Spark ~/.julia/packages/Spark/89BUd/src/core.jl:30
[8] _include_from_serialized(pkg::Base.PkgId, path::String, depmods::Vector{Any})
@ Base ./loading.jl:831
[9] _require_search_from_serialized(pkg::Base.PkgId, sourcepath::String, build_id::UInt64)
@ Base ./loading.jl:1039
[10] _require(pkg::Base.PkgId)
@ Base ./loading.jl:1315
[11] _require_prelocked(uuidkey::Base.PkgId)
@ Base ./loading.jl:1200
[12] macro expansion
@ ./loading.jl:1180 [inlined]
[13] macro expansion
@ ./lock.jl:223 [inlined]
[14] require(into::Module, mod::Symbol)
@ Base ./loading.jl:1144
during initialization of module Spark

can someone please help me?

djliden · 2023-08-05T16:18:35Z

I also encountered this. MacOS 13.5. I tried it with Julia versions 1.8 and 1.9.

dfdx · 2023-08-05T20:10:58Z

Could you please re-build Spark.jl and post the log here?

] build Spark

djliden · 2023-08-05T20:45:02Z

Thanks for looking! Here is the log.

build.log

dfdx · 2023-08-05T21:50:24Z

I don't see anything strange in your log and can't reproduce it, so let's start with a couple of tests.

Check if JavaCall works fine:

# Julia REPL
using JavaCall

JHashMap = @jimport java.util.HashMap
jmap = JHashMap(())
listmethods(jmap, "put")
jcall(jmap, "put", JObject, (JObject, JObject), "foo", "text value")

Check the versions and env vars:

# bash
java -version
mvn -version
echo $SPARK_HOME

Check that Spark.jl generated all the needed artifacts:

# bash
# you may need to install tree utility or anything to print the contents of the directory
tree ~/.julia/packages/Spark

# alternatively
ls ~/.julia/packages/Spark/*/jvm/sparkjl/target

djliden · 2023-08-06T13:56:20Z

Looks like something with JavaCall...I'll see what I can do in tracking down the issue from that direction.

julia> # Julia REPL
       using JavaCall

julia> JHashMap = @jimport java.util.HashMap
JavaObject{Symbol("java.util.HashMap")}

julia> jmap = JHashMap(())
ERROR: JavaCall.JavaCallError("JVM not initialised. Please run init()")
Stacktrace:
 [1] assertloaded
   @ ~/.julia/packages/JavaCall/MlduK/src/jvm.jl:241 [inlined]
 [2] jnew(::Symbol, ::Tuple{})
   @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:211
 [3] JavaObject{Symbol("java.util.HashMap")}(::Tuple{})
   @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:103
 [4] top-level scope
   @ REPL[3]:1

julia> listmethods(jmap, "put")
ERROR: UndefVarError: `jmap` not defined
Stacktrace:
 [1] top-level scope
   @ REPL[4]:1

julia> JavaCall.init()

julia> JHashMap = @jimport java.util.HashMap
JavaObject{Symbol("java.util.HashMap")}

julia> jmap = JHashMap(())
JavaObject{Symbol("java.util.HashMap")}(JavaCall.JavaLocalRef(Ptr{Nothing} @0x000000012312adc0))

julia> listmethods(jmap, "put")
1-element Vector{JMethod}:
Exception in thread "main" Error showing value of type Vector{JMethod}:
ERROR: JavaCall.JavaCallError("Java Exception thrown, but no details could be retrieved from the JVM")
Stacktrace:
  [1] geterror(allow::Bool)
    @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:411
  [2] geterror
    @ ~/.julia/packages/JavaCall/MlduK/src/core.jl:403 [inlined]
  [3] _jcall(::JMethod, ::Ptr{Nothing}, ::Ptr{Nothing}, ::Type, ::Tuple{})
    @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:373
  [4] jcall(::JMethod, ::String, ::Type, ::Tuple{})
    @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/core.jl:245
  [5] getname
    @ ~/.julia/packages/JavaCall/MlduK/src/reflect.jl:62 [inlined]
  [6] show(io::IOContext{IOBuffer}, method::JMethod)
    @ JavaCall ~/.julia/packages/JavaCall/MlduK/src/reflect.jl:170
  [7] sprint(f::Function, args::JMethod; context::IOContext{Base.TTY}, sizehint::Int64)
    @ Base ./strings/io.jl:112
  [8] sprint
    @ ./strings/io.jl:107 [inlined]
  [9] alignment_from_show
    @ ./show.jl:2817 [inlined]
 [10] alignment(io::IOContext{Base.TTY}, x::JMethod)
    @ Base ./show.jl:2836
 [11] alignment(io::IOContext{Base.TTY}, X::AbstractVecOrMat, rows::Vector{Int64}, cols::Vector{Int64}, cols_if_complete::Int64, cols_otherwise::Int64, sep::Int64, ncols::Int64)
    @ Base ./arrayshow.jl:69
 [12] _print_matrix(io::IOContext{Base.TTY}, X::AbstractVecOrMat, pre::String, sep::String, post::String, hdots::String, vdots::String, ddots::String, hmod::Int64, vmod::Int64, rowsA::UnitRange{Int64}, colsA::UnitRange{Int64})
    @ Base ./arrayshow.jl:207
 [13] print_matrix(io::IOContext{Base.TTY}, X::Vector{JMethod}, pre::String, sep::String, post::String, hdots::String, vdots::String, ddots::String, hmod::Int64, vmod::Int64)
    @ Base ./arrayshow.jl:171
 [14] print_matrix
    @ ./arrayshow.jl:171 [inlined]
 [15] print_array
    @ ./arrayshow.jl:358 [inlined]
 [16] show(io::IOContext{Base.TTY}, #unused#::MIME{Symbol("text/plain")}, X::Vector{JMethod})
    @ Base ./arrayshow.jl:399
 [17] (::REPL.var"#55#56"{REPL.REPLDisplay{REPL.LineEditREPL}, MIME{Symbol("text/plain")}, Base.RefValue{Any}})(io::Any)
    @ REPL /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/REPL/src/REPL.jl:276
 [18] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/REPL/src/REPL.jl:557
 [19] display(d::REPL.REPLDisplay, mime::MIME{Symbol("text/plain")}, x::Any)
    @ REPL /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/REPL/src/REPL.jl:262
 [20] display
    @ /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/REPL/src/REPL.jl:281 [inlined]
 [21] display(x::Any)
    @ Base.Multimedia ./multimedia.jl:340
 [22] print_response(errio::IO, response::Any, show_value::Bool, have_color::Bool, specialdisplay::Union{Nothing, AbstractDisplay})
    @ REPL /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/REPL/src/REPL.jl:0
 [23] (::REPL.var"#57#58"{REPL.LineEditREPL, Pair{Any, Bool}, Bool, Bool})(io::Any)
    @ REPL /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/REPL/src/REPL.jl:287
 [24] with_repl_linfo(f::Any, repl::REPL.LineEditREPL)
    @ REPL /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/REPL/src/REPL.jl:557
 [25] print_response(repl::REPL.AbstractREPL, response::Any, show_value::Bool, have_color::Bool)
    @ REPL /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/REPL/src/REPL.jl:285
 [26] (::REPL.var"#do_respond#80"{Bool, Bool, REPL.var"#93#103"{REPL.LineEditREPL, REPL.REPLHistoryProvider}, REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::Any, ok::Bool)
    @ REPL /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/REPL/src/REPL.jl:899
 [27] #invokelatest#2
    @ ./essentials.jl:816 [inlined]
 [28] invokelatest
    @ ./essentials.jl:813 [inlined]
 [29] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/REPL/src/LineEdit.jl:2647
 [30] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL /Applications/Julia-1.9.app/Contents/Resources/julia/share/julia/stdlib/v1.9/REPL/src/REPL.jl:1300
 [31] (::REPL.var"#62#68"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:514

versions:

shell> java -version
java version "11.0.19" 2023-04-18 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.19+9-LTS-224)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.19+9-LTS-224, mixed mode)

shell> mvn -version
Apache Maven 3.9.4 (dfbb324ad4a7c8fb0bf182e6d91b0ae20e3d2dd9)
Maven home: /opt/homebrew/Cellar/maven/3.9.4/libexec
Java version: 11.0.19, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk-11.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.5", arch: "aarch64", family: "mac"

djliden · 2023-08-06T14:14:45Z

I have a bit more testing to do but at least part of the issue appears to relate back to this warning in the JavaCall docs:

Setting JULIA_COPY_STACKS=yes in startup.jl will not work. It must be set before Julia starts. On *nix based systems, this can be done from the shell by using $ JULIA_COPY_STACKS=yes julia from a shell.

https://juliainterop.github.io/JavaCall.jl/

Now I'm hitting:

julia> spark = SparkSession.builder.appName("Main").master("local").getOrCreate()

[45697] signal (11.2): Segmentation fault: 11
in expression starting at REPL[3]:1
unknown function (ip: 0x28002f3ec)
Allocations: 4487659 (Pool: 4482903; Big: 4756); GC: 6
[1]    45697 segmentation fault  JULIA_COPY_STACKS=yes julia

Which seems unrelated to the original issue. So I still have some things to figure out but perhaps the original issue can be deemed resolved by setting JULIA_COPY_STACKS=yes before startup.jl

dfdx · 2023-08-08T09:20:16Z

Regarding segmentation fault, can you try a different combination of Julia and JDK? IIRC, we had a similar issue with Julia 1.2-1.5, and also a few issues with OpenJDK. Currently, Julia 1.9.2 and OpenJDK 11.0.19 work for me on Ubuntu 20.04, but I don't have MacOS to test this setup there.

djliden · 2023-08-08T16:42:41Z

I'll dig into that further a little later this week; thank you for the pointers.

djliden · 2023-08-19T17:24:37Z

Apologies for the slow response. I believe everything is working for me now. I'm not sure which exact combination of fixes got everything working; I hope to start over later and come up with a reproducible guide. But for reference, the major changes were:

installed OpenJDK 11 (after uninstalling JDK 11)
Made sure to start Julia with JULIA_COPY_STACKS=yes julia (see https://juliainterop.github.io/JavaCall.jl/)

At this point I have:

$ java -version              
openjdk version "11.0.20" 2023-07-18
OpenJDK Runtime Environment Homebrew (build 11.0.20+0)
OpenJDK 64-Bit Server VM Homebrew (build 11.0.20+0, mixed mode)

$ mvn -version
Apache Maven 3.9.4 (dfbb324ad4a7c8fb0bf182e6d91b0ae20e3d2dd9)
Maven home: /opt/homebrew/Cellar/maven/3.9.4/libexec
Java version: 11.0.20, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk@11/11.0.20/libexec/openjdk.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "13.5", arch: "aarch64", family: "mac"

Then I was able to do

$ JULIA_COPY_STACKS=yes julia

julia> using Spark
julia> spark = SparkSession.builder.appName("Main").master("local").getOrCreate()
julia> df = spark.createDataFrame([["Alice", 19], ["Bob", 23]], "name string, age long")
+-----+---+
| name|age|
+-----+---+
|Alice| 19|
|  Bob| 23|
+-----+---+

pawankukreja01 · 2023-10-15T21:32:08Z

The error message suggests that the JavaCall package is unable to find the org/apache/log4j/Level class. This error is usually caused by a missing dependency or a version mismatch between the packages.

Here are some steps you can try to resolve this issue:

Check if you have installed the required dependencies for the JavaCall and Spark packages. You can do this by running the following command in your Julia REPL:

Julia
This code is AI-generated. Review and use carefully.
JavaCall Spark

This will show you the list of installed packages and their versions. Make sure that all the required dependencies are installed and up-to-date.

Try updating the JavaCall package to the latest version by running the following command in your Julia REPL:

Julia
This code is AI-generated. Review and use carefully.
up JavaCall

If updating the package doesn’t work, try uninstalling and reinstalling the JavaCall package by running the following commands in your Julia REPL:

Julia
This code is AI-generated. Review and use carefully
rm JavaCall
add JavaCall

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

InitError: JavaCall.JavaCallError("Class Not Found org/apache/log4j/Level") #118

InitError: JavaCall.JavaCallError("Class Not Found org/apache/log4j/Level") #118

ai-ml-with-kapil commented Aug 4, 2023

djliden commented Aug 5, 2023

dfdx commented Aug 5, 2023

djliden commented Aug 5, 2023

dfdx commented Aug 5, 2023

djliden commented Aug 6, 2023

djliden commented Aug 6, 2023 •

edited

Loading

dfdx commented Aug 8, 2023

djliden commented Aug 8, 2023

djliden commented Aug 19, 2023 •

edited

Loading

pawankukreja01 commented Oct 15, 2023

InitError: JavaCall.JavaCallError("Class Not Found org/apache/log4j/Level") #118

InitError: JavaCall.JavaCallError("Class Not Found org/apache/log4j/Level") #118

Comments

ai-ml-with-kapil commented Aug 4, 2023

djliden commented Aug 5, 2023

dfdx commented Aug 5, 2023

djliden commented Aug 5, 2023

dfdx commented Aug 5, 2023

djliden commented Aug 6, 2023

djliden commented Aug 6, 2023 • edited Loading

dfdx commented Aug 8, 2023

djliden commented Aug 8, 2023

djliden commented Aug 19, 2023 • edited Loading

pawankukreja01 commented Oct 15, 2023

djliden commented Aug 6, 2023 •

edited

Loading

djliden commented Aug 19, 2023 •

edited

Loading