-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
SparkContext giving StackOverflowError #87
Comments
The most likely case is using Spark 3, which we didn't test against yet. Can you try it with Spark 2.4? |
Good point @dfdx Now I`m getting: julia> using Spark
julia> Spark.init()
ERROR: ArgumentError: invalid index: nothing of type Nothing
Stacktrace:
[1] to_index(::Nothing) at ./indices.jl:297
[2] to_index(::Array{String,1}, ::Nothing) at ./indices.jl:274
[3] to_indices at ./indices.jl:325 [inlined]
[4] to_indices at ./indices.jl:322 [inlined]
[5] getindex at ./abstractarray.jl:980 [inlined]
[6] load_spark_defaults(::Dict{Any,Any}) at /root/.julia/packages/Spark/3MVGw/src/init.jl:55
[7] init() at /root/.julia/packages/Spark/3MVGw/src/init.jl:5
[8] top-level scope at REPL[2]:1 What is your setup? |
I'm using Julia 1.5 and all the default settings, which result in Spark 2.4.7. How do you set up Spark version? Do you use SPARK_CONF or SPARK_HOME environment variables for this? |
I ran into the same problem with StackOverflowException and the problem was solved by switching to Java 8. I think there should be a warning when newer Java is used by accident, it was quite hard to find what caused the problems 😅 |
Closing as outdated. |
I'm following the basic steps in the tutorial and found some issues to load the SparkContext.
Btw, the
Spark.init()
only works if setJULIA_COPY_STACKS=1
. I think is good to clarify to the others in the documentation.Setup
Code
Error
The text was updated successfully, but these errors were encountered: