[BUG] *Windows SystemError: <PyCall.jlwrap on basic example*

[strifinopoulos]

I have done a fresh installation on windows (with pip) and I am running the basic example provided in the Introduction. I am getting a JULIA error. Thanks in advance for any help!

Version:

• Julia version [1.8.3]
• Python version [3.10.6]
• PySR version [0.11.11]

Error message

C:\tools\Anaconda3\envs\env_ai\lib\site-packages\pysr\sr.py:1257: UserWarning: Note: it looks like you are running in Jupyter. The progress bar will be turned off.
warnings.warn(
Traceback (most recent call last):

File "C:\tools\Anaconda3\envs\env_ai\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
exec(code, globals, locals)

File "c:\users\gorth\untitled0.py", line 25, in
model.fit(X, y)

File "C:\tools\Anaconda3\envs\env_ai\lib\site-packages\pysr\sr.py", line 1792, in fit
self._run(X, y, mutated_params, weights=weights, seed=seed)

File "C:\tools\Anaconda3\envs\env_ai\lib\site-packages\pysr\sr.py", line 1652, in run
self.raw_julia_state = SymbolicRegression.EquationSearch(

SystemError: <PyCall.jlwrap (in a Julia function called from Python)
JULIA: SystemError: opening file "hall_of_fame_2022-12-17_011150.694.csv": Invalid argument
Stacktrace:
[1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
@ Base .\error.jl:176
[2] #systemerror#80
@ .\error.jl:175 [inlined]
[3] systemerror
@ .\error.jl:175 [inlined]
[4] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Bool, append::Nothing)
@ Base .\iostream.jl:293
[5] open(fname::String, mode::String; lock::Bool)
@ Base .\iostream.jl:356
[6] open(fname::String, mode::String)
@ Base .\iostream.jl:355
[7] open(::SymbolicRegression.var"#48#77"{Options{typeof(loss), Int64, 0.86, 10}, Vector{PopMember{Float32}}, SymbolicRegression.CoreModule.DatasetModule.Dataset{Float32}}, ::String, ::Vararg{String}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ Base .\io.jl:382
[8] open
@ .\io.jl:381 [inlined]
[9] EquationSearch(::SymbolicRegression.CoreModule.ProgramConstantsModule.SRThreaded, datasets::Vector{SymbolicRegression.CoreModule.DatasetModule.Dataset{Float32}}; niterations::Int64, options::Options{typeof(loss), Int64, 0.86, 10}, numprocs::Nothing, procs::Nothing, addprocs_function::Nothing, runtests::Bool, saved_state::Nothing)
@ SymbolicRegression C:\Users\gorth.julia\packages\SymbolicRegression\37l4B\src\SymbolicRegression.jl:751
[10] EquationSearch(datasets::Vector{SymbolicRegression.CoreModule.DatasetModule.Dataset{Float32}}; niterations::Int64, options::Options{typeof(loss), Int64, 0.86, 10}, parallelism::String, numprocs::Nothing, procs::Nothing, addprocs_function::Nothing, runtests::Bool, saved_state::Nothing)
@ SymbolicRegression C:\Users\gorth.julia\packages\SymbolicRegression\37l4B\src\SymbolicRegression.jl:383
[11] EquationSearch(X::Matrix{Float32}, y::Matrix{Float32}; niterations::Int64, weights::Nothing, varMap::Vector{String}, options::Options{typeof(loss), Int64, 0.86, 10}, parallelism::String, numprocs::Nothing, procs::Nothing, addprocs_function::Nothing, runtests::Bool, saved_state::Nothing, multithreaded::Nothing)
@ SymbolicRegression C:\Users\gorth.julia\packages\SymbolicRegression\37l4B\src\SymbolicRegression.jl:320
[12] #EquationSearch#21
@ C:\Users\gorth.julia\packages\SymbolicRegression\37l4B\src\SymbolicRegression.jl:345 [inlined]
[13] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Any, NTuple{8, Symbol}, NamedTuple{(:weights, :niterations, :varMap, :options, :numprocs, :parallelism, :saved_state, :addprocs_function), Tuple{Nothing, Int64, Vector{String}, Options{typeof(loss), Int64, 0.86, 10}, Nothing, String, Nothing, Nothing}}})
@ Base .\essentials.jl:731
[14] pyjlwrap_call(f::Function, args::Ptr{PyCall.PyObject_struct}, kw::Ptr{PyCall.PyObject_struct})
@ PyCall C:\Users\gorth.julia\packages\PyCall\ygXW2\src\callback.jl:32
[15] pyjlwrap_call(self_::Ptr{PyCall.PyObject_struct}, args_::Ptr{PyCall.PyObject_struct}, kw_::Ptr{PyCall.PyObject_struct})
@ PyCall C:\Users\gorth.julia\packages\PyCall\ygXW2\src\callback.jl:44>

[MilesCranmer]

Does that file hall_of_fame_2022-12-17_011150.694.csv exist? Are there any weird permissions for that file or that directory?

[MilesCranmer]

Does the error appear if you run it from Python directly, rather than Jupyter?

[strifinopoulos]

So I am running from Spyder actually, I am not sure why it thinks that I am running it from Jupyter.

Every run generates hall_of_fame_.csv , hall_of_fame_.csv.bkup and hall_of_fame_*.pkl files at the directory of execution. Inside the csv there are 9 results (the last three have losses of ~3).

Today I tried to run the script from a python shell and I got the same error, but this time the csv file contains results up to loss of E-13 basically finding the result. It doesn't print though the result in the shell.

[MilesCranmer]

What are the contents of the csv files? Are they empty? It looks like Julia is basically failing to open the csv file for writing. I think it’s a permission issue, maybe Julia and Python have different permission settings…?

[strifinopoulos]

So the results are inside the csv files as well. So practically, I think I do have the results, but only if I run with the shell and not on Spyder. The error appears in both cases though. What kind of permission issue could it be? not sure how to check Julia's settings (it's already in the PATH)

[MilesCranmer]

Interesting. I am confused why it is having trouble opening the file then... It almost looks like the first time it opens the file and writes to it, it's fine, because the file doesn't exist. But the second time it opens the file, it complains...

JULIA: SystemError: opening file "hall_of_fame_2022-12-17_011150.694.csv": Invalid argument
Stacktrace:
[1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
@ Base .\error.jl:176

It's so strange that it says "invalid argument" here. Maybe the version of Julia which is actually being called is an earlier version, which is missing one of the parameters of the open command?

Maybe try running the Julia example directly: https://github.com/MilesCranmer/SymbolicRegression.jl/#quickstart and see if that works or not?

[strifinopoulos]

Indeed! It is a Julia issue. Maybe downgrade to an earlier working version?

[MilesCranmer]

Hm, I guess the precompilation experiences the same issue, because it attempts to run a small search to find relevant functions to compile.

Even though the precompilation failed, you should be able to run it normally. Could you try running the example without precompilation?

[strifinopoulos]

When I run "using SymbolicRegression" I get the same error and I cannot run the rest of the code (I get "UndefVarError: SymbolicRegression not defined").

[MilesCranmer]

You can try adding the package with this change: https://discourse.julialang.org/t/how-to-disable-automatic-precompilation-of-packages-in-julia-1-6/57053

[strifinopoulos]

I did use ENV["JULIA_PKG_PRECOMPILE_AUTO"]=0, and then Pkg.add("SymbolicRegression") returns "No Changes to C:\Users\gorth.julia\environments\v1.8\Projects.toml" and "...Manifest.toml". Then unfortunately I get the same Permission denied error when I try "using SymbolicRegression".
Since the code does perform the operation I guess it is ok for now, but hopefully there is a fix at some point (maybe a future version of julia?)

[MilesCranmer]

Very strange. Well, worst case, you can always use the docker container until figuring out what went wrong with the permissions?

[strifinopoulos]

Yes, for the time being I am relying on that. Thanks!

[strifinopoulos]

May I ask if opening the bkup file is done in the very end of the operation? Because a lot of times PySR halts just after one iteration. I am not sure if some internal stopping condition is satisfied or if it is due to this bug.

[MilesCranmer]

It's probably due to the bug. The equation export happens here: https://github.com/MilesCranmer/SymbolicRegression.jl/blob/de0e51c89ae1a88cfc5a616a9d101b7a4d331dcb/src/SymbolicRegression.jl#L755-L766. This occurs at the end of every iteration.

My initial idea was that this was a permissions issue in your folder you are running this. i.e., the Julia process doesn't have permission to write to that folder (maybe the way you installed Julia gave it weird permissions settings?). Perhaps it could be something in the filename – does your current working directory have spaces anywhere? e.g., like "\My Documents\..."?

[MilesCranmer]

You could try just loading up PyJulia and seeing if you can do this manually. Does the following code work?

from pysr.julia_helpers import init_julia
jl = init_julia()

jl.eval("""open("hall_of_fame.csv", "w") do io
    println(io, "Complexity,Loss,Equation")
end""")

This will open the file "hall_of_fame.csv" in the current directory and try writing to it.

[strifinopoulos]

I actually did not have PyJulia installed (did I miss it in the installation instructions?). I don't get the error anymore! Thanks a lot!

PS: I now get an "OSError: exception: access violation reading 0x0000026E9A4A1000" when the run is finished, but I think that has to do with utf-8 encoding and it definitely does not have any impact on the execution.

[MilesCranmer]

Weird... Installing PySR on pip or conda should also install PyJulia. All PySR calls to Julia are made using PyJulia, so I am not sure how you were able to run the other parts, or see the errors you saw in the original post (which are printed via PyJulia).

If that access violation error gives you any problems please raise a separate issue - that is definitely not expected!

[MilesCranmer]

For the access error, see the workaround in #266