Error when importing diffeqpy parallely in large number of sub-processes.

[sibyjackgrove]

I have a program that spawns a large number of sub-processes and imports diffeqpy within each sub-process using:
from diffeqpy import ode
It works fine when the number of sub-processes is less than 75. But above that I am getting the following error:
"exception: access violation writing 0x000001EC9127C000"

I am using the following:
diffeqpy: 1.2.0
pyjulia: 0.5.7
Julia 1.5.3, 1.6, 1.7

I have a feeling this could be a pyjulia problem. But I just wanted to check here to see if anyone else has encountered it.

[ChrisRackauckas]

Sounds like a pyjulia problem. I'm not sure we can solve it at this repo.

[sibyjackgrove]

@ChrisRackauckas I tried an experiment where I only imported the Main function using pyjulia with many subprocesses. In this case, I was able to start more than 75 sub-processes without encountering the error. I checked the memory footprints of the sub-processes. It seems to importing diffeqpy almost double the memory footprint of the Main function. Could the higher memory footprint when using diffeqpy be the cause of the problem?

[ChrisRackauckas]

Is it dying because you run out of memory? That would be a good way to diagnose that 😅

[sibyjackgrove]

@ChrisRackauckas No, the workstation I am using has 128 GB of memory. And only about 50% memory utilization is there at the time of the error.

[ChrisRackauckas]

@tkf might have an idea.

[tkf]

Maybe you are using fork? See: JuliaPy/pyjulia#448 (comment)

[JacobNeumann88]

Has this issue been resolved? I am trying to use the diffeqpy package in a general fitting software (PyBNF) but without the dask compatibility I am getting nowhere. If I try to use julia prior to submitting the jobs I get a error trying to pickle pycall objects. If I import julia in the function, it is extremely slow and prone to crashing.

[sibyjackgrove]

@JacobNeumann88 No my issue is not resolved. So in my application, I am limited to using less than 50 instances of diffeqpy. I have not used it with Dask.

If I import julia in the function, it is extremely slow and prone to crashing.
I am importing diffeqpy within each sub-process. Depending upon the number of sub-processes it takes anywhere from 10 seconds to 20 seconds. And if I have more than 50 sub-processes my program crashes (not immediately)

[ChrisRackauckas]

Was a PyJulia thing, now gone with PythonCall.