-
Notifications
You must be signed in to change notification settings - Fork 11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Large number of temporary shared arrays crashes Julia. Need a more aggressive gc? #35
Comments
Can you post the printed exception stack? On OSX and 0.5 I see 2 different errors. Running the example as is, I get
which goes away if I change
to
With the above change I get a
The first error is a bug in type inference I guess. |
JuliaLang/julia#15419 for the former issue. |
without changes to the above I get:
|
If I add |
strangely, when I leave out the |
This does look gc related, specifically gc not being called soon enough. I just tested on Linux, Julia 0.4.3 with a |
probably the same issue - JuliaLang/julia#15155 |
If you are building 0.4 from source, could you try with branch https://github.com/JuliaLang/julia/tree/amitm/backport_finalizeshmem ? With the initial code and a In my test the memory grows far more slowly. I interrupted the loop after 6000 iterations and performed a |
With your branch and |
I think it is apparent that the problem is being caused by |
SharedArray fails to gc() when called within a sequence of functions? 0.5.0-rc3
Calculating the same number of evaluations (500 x 500) it does not fail while it crashes before the same function is called 500 times And the failure is:
It also happens at 0.4.6, albeit a little different error:
In fact, even without the @sync @parallel in the for o function chisq() it still crashes; it crashes even without addprocs() if @Everywhere gc() called in the second function (at each function calling), it doesn't crash (but long gc() time). Is garbage collection not recognizing function creating SharedArrays being called many times and hitting system's limit of open files? This might be a common case, for example, when adjusting parameters by optimization of a chisquare function - and each simulation being done in parallel, whereas optimization method calling chisquare many times... Or I made something wrong? Best regards p.s.: could reproduce also in juliabox 0.5.0-dev (below) and 0.4.6, but not in a julia 0.4.5 32 bits windows system (also does not fail at 0.5.0rc3 windows 32 bits):
|
probably best to just increase |
I've currently stumbled through this problem while trying to benchmark a function with a Is there a good soul around that could look into fixing this? A bug involving parallelism and a GC is just waaay over my head. 😞 |
Some analysis, with workarounds at the end. There seem to be a couple of related issues here. One is that the The more subtle issue is that each
Then, in a new session, run the same thing except replacing
and the second and third
So, workarounds:
|
After using a
Without this, the shared array was effectively not gc'ed automatically (even after returning from the function). |
@magerton @donm I have a similar issue with SharedArrays that I have not been able to solve. It does not seem to be a memory issue to me because it works about 50% of the times. The issue is here https://discourse.julialang.org/t/systemerror-mmap-the-operation-completed-successfully-issue-with-sharedarrays/31579 . Using @donm 's workaround did not work out for me. |
On my computer julia always crashes on this code:
This triggers a
signal (7): Bus error
which results in the workers beeing killed.I tested this, because I wanted to isolate a memory leak that I suspect. I think it also occurs here, but Julia crashes before I can tell.
I think, if I leave out the Initialization:
by changing the line to
there is no memory leak, but then there are other errors. I am also not sure if the parallel loop is relevant for this.
versioninfo:
The text was updated successfully, but these errors were encountered: