running out of RAM suddenly

[marianstaggl]

Hi,
First of all, thanks for your great Tool. I have a question regarding the RAM consumption. I went through the discussions and I found a similar Topic at #61 but I think my problem is different. Since about one week I keep getting an overflow of RAM memory and it's strange because I can't remember having this issue before. I haven't changed the parameters of the regressor and I'm still using a dataset with the same number of parameters.

At the beginning I left my simulation running multiple days and I haven't noticed any problems but suddenly I run out of RAM within a few hours, even if I reduce the complexity limit. The only thing I remember having changed in the meantime was that I've tried around with warm start, even though I’ve reset to “False” again.

Some details: Right now, a simulation is running, with 25 features (which I know is quite a lot) and a maximum complexity of 10. There are 4 binary operators, 3 unary operators a max depth of 5 while the other parameters are left to default. After running for about 10h I have a RAM consumption of 15Gb and it keeps growing with a slow but steady rate.

Any help, hint or clarification would be very much appreciated. Even if I'm simply mistaken and this is a normal behavior.

Thank you very much for your support.

[MilesCranmer]

Hi @marianstaggl,

Could you answer the following questions to help debug it?

1. Julia version?
2. PySR version?
3. Dataset size?
4. Are you using batching=True or False?
5. Are you using multithreading=True or False?
6. How many procs?

I have personally seen this when (3) my dataset is quite large, in which case you would need to use batching mode. I also see it sometimes for (5) when Julia garbage collection is too passive. This happens in particular in multiprocessing mode (when multithreading=False). I raised an issue here: JuliaLang/julia#50673. But from the PySR side there isn't much to do right now. We could probably add a worker_exeflags argument to PySRRegressor which passes things like heap-size-hint to worker processes, so this garbage collection issue could be manually fixed.

But it could also be some other issue too.

Thanks,
Miles

[MilesCranmer]

The fact that the memory usage is growing at a slow but steady rate indicates that the Julia garbage collector is being too passive. The fix for this is to use --heap-size-hint set to some smaller number, like 1G, which will make the garbage collector work harder to free up memory when it gets close to that limit. I've had great success with this option in the past for related issues.

However I don't think PyJulia (used to connect Python and Julia) lets one configure this option right now. We should add it!

[marianstaggl]

That would be a great!

[MilesCranmer]

I posted an issue here: JuliaPy/pyjulia#537. If you have some time to help (I can get to it eventually but probably not until next week due to start-of-term busy-ness), you can see examples of how to add new options in this pull request: JuliaPy/pyjulia#530 (which adds the trace-compile option).

[marianstaggl]

I have to admit that I've never worked with julia so far but I'll have a look anyways. If I manage to resolve it, i'll let you know.

[marianstaggl]

And thank you very much for your very very quick response ^^ I very much appreciate it!

[MilesCranmer]

It is weird that you haven't seen it before. Were any libraries updated? Or Julia updated? And you are positive none of the settings changed?

[marianstaggl]

When the issues started, I've tried to update julia but that didn't resolve the problem. The only thing I can remember is that I've slightly increased the number of parameters (by 2 but I went back to the original number when the issues started) and that I've tried around with the warm_start option. I've reproduced the initially working settings but it doesn't seem to help.

[marianstaggl]

My initial guess was that I've triggered something within the julia package by changing the warm_start option but I don't know if that is even possible in this case.

[MilesCranmer]

So the immediate fix would be to try running it directly in Julia (https://github.com/MilesCranmer/SymbolicRegression.jl/). I understand that's a tall ask because it's a different language entirely, but depending on urgency you could try running it in Julia (the API is very similar in the Julia backend; here is the quick start: https://github.com/MilesCranmer/SymbolicRegression.jl/#mlj-interface) and starting Julia with --heap-size-hint=1G. It would at least help narrow down the issue to being one of garbage collection (if the issue goes away).

[marianstaggl]

Ok thank you very much! I'll try to run it directly in julia and I'll let you know if the issue persists or if it resolves the problem.

[marianstaggl]

Hi @MilesCranmer,

I've run the symbolic regression in julia and the --heap-size-hint flag seems to help for the multithreading mode. I've set the limit to 1G and I've noticed, that the memory consumption slowly climbs up to ~1.8Gb and then suddenly drops down to ~1Gb again.

I've also tried it in multiprocessing mode but I ran out of ram again but I had a higher limit of 3G and thus I need to check that once more with a lower limit.

Nevertheless the --heap-size-hint flag seems to help in my case!

Thanks!

[MilesCranmer]

Fantastic!

Okay now we just need to add it to the available Julia options. I wonder if I should set the heap size hint by default (say, to 1/10th the user’s memory).

[MilesCranmer]

Yes with multiprocessing mode I will definitely need to set this up manually due to the issue I shared about how heap size limits and Julia addprocs talk to each other.