Interaction between addprocs and --heap-size-hint

[MilesCranmer]

I think this may be a feature request/issue report rather than a question, so I am posting here rather than on the discourse.

I am basically wondering: do Distributed.addprocs and the CLI flag --heap-size-hint interact with eachother, and, if so, how? If I specify --heap-size-hint to the head Julia process, and then dynamically create Julia processes with addprocs, how (if?) does it get split up among each of the processes created?

My current workaround is to pass --heap-size-hint to exeflags in each addprocs call, and divvy up the memory into smaller chunks, but this seems a bit hacky.

Just wondering about this because I've ran into various segfaults which I've never been able to track down (#47957). I realized that memory usage seems to explode when using multiple processes which are allocated dynamically, but the memory can be controlled by setting, e.g., --heap-size-hint when dynamically creating the workers.

[MilesCranmer]

@gbaraldi do you know if your PR #50144 would help with this?

[elextr]

Let me just leave my observations from experience hacking A. N. Other (unnamed) language`s GC, on Linux.

On Linux its very difficult to determine how much memory is available due to its over committing behaviour and the unavailability of information about memory usage of everything else running on the machine. But that is the information the OOM killer uses to decide to act, and it acts on some heuristic of biggest and newest. So its hard for any single process GC to second guess OOM to decide how hard it should work to keep memory down to avoid being killed. User provided hints are the most reasonable method to tell the GC how hard to work, not let the GC try to calculate for itself, the Linux system just can't provide what it needs to do the calculation (or get a bigger machine and turn OOM off 😁).

Also the OOM killer looks at what to kill by cgroup, so if the workers are in another cgroup(s) from the parent, what gets killed may differ.

My current workaround is to pass --heap-size-hint to exeflags in each addprocs call, and divvy up the memory into smaller chunks, but this seems a bit hacky.

Far from hacky that would seem to be the right solution (specify manually), Julia can't know what the user is going to run on the workers and therefore what their memory needs are. Maybe the parent needs lots of memory and the workers will be heavily compute bound on small amounts of memory, or vice versa.

[MilesCranmer]

Thanks. I guess I'm just wondering what the default behavior is. Does the heap-size-hint propagate to workers, or does it get split up (from your message, my guess is no), or do the worker processes have no heap-size-hint? Furthermore, is this behavior different if you launch with -p {PROCS} versus if you allocate them dynamically with addprocs?

At the very least this seems like a documentation issue. I really have no idea what to expect my workers to do here...

For the record I don't really see this on a real workstation (though I do see memory blow up; it doesn't hit OOM errors yet). The more urgent issue is on GitHub actions, where my multiprocessing tests can sometimes segfault for inexplicable reasons. My guess is that it's an OOM issue.

[elextr]

from your message, my guess is no

I should have been clear, my searches found the number from --heap-size-hint is only ever used in gc.c. So its not passed to workers automatically AFAICT with -p or addprocs()

I realized that memory usage seems to explode when using multiple processes which are allocated dynamically, but the memory can be controlled by setting, e.g., --heap-size-hint when dynamically creating the workers.

This finding would seem to confirm that. So workers just behave as if no size hint exists unless addprocs() with heap-size-hint in exeflags is used.

[ufechner7]

Well, but that is not good, right? I mean, that is not what a user would expect. So it should be improved.

[oscardssmith]

Agreed.

[elextr]

@ufechner7 @oscardssmith improvement is good, but what do you suggest is an improvement?

[oscardssmith]

IMO addprocs should probably use the heap limit unless otherwise overruled. (in a dream world they would share the same heap limit capacity but that's probably hard).

[evetion]

I've hit this too, addprocs don't automatically share much of the main process. Since 1.9 they at least share the package environment (#43270), but maybe they should share also (some) exeflags?

The minimum fix here is to update the documentation with a warning.

[elextr]

@oscardssmith personally I'm not sure making all heap-size-hints the same is really an improvement, but maybe its just my experience of commonly having large size asymmetries between the parent and workers. At least its simple so its easily documented, and so long as it can be overruled, and that is documented, then the actual default behaviour is not so important.

Or the simplest solution is to leave it as is and as @evetion says document the behaviour and note the use of exeflags.

[oscardssmith]

to me it seems like using the same limit is the right default (since that's the guess we can make). We should also definitely document the way to overrule.

[JoelNelson]

@MilesCranmer you mention the workaround for adding heap size hint as an exeflag when doing addprocs. It doesn't seem to have affect and I'm experiencing a similar issue. Is this the syntax you used?

addprocs(10; exeflags=`--heap-size-limit=2G`)

[MilesCranmer]

@JoelNelson it should be --heap-size-hint rather than limit. Also make sure to also set the heap-size-hint on the head process as well so it doesn't also blow up memory. But yeah that fixed the issue for me on Julia 1.9.2 (maybe also try setting it smaller to see whether it has an effect or not?)

My full call was (on a Rocky Linux Slurm cluster)

julia> using Distributed, ClusterManagers

julia> procs = addprocs_slurm(8 * 64; exeflags=`--threads=1 --heap-size-hint=1G`)

which fixed my memory from exploding

[JoelNelson]

@MilesCranmer thanks!! I was doing dev inside a client VDI and was re-typing out on my normal PC and accidentally used limit instead of hint. However, I trimmed down the hint to 2G from 10G and that did the trick!

Appreciate the quick response!

[MilesCranmer]

Is anybody working on this? It seems like a major issue with garbage collection in the distributed interface. I run into OOM errors all the time from this unless I manually set up the correct --heap-size-hint.

[vtjnash]

The new GC pacer algorithm is supposed to help with this, as it dynamically allocates more memory to processes that are getting more CPU time, and forces processes that are getting less CPU (e.g. because they are starting to bottleneck on memory and swap) to use less memory