GC Ballast / Task Ballast question

[hostilefork]

I changed some code that started creating a lot of series--one per function call (though most of the time this series should just be GC'd). Unfortunately, tight loops wind up making quite a few before the GC has a chance to get called. So it's easy to hit high water marks in memory that don't get released.

There are some things that can be done about it, and I expected it to be a performance hit... but I saw something strange with it seeming to GC on just about every cycle. Then I saw this code here:

https://github.com/metaeducation/ren-c/blob/8d29c1169c69dc5b9e733bfe39fa7c0f7b3220dd/src/core/m-gc.c#L1244

The GC_Ballast has always been a bit of a mystery to me, and I imagine you have looked at it . But when I put some debugging code in to print out numbers at this point I got back things like this:

PG_Mem_Usage = 14515476
TASK_BALLAST = 60
GC_Ballast increased by half -52
Recycled 0
PG_Mem_Usage = 14515476
TASK_BALLAST = 90
GC Ballast cut in half 298
Recycled 2
PG_Mem_Usage = 14515476
TASK_BALLAST = 45
GC_Ballast increased by half -11
Recycled 0
PG_Mem_Usage = 14515476
TASK_BALLAST = 66
GC_Ballast left alone 114
Recycled 1
PG_Mem_Usage = 14515476
TASK_BALLAST = 66
GC_Ballast increased by half -46
Recycled 0
PG_Mem_Usage = 14515476
TASK_BALLAST = 99
GC_Ballast left alone 147

So it has gotten to the point where it is GC'ing very frequently with a tiny TASK_BALLAST that is just sort of flipping around at small numbers, and it's recycling one or 2 series at a time.

If you've looked at this code can you give some insight into what it should be doing? Is this expected behavior that it thinks 14 MB is too much and really won't keep going without a GC every cycle?

[zsx]

Yes, I added this water mark checking. I think the problem is how TASK_BALLAST
got to 60 bytes in the first place. I think it started with 3,000,000 (
https://github.com/metaeducation/ren-c/blob/8d29c1169c69dc5b9e733bfe39fa7c0f7b3220dd/src/core/b-init.c#L841
and
https://github.com/metaeducation/ren-c/blob/master/src/include/mem-pools.h#L111
).

The reason I added this was because our application uses a lot of memory
(over 1 Gigabytes), and it took forever to startup because of GC, so I
added this logic to adjust GC_Ballast as needed, and it worked well,
perhaps more testing is needed for low memory usage programs (perhaps
adding a min ballast?).

Best Regards

Shixin Zeng

On Sat, Jan 16, 2016 at 11:45 AM, Brian Dickens notifications@github.com
wrote:

I changed some code that started creating a lot of series--one per
function call *(though most of the time this series should just be GC'd).
Unfortunately, tight loops wind up making quite a few before the GC has a
chance to get called. So it's easy to hit high water marks in memory that
don't get released.

There are some things that can be done about it, and I expected it to be a
performance hit... but I saw something strange with it seeming to GC on
just about every cycle. Then I saw this code here:

https://github.com/metaeducation/ren-c/blob/8d29c1169c69dc5b9e733bfe39fa7c0f7b3220dd/src/core/m-gc.c#L1244

The GC_Ballast has always been a bit of a mystery to me, and I imagine you
have looked at it . But when I put some debugging code in to print out
numbers at this point I got back things like this:

PG_Mem_Usage = 14515476
TASK_BALLAST = 60
GC_Ballast increased by half -52
Recycled 0
PG_Mem_Usage = 14515476
TASK_BALLAST = 90
GC Ballast cut in half 298
Recycled 2
PG_Mem_Usage = 14515476
TASK_BALLAST = 45
GC_Ballast increased by half -11
Recycled 0
PG_Mem_Usage = 14515476
TASK_BALLAST = 66
GC_Ballast left alone 114
Recycled 1
PG_Mem_Usage = 14515476
TASK_BALLAST = 66
GC_Ballast increased by half -46
Recycled 0
PG_Mem_Usage = 14515476
TASK_BALLAST = 99
GC_Ballast left alone 147

So it has gotten to the point where it is GC'ing very frequently with a
tiny TASK_BALLAST that is just sort of flipping around at small numbers,
and it's recycling one or 2 series at a time.

If you've looked at this code can you give some insight into what it
should be doing? Is this expected behavior that it thinks 14 MB is too
much and really won't keep going without a GC every cycle?

—
Reply to this email directly or view it on GitHub
#32.

[giuliolunati]

@zsx I see you decrease TASK_BALLAST if is low relative to GC_BALLAST and increase if is high. That make sense? I'd expect viceversa... ?

[hostilefork]

The part I find puzzling is that a GC is triggered by the ballast reaching 0 or negative:

https://github.com/metaeducation/ren-c/blob/a1dbe1dbf6d22b7f2844d23d0d3adca26808cb22/src/core/m-pools.c#L722

So I don't understand what the comparisons are supposed to mean in this case. What good is comparing 0 or a small negative number to anything?

I guess the thing I'd be looking for here is "the story", in the form of a comment on GC_Ballast, of what the logic is supposed to be. And because it is coming up, this question of what to do with the situation I'm creating that doesn't seem like it should be unmanageable. Basically:

 my-nothing-function: does []
 loop 10000000 [
     my-nothing-function
 ]

I'm having a situation where a REBSER is being created on each call (no memory for series data, just the REBSER node). And a REBSER is not very big, it is the sizeof(REBVAL) plus a little, and it comes out of a pool so it's pretty fast. It's not needed usually when the call is over (as in above, there's no point to it), so it should go to the GC and be cleaned up pretty soon.

But this is wreaking havoc because a loop like the above will not have any printing or other operations which would stop it from just consuming all memory with the unused REBSERs before doing a GC, and once that memory is taken it will not be given back.

But I don't think what I'm doing is that "unreasonable" in the sense that it seems the system should be kicking in the GC automatically at some point when it sees a lot of allocations. So hopefully whatever the story is with the logic could accommodate such a situation.

[zsx]

I think (I didn't come up with this idea, BTW) the logic behind GC_Ballast is:

1. GC_Ballast starts with TASK_BALLAST, which is 3,000,000
2. GC_Ballast decreases when a new REBSER is allocated, and increase when a REBSER is released.
3. When GC_Ballast reaches zero, GC kicks in in the hope that some memory can be recycled.
   So basically, when more than 3,000,000 bytes (if no user adjustment by changing TASK_BALLAST or running recycle/ballast) memory is allocated, GC will kick in. However, for some applications, 3,000,000 is so small that GC kicks in very often and it has not much memory to release, which wastes a lot of CPU time. So I introduced water marks for GC_Ballast.

As explained above, GC will kick in when GC_Ballast gets to zero, so GC_Ballast indicates how much more memory can be allocated before next GC kicks in. The smaller GC_Ballast is, the more often GC kicks in, so increasing GC_Ballast when it's small makes GC run less often. However, if it's too big (when last GC released a lot of memory), GC might not run often enough and result in big memory pools. Whether it's too small or too big is determined by comparing to TASK_BALLAST, which can be changed by the user. I opted it to be too small when it's smaller than half of TASK_BALLAST, and it's too big if it's bigger than 2 times of TASK_BALLAST. I think the problem @hostilefork saw can be solved by putting a limit on the minimum value TASK_BALLAST can be.

[zsx]

@giuliolunati I am actually increasing GC_Ballast if it's low compared to TASK_BALLAST, and decrease it when it's high. Note this line: https://github.com/metaeducation/ren-c/blob/8d29c1169c69dc5b9e733bfe39fa7c0f7b3220dd/src/core/m-gc.c#L1262

[hostilefork]

I'm not sure why I only hit this some of the time, but I'm being forced to look at it because the interpreter grinds to a crawl and can't run the tests... :-(

GC_Ballast decreases when a new REBSER is allocated, and increase when a REBSER is released.

So it does increase when a REBSER is released. The issue seems to be is that if a large amount of memory gets released, then GC Ballast becomes bigger than the task ballast. So it causes this condition to kick in...

if (GC_Ballast >= VAL_INT64(TASK_BALLAST) * 2) {
    //reduce ballast by half
    VAL_INT64(TASK_BALLAST) /= 2;
}

...BUT it doesn't kick in just once. It kicks in every time a recycle runs (which for instance in the test suite is done on each test). It only takes a few times of being divided by 2 for TASK_BALLAST to get driven down to a 0-ish point. And it never recovers.

However, for some applications, 3,000,000 is so small that GC kicks in very often and it has not much memory to release, which wastes a lot of CPU time. So I introduced water marks for GC_Ballast.

Seems that you're saying you were chewing through memory a lot--such that 3MB of allocations happens often, but you didn't want GC to run in that case? You wanted bigger numbers to trigger it (say 30MB, or whatever)...so you wanted to grow the TASK_BALLAST to a larger number, where each GC exhaustion would use that larger number. Then you wanted to have a way to back it down again if it seemed a bunch of memory had been released.

However, the mechanism is backfiring. Because if a single allocation is extremely large (as with my debug-instrumented R3 executable getting loaded in a BINARY!) then its free will be very large. And then manual calls to recycle will drive TASK_BALLAST to 0-ish, which then keeps resetting GC_Ballast to 0-ish, perpetuating the cycle.

(Note: also it may be (?) that when you added the conditions you were interpreting it as if at that point in the code, the GC_Ballast had its initialization value from the last recycle vs. the current one. Because in the non-manual-recycle case, these are comparisons against the exhausted GC_Ballast. So you're comparing against 0--which was my initial confused observation.)

I don't know if there's a good automatic heuristic to pick from here. Garbage collection is a deep topic, and what's really needed here is an incremental GC and pretty much an overhaul of it. Perhaps we could drop the heuristics in favor of just using manual control, so that in your app you could use the recycle/ballast to pick a number fitting for your situation?

[zsx]

I think the easiest way to fix "resetting GC_Ballast to 0-ish" would be introducing a minimum (non-zero) TASK_BALLAST, which could probably be the one it initially started, and could be set by recycle/ballast

[hostilefork]

I think the easiest way to fix "resetting GC_Ballast to 0-ish" would be introducing a minimum (non-zero) TASK_BALLAST, which could probably be the one it initially started, and could be set by recycle/ballast

I don't mind there being some bounds of minimum or maximum of "water marks". But if that's going to be added then the logic of the current water mark has to make sense...

Unless I'm reading something wrong here...I'll ask again if you can explain what this code is meant to do if--in the main relevant case of Recycle (automatic vs. manual)--a GC does not run unless GC_Ballast is <= 0:

    if (GC_Ballast <= VAL_INT32(TASK_BALLAST) / 2
        && VAL_INT64(TASK_BALLAST) < MAX_I32) {
        //increasing ballast by half
        VAL_INT64(TASK_BALLAST) /= 2;
        VAL_INT64(TASK_BALLAST) *= 3;
    } else if (GC_Ballast >= VAL_INT64(TASK_BALLAST) * 2) {
        //reduce ballast by half
        VAL_INT64(TASK_BALLAST) /= 2;
    }

    // avoid overflow
    if (
        VAL_INT64(TASK_BALLAST) < 0
        || VAL_INT64(TASK_BALLAST) >= MAX_I32
    ) {
        VAL_INT64(TASK_BALLAST) = MAX_I32;
    }

Is there something about the GC_Ballast at this point in the code that suggests to you that it carries useful information for adapting the value to which GC_Ballast will be reset? (?)

[hostilefork]

While there's no question the GC could be improved and understood better, this code created problems...and was commented out for many years (to revert to R3-Alpha), which resolved the problems it caused. So I've deleted it in Ren-C and will close this issue:

metaeducation@a25ec58