GetAllocatedBytesForCurrentThread unpredictably returns inaccurate values

[jorive]

I have been working on adding new metrics to Xunit-Performance-API. One of those metrics is the "Allocated bytes in current thread".

While writing tests to exercise the code I noticed I was not getting the expected values. Below I am adding two cases:

var method = typeof(GC)
    .GetTypeInfo()
    .GetMethod("GetAllocatedBytesForCurrentThread", BindingFlags.Static | BindingFlags.Public | BindingFlags.NonPublic);

var nBytesBefore = (long)method.Invoke(null, null);
var nBytesAfter = (long)method.Invoke(null, null);
var nBytesExtra = nBytesAfter - nBytesBefore;

for (int i = 0; i < 1000; ++i)
{
    GC.Collect();
    nBytesBefore = (long)method.Invoke(null, null);
    GC.Collect();
    nBytesAfter = (long)method.Invoke(null, null);

    var totalAllocatedBytes = nBytesAfter - (nBytesBefore + nBytesExtra);
    if (totalAllocatedBytes != 0)
        throw new Exception($"Unexpected number of bytes. [{i}]: {totalAllocatedBytes} bytes");
}

var method = typeof(GC)
    .GetTypeInfo()
    .GetMethod("GetAllocatedBytesForCurrentThread", BindingFlags.Static | BindingFlags.Public | BindingFlags.NonPublic);

var nBytesBefore = (long)method.Invoke(null, null);
var nBytesAfter = (long)method.Invoke(null, null);
var nBytesExtra = nBytesAfter - nBytesBefore;

for (int i = 0; i < 1000; ++i)
{
    nBytesBefore = (long)method.Invoke(null, null);
    nBytesAfter = (long)method.Invoke(null, null);

    var totalAllocatedBytes = nBytesAfter - (nBytesBefore + nBytesExtra);
    if (totalAllocatedBytes != 0)
        throw new Exception($"Unexpected number of bytes. [{i}]: {totalAllocatedBytes} bytes");
}

Impact:

1. The GetAllocatedBytesForCurrentThread API is already in netcoreapp1.1, but it is producing incorrect values
2. The Xunit-Performance-API is used by the performance tests of the coreclr and corefx repositories, and it targets netstandard1.5

Having this bug fixed on 1.1 would be essential for correctness of the collected metrics. Bytes allocated is a valuable metric to collect as it can help correlate GC behavior with changes in other metrics when measuring the performance impact of repository changes.

[Maoni0]

dotnet/coreclr#10162

[Maoni0]

FYI - this was checked into 1.1.0: dotnet/coreclr#10277

[Maoni0]

@jorive I am closing this issue. please feel free to reactivate if you are seeing problems.

[OrbintSoft]

I dunno if related, but when I use GetAllocatedBytesForCurrentThread in my xunit project I get non realistic extremely high values, like 26111439440 bytes

[jorive]

@OrbintSoft I will suggest opening a new issue with detailed information about how the API is being used, runtime version, repro case, etc.
Cheers,
-José

[Maoni0]

@OrbintSoft I presume you meant that's a non realistic extremely high values because you don't think you've allocated that much on that thread? that's about 25GB of allocations...pretty easily reached if you allocate a lot (eg, if you are running a test that mostly just allocates). if you haven't allocated 25GB yet it tells you 25GB, clearly there's a problem we need to look at; if that's the case please open a new issue as @jorive suggested.

[OrbintSoft]

@Maoni0 yes there is some bad code I was optimize, but 25 Gb I feel too much, that is more than the available RAM.

With

Process currentProcess = Process.GetCurrentProcess();
long memory = currentProcess.WorkingSet64;

I get no more than 400 MiB.
I was investigating to get the allocated virtual memory by the thread before opening a new issue, but VirtualMemorySize64 on .net core is not usable due to Control Flow Guard

Let me do some more investigation before opening a new issue.

[Maoni0]

allocated bytes can certainly be way larger than the available RAM and working set if you allocate something very temporary. GC ETW/lttng events will tell you how much you allocated (total).