Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: Additional Allocator Metrics in runtime.MemStats #11890

matttproud opened this issue Jul 27, 2015 · 9 comments

runtime: Additional Allocator Metrics in runtime.MemStats #11890

matttproud opened this issue Jul 27, 2015 · 9 comments
compiler/runtime Issues related to the Go compiler and/or runtime.


Copy link

matttproud commented Jul 27, 2015

Hi, I am wondering if it would be amenable to include several additional core metrics in runtime.MemStats, namely the following measures:

  1. No. of spans pending being released by size class.

    This helps server authors understand the discrepancy between reported
    heap reservation/allocation versus process RSS.

  2. No. of live spans (with active allocations contained therein) by size class.

    Essential corollary for no. 1.

  3. Measure of sum of span occupancy by size class.

    This helps server and runtime authors understand level of span reuse
    and divine potential problems with heap fragmentation viz-a-viz no. 2.

  4. Sum of span allocations by size class (not of inner object allocations).

    This helps server authors understand the aggregate throughput of
    memory flow in realtime, a measure of efficiency that is useful to
    capture when comparing releases and automated canary tests.

  5. Sum of span releases by size class (not of inner object allocations).

    Useful corollary for no. 4.

  6. Summary statistics about age for spans by size classes: min, median, average, max.

    Throughput in no. 4 and no. 5 is useful, but this takes the level of detail to a
    deeper level.

  7. Cumulative sums of individual allocations made for each span size class.

    Useful throughput measure for individual allocations.

These would be inordinately beneficial in gross measurement of fleet memory allocator performance as well as offer server authors deeper telemetric insights into the lifecycles of their allocations. pprof is great for one-off diagnosis but not real-time analytics of the fleet.

I would be happy coming to compromise on these, especially to enhance the language and requirements as well as to possibly volunteer time in the implementation of these representations should we come to agreement.

@ianlancetaylor ianlancetaylor changed the title Additional Allocator Metrics in runtime.MemStats runtime: Additional Allocator Metrics in runtime.MemStats Jul 27, 2015
@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Jul 27, 2015
Copy link

bradfitz commented Jul 28, 2015

/cc @RLH @aclements

Copy link

RLH commented Jul 28, 2015

Go seem to have exposed the internal implementation concept of a span in memstats but not in any meaningful way such as indicating the size of a span. This is good since it allows future implementations freedom to change the size of a span. Go should preserve this freedom. This proposal seems to imply that one will need to know the size of a span for it to be useful. Is that true?

Copy link
Contributor Author

matttproud commented Jul 28, 2015

Yes, that is true. The core reason why is the size class is a useful hint in indicating the size of the individual objects involved in the measurement. For instance, this is to help differentiate between low volume of large object churn versus high volume of small object churn.

As for hiding the implementation details of the runtime, let me offer a few remarks:

  1. Metrics that abstract/hide what the runtime is really doing under the hood are of specious value,
    because it forces the consumers of said metrics to study the runtime in much greater detail than
    the case of just exposing the raw internals bare. A quick analogy of this from the world of
    Java: MBean and MXBeans (example package) are the
    training wheels of internal runtime metrics; whereas anyone worth his or her salt uses HSPerfData
    example because of the level of actionable detail it provides. At
    the very worst, the Bean metric approach shoehorns implementation-specific details into
    standardized types that either do not make sense or lose needed detail. Most of this pain occurs
    at runtime when one gets a runtime exception or zero values from a bean that exists as a
    placeholder but has no relevance for the runtime in use.

    The question I have is this: what happens the moment there are alternative runtimes available for
    Go? It seems doubtful that each runtime would be able to meaningfully shoehorn its allocator
    metrics into runtime.MemStats unless they all use the same paradigms.

  2. The Go 1.X Stability Grant makes a lot of sense in general, but
    it may not be well to apply it doctrinally to the runtime package since that one is out of all of the
    packages the most likely to undergo churn behind the scenes.

  3. The domain of runtime is ultimately implementation-specific. Treating it as less than that is a
    disservice to the user.

I think there is a middleground; just not sure what it is.

Copy link

aclements commented Jul 28, 2015

Details aside, there's been some discussion of the philosophy of MemStats over on #10323. I agree that we have to expose implementation details in order for it to be truly useful, but we've already made the mistake of exposing such details in MemStats that are no longer really relevant, but are covered under Go 1 compatibility.

Copy link
Contributor Author

matttproud commented Jul 28, 2015

Yep, that bug had been the impetus for filing this one (just get the need
out there on public record).

Is it possible to, for instance, keep the legacy fields in the struct and
mark them as "do not use" in the field comments or let go fix assist in
auto-migrations where it makes sense? Alternatively, could we create a
proposal to weaken the stability grant around select aspects of the
runtime? Each of these seems tractable, admittedly with differing levels
of satisfaction, smell, and disgust.

Copy link

dvyukov commented Dec 3, 2015

Current HeapSys/HeapAlloc/HeapInuse/HeapIdle/HeapReleased allow to answer 1, 2 and 3 questions (minus by-size-class part, which is more of an implementation detail, and when you think about RSS size classes are more-of-less irrelevant).
I don't understand what exactly you want to do with 4, 5, 6 and 7 numbers. Please elaborate.

Copy link

cristaloleg commented Jul 8, 2021

Kindly ping @matttproud 😉

Copy link
Contributor Author

matttproud commented Jul 14, 2021

@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 7, 2022
Copy link

mknyszek commented Sep 29, 2022

A lot of these are now doable with runtime/metrics, since we have a path out for implementation-defined metrics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
compiler/runtime Issues related to the Go compiler and/or runtime.
None yet

No branches or pull requests

9 participants