Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: expose total stack usage in MemStats #7468

Closed
randall77 opened this issue Mar 4, 2014 · 15 comments
Closed

runtime: expose total stack usage in MemStats #7468

randall77 opened this issue Mar 4, 2014 · 15 comments
Milestone

Comments

@randall77
Copy link
Contributor

randall77 commented Mar 4, 2014

Non-standard-sized stack allocations should be accounted under StackSys.  Right now they
are treated as regular malloc'd memory.
@dvyukov
Copy link
Member

dvyukov commented Mar 6, 2014

Comment 1:

> Right now they are treated as regular malloc'd memory.
They are not profiled by mprof.
> Non-standard-sized stack allocations should be accounted under StackSys.
If we just account it in StackSys, then part of stacks (std size) will be accounted only
in StackSys, and some part will be accounted both in StackSys and HeapSys. Since you
don't know what is the proportion between these parts, you can't make sense out of
HeapSys and heap profile -- heap profile shows less than HeapSys, but you don't know
what part of that difference is stacks.
mprof profile and memstats must be consistent and allow to verify sanity (tell whether
mprof lies or not).

Labels changed: added release-go1.3, repo-main.

@randall77
Copy link
Contributor Author

randall77 commented Mar 8, 2014

Comment 2:

Dmitry, did your change https://golang.org/cl/72440043 fix this?

@dvyukov
Copy link
Member

dvyukov commented Mar 11, 2014

Comment 3:

No, it did not. That change only fixes g->stacksize, which is an internal counter used
to detect infinite recursion.

@dvyukov
Copy link
Member

dvyukov commented Mar 11, 2014

Comment 4:

StackInuse is also broken (StackInuse decreases during function execution):
go version devel +3cd546ac3a99 Tue Mar 11 16:36:14 2014 +1100 linux/amd64
$ go test -v -cpu 1,1,1,1 runtime -test.run=TestStack
=== RUN TestStackOverflow
--- PASS: TestStackOverflow (0.68 seconds)
=== RUN TestStackSplit
--- PASS: TestStackSplit (0.01 seconds)
=== RUN TestStackMem
--- PASS: TestStackMem (3.09 seconds)
    stack_test.go:1576: Consumed 2MB for stack mem
    stack_test.go:1582: Inuse 0MB for stack mem
=== RUN TestStackOverflow
--- PASS: TestStackOverflow (0.67 seconds)
=== RUN TestStackSplit
--- PASS: TestStackSplit (0.01 seconds)
=== RUN TestStackMem
--- FAIL: TestStackMem (3.10 seconds)
    stack_test.go:1576: Consumed 0MB for stack mem
    stack_test.go:1582: Inuse 17592186044415MB for stack mem
    stack_test.go:1584: Stack inuse: want 4194304, got 18446744073709547520

@rsc
Copy link
Contributor

rsc commented Mar 11, 2014

Comment 5:

StackSys is listed as one of a collection under "Low-level fixed-size structure
allocator statistics." More generally the Sys numbers are supposed to correspond 1:1
with mmap requests made to the system. I don't think we should redefine what
StackSys/StackInuse mean to be different than the other Sys/Inuse pairs. In particular
the sum of all the XxxSys fields needs to match the Sys field.
Maybe instead we can add a new field that counts total stack usage? Maybe just
StackAlloc.
It's fine for Go 1.3 if this is a noninvasive change.

Labels changed: added release-go1.3maybe, removed release-go1.3.

Status changed to Accepted.

@dvyukov
Copy link
Member

dvyukov commented Mar 11, 2014

Comment 6:

> Maybe instead we can add a new field that counts total stack usage? Maybe just
StackAlloc.
We also must be able to understand what is heap size w/o stacks.
There is already StackInuse which currently counts only fixed-size cache allocations.
Maybe we can make StackInuse represent total stack memory (cache + mallocgc), and then
subtract malloced stacks from HeapAlloc?
But this introduces another problem: HeapAlloc and HeapSys won't match...

@rsc
Copy link
Contributor

rsc commented Mar 11, 2014

Comment 7:

It's too confusing to redefine existing counters. Maybe add StackAlloc (all
stack in use) and StackHeap (all stack in use from heap). Then you can use
HeapAlloc - StackHeap to get "heap size without stacks".

@dvyukov
Copy link
Member

dvyukov commented Mar 11, 2014

Comment 8:

For now #7 sounds like the best option... and we will need to introduce mstats paid
certification for developers :)

@dvyukov
Copy link
Member

dvyukov commented Mar 12, 2014

Comment 9:

This issue was updated by revision 00e6fc1.

LGTM=rsc
R=golang-codereviews, bradfitz
CC=golang-codereviews, khr, rsc
https://golang.org/cl/74010043

@rsc
Copy link
Contributor

rsc commented May 9, 2014

Comment 10:

Labels changed: added release-go1.4, removed release-go1.3maybe.

@gopherbot
Copy link

gopherbot commented Jun 24, 2014

Comment 11:

CL https://golang.org/cl/104200047 mentions this issue.

@randall77
Copy link
Contributor Author

randall77 commented Jul 1, 2014

Comment 12:

This issue was closed by revision 7c13860.

Status changed to Fixed.

@randall77
Copy link
Contributor Author

randall77 commented Jul 1, 2014

Comment 13:

This issue was closed by revision 3cf83c1.

@gopherbot
Copy link

gopherbot commented Jul 17, 2014

Comment 14:

CL https://golang.org/cl/112240044 mentions this issue.

@randall77
Copy link
Contributor Author

randall77 commented Jul 17, 2014

Comment 15:

This issue was closed by revision f378f30.

@rsc rsc added this to the Go1.4 milestone Apr 14, 2015
@rsc rsc removed the release-go1.4 label Apr 14, 2015
@golang golang locked and limited conversation to collaborators Jun 25, 2016
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jun 25, 2018
In order to move malloc to Go, we need to have a
separate stack allocator.  If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.

Stacks are the last remaining FlagNoGC objects in the
GC heap.  Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)

Fixes golang#7468
Fixes golang#7424

LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jun 25, 2018
Breaks windows and race detector.
TBR=rsc

««« original CL description
runtime: stack allocator, separate from mallocgc

In order to move malloc to Go, we need to have a
separate stack allocator.  If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.

Stacks are the last remaining FlagNoGC objects in the
GC heap.  Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)

Fixes golang#7468
Fixes golang#7424

LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»

TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jun 25, 2018
redo stack allocation.  This is mostly the same as
the original CL with a few bug fixes.

1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.

Should fix bugs and memory leaks in original changelist.

««« original CL description
undo CL 104200047 / 318b04f28372

Breaks windows and race detector.
TBR=rsc

««« original CL description
runtime: stack allocator, separate from mallocgc

In order to move malloc to Go, we need to have a
separate stack allocator.  If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.

Stacks are the last remaining FlagNoGC objects in the
GC heap.  Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)

Fixes golang#7468
Fixes golang#7424

LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»

TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»

LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jul 9, 2018
In order to move malloc to Go, we need to have a
separate stack allocator.  If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.

Stacks are the last remaining FlagNoGC objects in the
GC heap.  Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)

Fixes golang#7468
Fixes golang#7424

LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jul 9, 2018
Breaks windows and race detector.
TBR=rsc

««« original CL description
runtime: stack allocator, separate from mallocgc

In order to move malloc to Go, we need to have a
separate stack allocator.  If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.

Stacks are the last remaining FlagNoGC objects in the
GC heap.  Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)

Fixes golang#7468
Fixes golang#7424

LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»

TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
wheatman pushed a commit to wheatman/go-akaros that referenced this issue Jul 9, 2018
redo stack allocation.  This is mostly the same as
the original CL with a few bug fixes.

1. add racemalloc() for stack allocations
2. fix poolalloc/poolfree to terminate free lists correctly.
3. adjust span ref count correctly.
4. don't use cache for sizes >= StackCacheSize.

Should fix bugs and memory leaks in original changelist.

««« original CL description
undo CL 104200047 / 318b04f28372

Breaks windows and race detector.
TBR=rsc

««« original CL description
runtime: stack allocator, separate from mallocgc

In order to move malloc to Go, we need to have a
separate stack allocator.  If we run out of stack
during malloc, malloc will not be available
to allocate a new stack.

Stacks are the last remaining FlagNoGC objects in the
GC heap.  Once they are out, we can get rid of the
distinction between the allocated/blockboundary bits.
(This will be in a separate change.)

Fixes golang#7468
Fixes golang#7424

LGTM=rsc, dvyukov
R=golang-codereviews, dvyukov, khr, dave, rsc
CC=golang-codereviews
https://golang.org/cl/104200047
»»»

TBR=rsc
CC=golang-codereviews
https://golang.org/cl/101570044
»»»

LGTM=dvyukov
R=dvyukov, dave, khr, alex.brainman
CC=golang-codereviews
https://golang.org/cl/112240044
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants