New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: let idle OS threads exit #14592

Open
brianmario opened this Issue Mar 2, 2016 · 17 comments

Comments

Projects
None yet
10 participants
@brianmario

brianmario commented Mar 2, 2016

  1. What version of Go are you using (go version)?
    1.5.3, 1.6
  2. What operating system and processor architecture are you using (go env)?
    x86_64 - OSX and Linux
  3. What did you do?
    Any golang program will create a new OS thread when it needs to if things are blocked. But these threads aren't ever destroyed. For example, a program using 7 goroutines might have 40+ OS threads hanging around. The numbers will surely get much higher as traffic fluctuates against a golang server process throughout the day.
  4. What did you expect to see?
    Once an OS thread has been idle long enough, I would have expected it to be destroyed. Being recreated if needed. Expanding and contracting like the relationship between the heap and GC.
  5. What did you see instead?
    The many OS threads that were created hang around even with an idle program and very few goroutines.

After doing some reading into other (closed) issues on this repo dating back to 2012 - including the one where SetMaxThreads was introduced - I'm curious, why keep the OS threads around instead of cleaning them up?

@ianlancetaylor ianlancetaylor changed the title from Cleanup OS threads during GC? to runtime: cleanup OS threads during GC? Mar 2, 2016

@ianlancetaylor ianlancetaylor added this to the Unplanned milestone Mar 2, 2016

@diegomontoya

This comment has been minimized.

diegomontoya commented May 9, 2016

Cleaning up the idles would benefit mobile (android) deployments since threads are very precious resource there. Perhaps a settings to toggle cleanup idle threads or not is appropriate for those that would trade absolute performance for reduced resource usage.

@bradfitz

This comment has been minimized.

Member

bradfitz commented May 9, 2016

I don't think this has anything to do with GC. It sounds like a general runtime issue instead. Assigning to @aclements to triage for Go 1.8.

@brianmario

This comment has been minimized.

brianmario commented May 9, 2016

I don't think this has anything to do with GC.

Cleaning up new threads right after they've been used seemed potentially wasteful, so obviously some form of periodic cleanup felt more realistic. GC is a timeframe the runtime is already setting aside for periodic cleanup, so that's what I went with initially. But I agree that it doesn't necessarily need to be tied to the GC for any particular reason.

@quentinmit quentinmit added the NeedsFix label Oct 11, 2016

@rsc rsc modified the milestones: Go1.9, Go1.8Maybe Oct 21, 2016

@rsc rsc changed the title from runtime: cleanup OS threads during GC? to runtime: let idle OS threads exit Oct 21, 2016

@rsc rsc added NeedsDecision and removed NeedsFix labels Oct 21, 2016

@brianmario

This comment has been minimized.

brianmario commented Feb 16, 2017

I know there's plenty of other work on everyone's respective plates but has anyone had a chance to think about this some more?

@ayanamist

This comment has been minimized.

ayanamist commented Feb 17, 2017

As i know, GC will return allocated heap memory to system after 5 minutes idle, can it also exit idle threads?

@aclements

This comment has been minimized.

Member

aclements commented Feb 17, 2017

This is certainly technically possible, but the devil's in the details. For example, we often manipulate pointers to Ms (the structure representing an OS thread) in contexts that don't interact with stop-the-world and can't have write barriers, which means it's difficult to know when we're actually done with an M and can safely recycle it. Even if we don't release the M structure itself and just release the OS thread, we have to know when we can safely reuse the M for another OS thread to avoid races.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 17, 2017

I'm probably missing some subtlety, but it seems to me that in stopm we could change the notesleep to a notetsleep that sleeps for five minutes. If that sleep times out, we discard the thread and go back to a regular notesleep. If we're woken up from that, we spin up a new thread.

I don't know whether this is actually worth doing--does an idle thread really use significant resources?--but I think it seems feasible to do it.

@aclements

This comment has been minimized.

Member

aclements commented Feb 17, 2017

@ianlancetaylor, I'm not sure I understand your suggestion. If we discard the thread, who's calling notesleep?

@ayanamist

This comment has been minimized.

ayanamist commented Feb 17, 2017

@ianlancetaylor Pure go code will not be affected by these kind of idle threads since there won't be many idle threads, however, cgo codes or blocking syscalls (like cgo dns resolve or disk io related operations) will make a lot of threads, which may slow down the machine.

@ianlancetaylor

This comment has been minimized.

Contributor

ianlancetaylor commented Feb 17, 2017

@aclements You're right, of course. That was dumb.

@bradfitz bradfitz modified the milestones: Go1.10, Go1.9 May 24, 2017

@bradfitz

This comment has been minimized.

Member

bradfitz commented May 24, 2017

Punting to Go 1.10.

@gopherbot

This comment has been minimized.

gopherbot commented Jun 16, 2017

CL https://golang.org/cl/46037 mentions this issue.

gopherbot pushed a commit that referenced this issue Oct 11, 2017

runtime: make it possible to exit Go-created threads
Currently, threads created by the runtime exist until the whole
program exits. For #14592 and #20395, we want to be able to exit and
clean up threads created by the runtime. This commit implements that
mechanism.

The main difficulty is how to clean up the g0 stack. In cgo mode and
on Solaris and Windows where the OS manages thread stacks, we simply
arrange to return from mstart and let the system clean up the thread.
If the runtime allocated the g0 stack, then we use a new exitThread
syscall wrapper that arranges to clear a flag in the M once the stack
can safely be reaped and call the thread termination syscall.

exitThread is based on the existing exit1 wrapper, which was always
meant to terminate the calling thread. However, exit1 has never been
used since it was introduced 9 years ago, so it was broken on several
platforms. exitThread also has the additional complication of having
to flag that the stack is unused, which requires some tricks on
platforms that use the stack for syscalls.

This still leaves the problem of how to reap the unused g0 stacks. For
this, we move the M from allm to a new freem list as part of the M
exiting. Later, allocm scans the freem list, finds Ms that are marked
as done with their stack, removes these from the list and frees their
g0 stacks. This also allows these Ms to be garbage collected.

This CL does not yet use any of this functionality. Follow-up CLs
will. Likewise, there are no new tests in this CL because we'll need
follow-up functionality to test it.

Change-Id: Ic851ee74227b6d39c6fc1219fc71b45d3004bc63
Reviewed-on: https://go-review.googlesource.com/46037
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
@aclements

This comment has been minimized.

Member

aclements commented Oct 27, 2017

Issue #22439 has an example of a problem caused by not exiting idle Ms (and freeing their stack memory).

@rsc rsc modified the milestones: Go1.10, Go1.11 Nov 22, 2017

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Nov 28, 2017

I just discovered that exiting threads might affect IO. From WSASendTo documentation https://msdn.microsoft.com/en-us/library/windows/desktop/ms741693(v=vs.85).aspx - "... Note All I/O initiated by a given thread is canceled when that thread exits. For overlapped sockets, pending asynchronous operations can fail if the thread is closed before the operations complete. See ExitThread for more information. ..."

Alex

@aclements

This comment has been minimized.

Member

aclements commented Nov 28, 2017

I just discovered that exiting threads might affect IO.

Oh, geez. Good call.

Could this be a problem even for the current thread exiting we do? Something like:

  1. G1 on M1 starts an overlapped I/O operation.
  2. G1 migrates to M2.
  3. G2 runs on M1, calls runtime.LockOSThread, and then exits, causing M1 to exit.

Is there any way to query whether there are pending I/O operations on a thread? I'm wondering if we could just block a thread from exiting until all of its pending I/Os are done.

@alexbrainman

This comment has been minimized.

Member

alexbrainman commented Nov 29, 2017

Could this be a problem even for the current thread exiting we do? Something like:

G1 on M1 starts an overlapped I/O operation.
G1 migrates to M2.
G2 runs on M1, calls runtime.LockOSThread, and then exits, causing M1 to exit.

I did not know that OS threads are exiting at this moment. Is there a way to verify your scenario above? What would be the program to do that? I can do 1. by reading from a TCP connection, but how would do I force 2. and 3?

Is there any way to query whether there are pending I/O operations on a thread?

I do not know of any. You could call GetCurrentThreadId Windows API to get current thread id, but you, probably, can already work that information out of whatever you have recorded for Gs and Ms. Surely we could mark Ms when IO starts, and remove that mark once IO completes.

Alex

@gopherbot

This comment has been minimized.

gopherbot commented Jan 2, 2018

Change https://golang.org/cl/85662 mentions this issue: runtime: remove special handling of g0 stack

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment