Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: scheduler: go-routine starvation #21053

Closed
prasannavl opened this issue Jul 17, 2017 · 7 comments

Comments

Projects
None yet
4 participants
@prasannavl
Copy link

commented Jul 17, 2017

What version of Go are you using (go version)?

go version go1.8.3 windows/amd64

What operating system and processor architecture are you using (go env)?

set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=D:\Workspace\Golang
set GORACE=
set GOROOT=D:\Apps\Scoop\apps\go\current
set GOTOOLDIR=D:\Apps\Scoop\apps\go\current\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set PKG_CONFIG=pkg-config
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2

What did you do?

Consider the below program:

package main

import (
	"fmt"
	"sync/atomic"
	"time"
)

func main() {

	var ops uint64 = 0
        // This is the key value. When this is >= to the num 
        // of physical cores, the program will never terminate. 
        // Otherwise it should work as expected.
        iters := 4
	for i := 0; i < iters; i++ {
		go func() {
			for {
				atomic.AddUint64(&ops, 1)
			}
		}()
	}

	// Wait a second to allow some ops to accumulate.
	time.Sleep(time.Second)

	opsFinal := atomic.LoadUint64(&ops)
	fmt.Println("ops:", opsFinal)
}

I was trying to get an estimate of the ops of atomics and mutexes. I wrote the above example. When instead of atomics, mutexes are used, it works as expected. However, when using atomics, when the number of go-routines are greater than the number of the actual physical threads on the system, the goroutine that's waiting on the timer (the main go-routine) is starved - I'm guessing is because of the CAS semantics of atomics that staves the runtime due to SpinWaits.

What did you expect to see?

The program actually completes in a near a second.

What did you see instead?

The program never ends using 100% of the CPU.

This tells me that the scheduler ends up starving the goroutine that's waiting on the timer. Is it possible to improve the scheduler to detect this somehow and not end up starving? This is a unique problem due to the nature of go-routines. Coming from a .NET eco-system, where the Task units are run from a thread pool, which uses physical threads, the OS scheduler handles this smoothly, and this was quite a surprise to me - but makes sense considering the nature of go-routines.

UPDATE

Surprisingly, looks like it doesn't actually have anything to do with atomics. Just turning it into an empty loop, causes the main go-routine to be starved. Quite confused on what to make of this. Is this some kind of a scheduler bug?

@prasannavl prasannavl changed the title Scheduler starvation when using atomics scheduler: go-routine starvation when using atomics Jul 17, 2017

@ALTree

This comment has been minimized.

Copy link
Member

commented Jul 17, 2017

This is #10958.

Note that if you build with GOEXPERIMENT=preemptibleloops, the program terminates. On my 4-cores machine:

go1.8.3:

$ go run prova.go
  [does not terminate]

go1.9beta2, built with preemptibleloops:

$ ~/go/bin/go run prova.go
ops: 30388899

This will be fixed when (if) preemptibleloops-by-default lands.

@prasannavl prasannavl changed the title scheduler: go-routine starvation when using atomics scheduler: go-routine starvation Jul 17, 2017

@prasannavl

This comment has been minimized.

Copy link
Author

commented Jul 17, 2017

@ALTree - Ah. Glad to see it work in progress. Shall I close this as dup?

@bradfitz

This comment has been minimized.

Copy link
Member

commented Jul 17, 2017

Yes, this is a dup. It's targeted to Go 1.10.

@bradfitz bradfitz closed this Jul 17, 2017

@prasannavl

This comment has been minimized.

Copy link
Author

commented Jul 17, 2017

@ALTree, does the GOEXPERIMENT apply to all platforms? I tried it in windows - doesn't seem to work. Still hangs.

PS D:\Workspace\Labs\go-play> go version
go version go1.9beta2 windows/amd64
PS D:\Workspace\Labs\go-play> $env:GOEXPERIMENT
preemptibleloops
PS D:\Workspace\Labs\go-play> go run .\play.go
@bradfitz

This comment has been minimized.

Copy link
Member

commented Jul 17, 2017

You can't just set the environment variable, IIRC. You have to set the variable and then rebuild the Go toolchain.

@ALTree

This comment has been minimized.

Copy link
Member

commented Jul 17, 2017

Yeah you need to rebuild the toolchain from source with the variable enabled, e.g. GOEXPERIMENT=preemptibleloops ./make.bash in go/src

@prasannavl

This comment has been minimized.

Copy link
Author

commented Jul 17, 2017

@bradfitz, ah - thanks. Thought it was a runtime config. Rebuilding the toolchain now.

@mikioh mikioh changed the title scheduler: go-routine starvation runtime: scheduler: go-routine starvation Jul 21, 2017

@golang golang locked and limited conversation to collaborators Jul 21, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.