Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: fatal error polling some file descriptors on macOS Mojave #37726

Closed
jbardin opened this issue Mar 7, 2020 · 13 comments
Closed

runtime: fatal error polling some file descriptors on macOS Mojave #37726

jbardin opened this issue Mar 7, 2020 · 13 comments

Comments

@jbardin
Copy link
Contributor

@jbardin jbardin commented Mar 7, 2020

What version of Go are you using (go version)?

go1.14
master

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

darwin_amd64
macOS Mojave

What did you do?

The following test file crashes on macOS Mojave

package main

import (
	"os"
	"testing"
	"time"
)

func TestFD(t *testing.T) {
	os.NewFile(4, "")
	time.Sleep(time.Second)
}

With the output:

$ go test fd_test.go
runtime: netpoll: break fd ready for -2
fatal error: runtime: netpoll: break fd ready for something unexpected

runtime stack:
runtime.throw(0x29c482a, 0x39)
/usr/local/Cellar/go/1.14/libexec/src/runtime/panic.go:1112 +0x72
runtime.netpoll(0x176414, 0xc00004c001)
/usr/local/Cellar/go/1.14/libexec/src/runtime/netpoll_kqueue.go:140 +0x30a
runtime.findrunnable(0xc00004e800, 0x0)
/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2323 +0x72b
runtime.schedule()
/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2520 +0x2fc
runtime.park_m(0xc000182600)
/usr/local/Cellar/go/1.14/libexec/src/runtime/proc.go:2690 +0x9d
runtime.mcall(0x80000)
/usr/local/Cellar/go/1.14/libexec/src/runtime/asm_amd64.s:318 +0x5b
...

The same error was seen in #35566, but that fix was only for the test in question.
This isn't directly related to testing however, as we've seen this with normal production binaries.

@chenweiyj

This comment has been minimized.

Copy link

@chenweiyj chenweiyj commented Mar 9, 2020

I have found that if the _test file is compiled first with -c flag as running by go test -c, and run the executable object, the runtime error won't occur.

However, when run the go source file directly by go test, the runtime error occurs.

@chenweiyj

This comment has been minimized.

Copy link

@chenweiyj chenweiyj commented Mar 9, 2020

Also, in macos when calling os.NewFile(4, ""), it will crash if and only if the fd equals 4.

@toothrot toothrot added this to the Backlog milestone Mar 9, 2020
@toothrot

This comment has been minimized.

Copy link
Contributor

@toothrot toothrot commented Mar 9, 2020

/cc @ianlancetaylor for maybe related issue #35566

@maurorappa

This comment has been minimized.

Copy link

@maurorappa maurorappa commented Mar 31, 2020

nothing yet ? :(

@toothrot

This comment has been minimized.

Copy link
Contributor

@toothrot toothrot commented Mar 31, 2020

@maurorappa While every issue is important to us, please try to be patient as many of us are also juggling personal health and responses to the pandemic.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Mar 31, 2020

What is the real code doing that causes this problem? While clearly using os.NewFile(4, "") ought to work, why is this an important case?

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Mar 31, 2020

If the problem only happens with file descriptor 4, then it's because file descriptor 4 is being used by the runtime package. The runtime package treats the descriptor specially, and only expects it to be polled for read. Using os.NewFile(4, ''") is expected to fail.

@jbardin

This comment has been minimized.

Copy link
Contributor Author

@jbardin jbardin commented Apr 1, 2020

A number of our tools run primarily in a subprocess, and the main process passes stdin, stdout and stderr in via Cmd.ExtraFiles in order to provide access to things like the raw terminal. This of course puts the original stdout in FD 4, which we need to convert into a File to use.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 1, 2020

@jbardin Can you show a complete example that demonstrates the problem, and show what happens when you run it? The example above is different in relevant ways from what you are describing. Thanks.

@jbardin

This comment has been minimized.

Copy link
Contributor Author

@jbardin jbardin commented Apr 1, 2020

Thanks @ianlancetaylor, I'll try to put together a standalone example, (need to find a Mojave machine to debug on). The test example above has the exact same fatal error output as what we're experiencing, so thought it might be close enough.

I did however confirm that simply passing stdin, stdout, stderr on 3, 4, 5 works, so there must be something else involved here, and it also seems strange it's related to a specific OS version as well.

@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 2, 2020

I'm fairly certain that the error break fd ready for something unexpected can only happen if a program erroneously uses the file descriptor number that happens to have been created to use to wake up a sleeping kevent.

@jbardin

This comment has been minimized.

Copy link
Contributor Author

@jbardin jbardin commented Apr 4, 2020

Thanks again Ian, I see what's going on now. I had time to track this down and find where the file descriptors were being used in the incorrect process.

@jbardin jbardin closed this Apr 4, 2020
@ianlancetaylor

This comment has been minimized.

Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Apr 4, 2020

Thanks for following up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants
You can’t perform that action at this time.