Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/net/ipv6: TestPacketConnReadWriteMulticast{UDP,ICMP} failing with "i/o timeout" on OpenBSD 6.8 and 7.0 #42064

Closed
bcmills opened this issue Oct 19, 2020 · 29 comments

Comments

@tklauser
Copy link
Member

@tklauser tklauser commented Dec 2, 2020

This test consistently fails on the openbsd-arm64-jsing and the 386/amd64 builders running OpenBSD 6.8 as well:

openbsd-arm64-jsing: https://build.golang.org/log/be54c3c8bfe2313b55b63b76bc5af92d6c39a524
openbsd-386-68: https://build.golang.org/log/eea81ada598c39d224d3a9a40b4739bbbe739b09
openbsd-amd64-68: https://build.golang.org/log/9039640721536372ca5170299ee6e923aa3775dd

The test also failed on https://golang.org/cl/274772 on the openbsd-mips64-jsing builder.

It doesn't seem to fail on builder running OpenBSD 6.2 or 6.4.

Loading

@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Dec 14, 2020

It doesn't seem to fail on builder running OpenBSD 6.2 or 6.4.

It'd be helpful to find out if the test runs or gets skipped on OpenBSD 6.4. (It's possible it's passing in 6.4 because IPv6 isn't supported there and it gets skipped.)

Loading

@bcmills bcmills changed the title x/net/ipv6: TestPacketConnReadWriteMulticastUDP failing with "i/o timeout" on openbsd-arm-jsing builders x/net/ipv6: TestPacketConnReadWriteMulticastUDP failing with "i/o timeout" on OpenBSD builders Dec 21, 2020
@bcmills bcmills removed the arch-arm label Dec 21, 2020
@bcmills
Copy link
Member Author

@bcmills bcmills commented Dec 21, 2020

Marking as release-blocker via #11811, especially since this doesn't seem builder- or arch-specific.

But note that OpenBSD is not a first-class port, so it may be viable to unblock the release by skipping the test on the affected builders — especially given that this appears to be either a regression in the kernel or an incompatible ABI change.

Loading

@bcmills bcmills removed this from the Unreleased milestone Dec 21, 2020
@bcmills bcmills added this to the Go1.16 milestone Dec 21, 2020
@gopherbot
Copy link

@gopherbot gopherbot commented Dec 21, 2020

Change https://golang.org/cl/279512 mentions this issue: dashboard: promote OpenBSD 6.8 builders to be default

Loading

gopherbot pushed a commit to golang/build that referenced this issue Dec 23, 2020
The known issue with OpenBSD 6.8 builders appears to be resolved
via CL 278732 and CL 279134. Promote them to the primary OpenBSD
post-submit builders and TryBots.

Having test coverage from OpenBSD 6.8 and 6.4 builders gives us
us more confidence that Go works on supported OpenBSD versions
(which are 6.8 and 6.7 at this time, per past policy decision
in https://golang.org/issue/15227#issuecomment-293319876).

Reduce numTryTestHelpers from 5 to 4 based on some data from
golang.org/issue/37439 showing that going from 3 to 5 helpers
doesn't make a significant difference. We can adjust further
if we learn that OpenBSD TryBots become the bottleneck.

Fixes golang/go#35712.
For golang/go#42542.
For golang/go#42064.
Updates golang/go#42426.

Change-Id: Id2fa4be7c3161f89752c1428146846fe06ca63db
Reviewed-on: https://go-review.googlesource.com/c/build/+/279512
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

@gopherbot gopherbot commented Dec 23, 2020

Change https://golang.org/cl/280052 mentions this issue: ipv6: skip known-failing tests on OpenBSD 6.8 builders

Loading

gopherbot pushed a commit to golang/net that referenced this issue Dec 24, 2020
These tests are known to be failing on OpenBSD 6.8 and need
investigation. The OpenBSD 6.8 builders are default TryBots
now, so skip them to prevent people working on other packages
from constantly getting a failure.

Issue 42064 remains open to do the work of determining if the
ipv6 package needs to be fixed, or if OpenBSD 6.8 has a problem
and the test can be permanently skipped as unsupported.

For golang/go#11811.
Updates golang/go#42064.
Updates golang/go#35712.

Change-Id: I59883384e4dbce679cb1a4e8e2c3f0fa7983010b
Reviewed-on: https://go-review.googlesource.com/c/net/+/280052
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
@dmitshur dmitshur changed the title x/net/ipv6: TestPacketConnReadWriteMulticastUDP failing with "i/o timeout" on OpenBSD builders x/net/ipv6: TestPacketConnReadWriteMulticast{UDP,ICMP} failing with "i/o timeout" on OpenBSD 6.8 Dec 24, 2020
@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Dec 24, 2020

@4a6f656c Do openbsd-{arm,arm64,mips64}-jsing builders all run OpenBSD 6.8? If not, the issue title may need to be adjusted further.

Loading

@toothrot
Copy link
Contributor

@toothrot toothrot commented Jan 7, 2021

@4a6f656c Just a friendly ping on this release-blocker in the new year. Do you have any updates?

Loading

@cagedmantis
Copy link
Contributor

@cagedmantis cagedmantis commented Jan 13, 2021

@4a6f656c Have there been any updates with this issue. We are approaching the release and this is a release-blocker.

Loading

@tklauser
Copy link
Member

@tklauser tklauser commented Jan 14, 2021

I quickly checked the arm64 and mips64 builders using gomote (the arm builder seems to be offline). They seem to be running OpenBSD 6.8 (uname -a output):

openbsd-arm64-jsing:
OpenBSD gopher-arm64.sing.id.au 6.8 GENERIC.MP#2 arm64

openbsd-mips64-jsing:
OS: OpenBSD er4.sing.id.au 6.8 GENERIC.MP#0 octeon

Assuming that the offline openbsd-arm-jsing builder also runs at least 6.7 (see #25498 (comment)) and that OpenBSD version might be affected the same, I've sent https://golang.org/cl/283752 to include openbsd-{arm,arm64,mips64}-jsing in the list of builders to skip the test on.

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Jan 14, 2021

Change https://golang.org/cl/283752 mentions this issue: ipv6: skip known-failing tests on openbsd-{arm,arm64,mips64}-jsing builders

Loading

gopherbot pushed a commit to golang/net that referenced this issue Jan 19, 2021
…ilders

The arm64 and mips64 builder run OpenBSD 6.8:

openbsd-arm64-jsing:
OpenBSD gopher-arm64.sing.id.au 6.8 GENERIC.MP#2 arm64

openbsd-mips64-jsing:
OS: OpenBSD er4.sing.id.au 6.8 GENERIC.MP#0 octeon

The openbsd-arm-jsing builder was offline but according to
golang/go#25498 (comment)
it's running at least OpenBSD 6.7. Assuming that this version is
affected as well (or the builder was upgraded to 6.8 in the meantime),
skip TestPacketConnReadWriteMulticastUDP on this builder as well.

For golang/go#11811.
Updates golang/go#42064.

Change-Id: Ia9919057bfc2a69b8884228508ae919ce9d624e9
Reviewed-on: https://go-review.googlesource.com/c/net/+/283752
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Joel Sing <joel@sing.id.au>
@dmitshur
Copy link
Contributor

@dmitshur dmitshur commented Jan 21, 2021

This is still a valid issue: the golang.org/x/net/ipv6 package has a problem on OpenBSD 6.8 and possibly 6.7, which may be a bug in Go code or a limitation/bug of OpenBSD itself. Work needs to be done to determine which, and the next steps.

The known-failing test is now skipped (for #11811), and this doesn't need to block the Go 1.16 release.

Loading

@dmitshur dmitshur removed this from the Go1.16 milestone Jan 21, 2021
@dmitshur dmitshur added this to the Unreleased milestone Jan 21, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Apr 28, 2021

Change https://golang.org/cl/314598 mentions this issue: [internal-branch.go1.16-vendor] ipv6: skip known-failing tests on openbsd-{arm,arm64,mips64}-jsing builders

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 28, 2021

Change https://golang.org/cl/314599 mentions this issue: [release-branch.go1.15] ipv6: skip known-failing tests on openbsd-{arm,arm64,mips64}-jsing builders

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Apr 28, 2021

Change https://golang.org/cl/314749 mentions this issue: [internal-branch.go1.16-vendor] ipv6: skip known-failing tests on OpenBSD 6.8 builders

Loading

gopherbot pushed a commit to golang/net that referenced this issue Apr 28, 2021
…nBSD 6.8 builders

These tests are known to be failing on OpenBSD 6.8 and need
investigation. The OpenBSD 6.8 builders are default TryBots
now, so skip them to prevent people working on other packages
from constantly getting a failure.

Issue 42064 remains open to do the work of determining if the
ipv6 package needs to be fixed, or if OpenBSD 6.8 has a problem
and the test can be permanently skipped as unsupported.

For golang/go#11811.
Updates golang/go#42064.
Updates golang/go#35712.

Change-Id: I59883384e4dbce679cb1a4e8e2c3f0fa7983010b
Reviewed-on: https://go-review.googlesource.com/c/net/+/280052
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
(cherry picked from commit 6772e93)
Reviewed-on: https://go-review.googlesource.com/c/net/+/314749
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
gopherbot pushed a commit to golang/net that referenced this issue Apr 28, 2021
…BSD 6.8 builders

These tests are known to be failing on OpenBSD 6.8 and need
investigation. The OpenBSD 6.8 builders are default TryBots
now, so skip them to prevent people working on other packages
from constantly getting a failure.

Issue 42064 remains open to do the work of determining if the
ipv6 package needs to be fixed, or if OpenBSD 6.8 has a problem
and the test can be permanently skipped as unsupported.

For golang/go#11811.
Updates golang/go#42064.
Updates golang/go#35712.

Change-Id: I59883384e4dbce679cb1a4e8e2c3f0fa7983010b
Reviewed-on: https://go-review.googlesource.com/c/net/+/280052
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
(cherry picked from commit 6772e93)
Reviewed-on: https://go-review.googlesource.com/c/net/+/314751
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
gopherbot pushed a commit to golang/net that referenced this issue Apr 28, 2021
… builders

These tests are known to be failing on OpenBSD 6.8 and need
investigation. The OpenBSD 6.8 builders are default TryBots
now, so skip them to prevent people working on other packages
from constantly getting a failure.

Issue 42064 remains open to do the work of determining if the
ipv6 package needs to be fixed, or if OpenBSD 6.8 has a problem
and the test can be permanently skipped as unsupported.

For golang/go#11811.
Updates golang/go#42064.
Updates golang/go#35712.

Change-Id: I59883384e4dbce679cb1a4e8e2c3f0fa7983010b
Reviewed-on: https://go-review.googlesource.com/c/net/+/280052
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Alexander Rakoczy <alex@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
(cherry picked from commit 6772e93)
Reviewed-on: https://go-review.googlesource.com/c/net/+/314750
Trust: Katie Hockman <katie@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
@bcmills
Copy link
Member Author

@bcmills bcmills commented Nov 17, 2021

Looks like this also affects the new openbsd-.*-70 builders. Either the skip needs to be updated to include them, or the underlying failure needs to be fixed.

greplogs --dashboard -md -l -e '(?ms)\Aopenbsd-(?:386|amd64)-70.*FAIL: TestPacketConnReadWrite'

2021-11-17T14:10:29-47ca1ff-4083a6f/openbsd-386-70
2021-11-17T14:10:29-47ca1ff-4083a6f/openbsd-amd64-70
2021-11-17T04:55:12-47ca1ff-1d004fa/openbsd-386-70
2021-11-17T04:55:12-47ca1ff-1d004fa/openbsd-amd64-70
2021-11-17T04:32:11-47ca1ff-03dd049/openbsd-386-70
2021-11-17T04:32:11-47ca1ff-03dd049/openbsd-amd64-70
2021-11-17T01:56:19-47ca1ff-3d7cb23/openbsd-386-70
2021-11-17T01:56:19-47ca1ff-3d7cb23/openbsd-amd64-70
2021-11-17T00:02:01-47ca1ff-a2b8231/openbsd-386-70
2021-11-17T00:02:01-47ca1ff-a2b8231/openbsd-amd64-70
2021-11-16T23:52:13-47ca1ff-01b6cf0/openbsd-386-70
2021-11-16T23:52:13-47ca1ff-01b6cf0/openbsd-amd64-70
2021-11-16T23:13:41-47ca1ff-5e59d6e/openbsd-386-70
2021-11-16T23:13:41-47ca1ff-5e59d6e/openbsd-amd64-70
2021-11-16T23:12:05-47ca1ff-f659183/openbsd-386-70
2021-11-16T23:12:05-47ca1ff-f659183/openbsd-amd64-70
2021-11-16T23:12:05-47ca1ff-b954f58/openbsd-386-70
2021-11-16T23:12:05-47ca1ff-b954f58/openbsd-amd64-70
2021-11-16T23:12:05-47ca1ff-3729a67/openbsd-amd64-70
2021-11-16T21:27:03-e8b54de-f659183/openbsd-386-70
2021-11-16T21:27:03-e8b54de-f659183/openbsd-amd64-70
2021-11-16T21:27:03-e8b54de-b954f58/openbsd-386-70
2021-11-16T21:27:03-e8b54de-b954f58/openbsd-amd64-70
2021-11-16T21:27:03-e8b54de-3729a67/openbsd-amd64-70

Loading

@bcmills bcmills removed this from the Unreleased milestone Nov 17, 2021
@bcmills bcmills added this to the Go1.18 milestone Nov 17, 2021
@bcmills bcmills changed the title x/net/ipv6: TestPacketConnReadWriteMulticast{UDP,ICMP} failing with "i/o timeout" on OpenBSD 6.8 x/net/ipv6: TestPacketConnReadWriteMulticast{UDP,ICMP} failing with "i/o timeout" on OpenBSD 6.8 and 7.0 Nov 17, 2021
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 22, 2021

Change https://golang.org/cl/366181 mentions this issue: ipv6: remove arbitrary deadlines from tests

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 22, 2021

Some notes from looking at this in a gomote.

When I run TestPacketConnReadWriteMulticastUDP by itself it always fails.

> go test -test.v -test.run=TestPacketConnReadWriteMulticastUDP
=== RUN   TestPacketConnReadWriteMulticastUDP
    multicast_test.go:120: read udp [::]:20756: raw-read udp6 [::]:20756: i/o timeout
--- FAIL: TestPacketConnReadWriteMulticastUDP (0.21s)
FAIL

When I simply run go test it pretty much always passes. The essential difference is that when I run plain go test, the test is skipped.

> go test -v
...
=== RUN   TestPacketConnReadWriteMulticastUDP
    multicast_test.go:48: not available on openbsd
...

The test is apparently being skipped because nettest.RoutedInterface is unable to find an interface that supports multicast loopback. But I don't yet understand why that changes based on whether I run other tests or not.

By the way, the test loops over addresses; the failure is always on the first address, [ff02::].

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 23, 2021

For #36435 we've converted the standard library syscall package to use OpenBSD library functions. However, that conversion has not been in golang.org/x/net/route. When I make that change, the test consistently fails when run with plain go test.

Loading

@ianlancetaylor
Copy link
Contributor

@ianlancetaylor ianlancetaylor commented Nov 23, 2021

As far as I can tell these tests just don't work on OpenBSD. I haven't looked deeper into it but I'm guessing that they fail for the same reason that they fail on FreeBSD. I think the fact that they consistently fail on OpenBSD has been hidden by the fact that they are normally skipped on OpenBSD; when I correct the code that was incorrectly skipping the test, they consistently fail.

I will send a CL.

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 23, 2021

Change https://golang.org/cl/366354 mentions this issue: route, internal/socket: always go through syscall package

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 23, 2021

Change https://golang.org/cl/366355 mentions this issue: ipv6: OpenBSD does not appear to support multicast loopback

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Nov 23, 2021

Change https://golang.org/cl/366195 mentions this issue: internal/socket: always go through syscall package

Loading

gopherbot pushed a commit to golang/net that referenced this issue Nov 23, 2021
Instead of relying on syscall.Syscall, always reach into the syscall
package to call the appropriate functions on Unix systems. We were
already doing this on Darwin. We also have to do this on
OpenBSD. Rather than sometimes reach into syscall and sometimes not,
just always reach in.

For golang/go#42064

Change-Id: Ie292a56766080d0c5ae6b6723d42b5475128290c
Reviewed-on: https://go-review.googlesource.com/c/net/+/366354
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot gopherbot closed this in fe8b969 Nov 23, 2021
gopherbot pushed a commit to golang/net that referenced this issue Nov 23, 2021
Instead of relying on syscall.Syscall, always reach into the syscall
package to call the appropriate functions on Unix systems. We were
already doing this on Darwin and AIX. We also have to do this on
OpenBSD, and it's simpler to do it on Linux 386 and s390x. Rather than
sometimes reach into syscall and sometimes not, just always reach in.

For golang/go#42064

Change-Id: I0adb1c7cc623f2c1247465b3852fefd8d09975d2
Reviewed-on: https://go-review.googlesource.com/c/net/+/366195
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

@gopherbot gopherbot commented Nov 23, 2021

Change https://golang.org/cl/366756 mentions this issue: vendor: update golang.org/x/net to tip

Loading

gopherbot pushed a commit that referenced this issue Nov 29, 2021
This brings in a fix for OpenBSD that lets it correctly gather network
interface information.

For #42064

Change-Id: Ib88fd2f494bb2ee86fd2725d8375b2df1404c4ca
Reviewed-on: https://go-review.googlesource.com/c/go/+/366756
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
@neild
Copy link
Contributor

@neild neild commented Dec 1, 2021

@gopherbot Please open backport issues for 1.16 and 1.17. This is causing trybot failures.

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 1, 2021

Backport issue(s) opened: #49908 (for 1.16), #49909 (for 1.17).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 1, 2021

Change https://golang.org/cl/368534 mentions this issue: [internal-branch.go1.17-vendor] ipv6: OpenBSD does not appear to support multicast loopback

Loading

@gopherbot
Copy link

@gopherbot gopherbot commented Dec 1, 2021

Change https://golang.org/cl/368514 mentions this issue: [internal-branch.go1.16-vendor] ipv6: OpenBSD does not appear to support multicast loopback

Loading

gopherbot pushed a commit to golang/net that referenced this issue Dec 1, 2021
…ort multicast loopback

Skip the tests for it, as we already skip them on FreeBSD.

For golang/go#42064
For golang/go#49908

Change-Id: I058f25309020367e686625033b4f25cb614c7f99
Reviewed-on: https://go-review.googlesource.com/c/net/+/366355
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-on: https://go-review.googlesource.com/c/net/+/368514
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
gopherbot pushed a commit to golang/net that referenced this issue Dec 1, 2021
…ort multicast loopback

Skip the tests for it, as we already skip them on FreeBSD.

For golang/go#42064
For golang/go#49909

Change-Id: I058f25309020367e686625033b4f25cb614c7f99
Reviewed-on: https://go-review.googlesource.com/c/net/+/366355
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-on: https://go-review.googlesource.com/c/net/+/368534
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants