x/crypto/ssh: test consistently runs out of memory on js-wasm builder

[bcmills]

The golang.org/x/crypto/ssh test consistently fails on the js-wasm builder with the error message runtime: out of memory: cannot allocate 8192-byte block.

Can the test be made to fit within the runtime on that architecture? If not, it should be skipped, instead of continuing to fail and potentially masking more serious problems.

CC @hanwen @neelance @FiloSottile

[agnivade]

The code is allocating memory in a tight loop. Somehow the wasm runtime is unable to schedule a GC run since it runs on a single thread. Although GOMAXPROCS=1 with amd64 passes.

I believe this is a valid bug with wasm runtime (I have seen this happen once to someone in gophers slack), but for now, adding this patch fixes it -

diff --git a/ssh/handshake_test.go b/ssh/handshake_test.go
index 91d4935..742678a 100644
--- a/ssh/handshake_test.go
+++ b/ssh/handshake_test.go
@@ -455,6 +455,9 @@ func testHandshakeErrorHandlingN(t *testing.T, readLimit, writeLimit int, couple
                                        if err != nil {
                                                break
                                        }
+                                       if runtime.GOOS == "js" && i%10 == 0 {
+                                               runtime.GC()
+                                       }
                                }
                                wg.Done()
                                c.Close()

Perhaps we can file an issue and add this patch for now, pointing to the issue.

EDIT: or maybe even track this issue for it.

[hanwen]

the explicit GC call is kludge. As kludges go, I prefer to simply t.Skip() on wasm because it is more explicit.

[agnivade]

SGTM.

[gopherbot]

Change https://golang.org/cl/184397 mentions this issue: ssh: skip testHandshakeErrorHandlingN on js/wasm

[bcmills]

I think this is a bug in the test itself, not a bug in the wasm/js runtime.

As far as I can tell, the calls to c.writePacket end up going to an underlying memTransport, and its writePacket method enqueues a copy of the data without any sort of backpressure or flow-control. That implies that the cases that pass -1 for writeLimit can buffer an arbitrarily large amount of (reachable!) data without ever needing to yield execution.

I suspect that the explicit call to runtime.GC() appears to fix the test because it causes the goroutine to yield execution, not because it collects actual garbage. (If that hypothesis is correct, then substituting a call to runtime.Gosched() will likely have a similar effect.)

If this diagnosis is correct, then an appropriate fix may be to bound (*memTransport).writePacket to a fixed number of packets.

[agnivade]

I had thought about that. If that is so, shouldn't it fail in amd64 with GOMAXPROCS=1 ? What is different in the amd64 runtime that makes it to pass ?

Yes, it is a scheduling issue. runtime.Gosched() is a better substitute.

[bcmills]

What is different in the amd64 runtime that makes it to pass?

I don't actually know, but in general the set of preemption points is different across different go versions. (And they're presumably going to get even more different when #24543 lands!)

My best guess (and, to be clear, it's only a guess) is that on amd64 the runtime checks for goroutine preemption any time it makes a stack check, whereas on js in only preempts when the program blocks. (Perhaps that's due to the relative costs on js, or perhaps there is no mechanism to asynchronously signal a running goroutine to yield?)

[agnivade]

Right, and when I said it's a bug in wasm runtime, I meant for something in needsInvestigation stage, not needsFix. i.e. - is it at all possible to cause a goroutine to prempt in these conditions, or this is something by design ? If it's the previous, then it is a bug worth investigating. If it's the latter, then we can just document it and move on with improving the test.

cc @neelance for some thoughts.

[hanwen]

I guess the problem is the unlimited write in https://go.googlesource.com/crypto/+/22d7a77e9e5f409e934ed268692e56707cd169e5/ssh/handshake_test.go#452 ?

you could try to substitute -1 for writeLimit with a large number (100 would probably do the trick) and see if that works better.

[neelance]

My best guess (and, to be clear, it's only a guess) is that on amd64 the runtime checks for goroutine preemption any time it makes a stack check, whereas on js in only preempts when the program blocks. (Perhaps that's due to the relative costs on js, or perhaps there is no mechanism to asynchronously signal a running goroutine to yield?)

This is correct. Preemption of goroutines requires that the sysmon function runs in a separate OS thread. Since wasm has no threads yet, we can not run sysmon.

[tklauser]

The same seems to occur on wasip1/wasm: https://build.golang.org/log/406c7cf09ff59bad9d5dbf1598bdb99ccaa39798

I've sent https://go.dev/cl/485695 to skip the affected test on that platform too.

[gopherbot]

Change https://go.dev/cl/485695 mentions this issue: ssh: skip unsupported tests on wasip