proposal: net/http/httptest: synctest support

[rogpeppe]

Proposal Details

I use the httptest package a lot, and in particular NewServer rather than ResponseRecorder, because
it picks up subtle real-world issues that don't appear when not using the entire connection machinery,
and can be used for long-lived connections such as websockets too.

It would be really nice to be able to use httptest.NewServer in a manner compatible with testing/synctest;
that is, with a fake network connection rather than using the OS's network stack.

It would also be nice to have some fake network connection code in the standard library
in addition to net.Pipe, but that requires more API surface area and raises tricky
questions about global variables.

For the record, here's how I'm doing it currently; it took me a little time to work it out.

https://go.dev/play/p/AVXzqqwiJPn

Concrete proposal:

// NewSynctestServer returns and starts a new [Server] suitable
// for using with [testing/synctest]. It does not use the underlying
// system's network stack, and thus it's not possible to use
// [net.Dial] etc to access the server. Use methods on the [net/http.Client]
// value returned by [Server.Client]
// instead of invoking top-level HTTP functions such as [net/http.Get].
func NewSynctestServer(handler http.Handler) *Server

[seankhliao]

I suppose this counts as a concrete proposal for #14200 ?
There doesn't appear to be anything synctest specific about it though.

[gabyhelp]

Related Issues

• proposal: net/http/httptest: ability to use net.Pipe() for testing/synctest #75294 (closed)
• proposal: x/net/nettest: add MakeLocalPipe to construct local listener and connections #30984
• net/http/httptest: optional faster test server #14200
• proposal: net/http/httptest: listen on IPv6/IPv4-agnostic loopback address #56998 (closed)
• proposal: net/http/httptest: add support for 1XX responses #56346
• proposal: net/http/httpest: add ResponseRecorder.EnableFullDuplex #65120
• proposal: testing: less syntactic overhead to synctest tests #76607
• testing/synctest: new package for testing concurrent code #67434 (closed)
• proposal: net/http/httptest: add support for http.ResponseController to ResponseRecorder #60229 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[rogpeppe]

There doesn't appear to be anything synctest specific about it though.

Nice find! Yeah, that is indeed the case. I thought of other names, like "NewMemServer" but didn't think of anything nice.
I guess the main thing that distinguishes this from #14200 is the goal: the aim here is not to make things fast but to make them testable, and that might have different design implications.

[neild]

Thanks for this proposal. We definitely need an easier way to start a synctest-friendly httptest server.

This is indeed basically the same as #14200; not sure if we should continue to use that issue or use this one as an actual concrete proposal.

A server can use real or fake networking (NewServer vs. NewSynctestServer), and it can use unencrypted TCP or TLS (NewServer vs. NewTLSServer). Would we need a NewTLSSynctestServer? If we add another option will we need four more New...Server methods?

Alternatively, this could be a field on httptest.Server:

package httptest

type Server struct {
  // FakeNet controls whether the server uses localhost TCP sockets or an in-memory fake network.
  FakeNet bool

  // ...
}

srv := &httptest.Server{
  FakeNet: true,
  Config: &http.Server{
    Handler: handler,
  }
}
srv.Start()

...but that's certainly more verbose than:

srv := httptest.NewSynctestServer(handler)

[seankhliao]

I was thinking

srv := httptest.NewUnstartedServer(handler)
srv.FakeNet()

[neild]

Related: #77362, a proposal to add synctest-friendly networking primitives to std.

[neild]

srv := httptest.NewUnstartedServer(handler)
srv.FakeNet()

httptest.NewUnstartedServer creates a loopback listener with net.Listen. It'd be nice if the simple path skipped creating a TCP listener that will only be immediately closed.