Feature request: sidecar / server wrapper for client/server tests

[jtwaleson]

Motivation

While running a test suite, I typically want to run a server process and a lot of integration tests against it. I want the tests to be super fast and parallel and also stress test the server.

Proposal

Have a "server-wrapper", it's bit like the existing wrapper feature. It just wraps multiple tests. When the first test that needs this "server wrapper" is executed, we spawn the server process and capture its stderr/stdout. Each integration test captures a copy of the server stderr/stdout, which is interleaved with other concurrent test, but at least the output is there.

I've implemented an MVP for my company last weekend (using AI coding) and we're running with this model now in CI. It speeds up our test suite a LOT and we benefit from most of the benefits of nextest (isolation model, better logs, timing, retries, more control of of test execution).

We can now run our full 150+ integration test suite in 40 seconds instead of 15 minutes if we'd do each test with its own server.

See the docs here:
https://github.com/comper-io/nextest/blob/main/site/src/docs/configuration/server-wrappers.md

Alternatives

It's nice for isolation if each integration test spawns up its own server, but that's slow and doesn't guard us against any concurrency bugs. In a way it's great if we can "stress test" the server like this.

Additional context

First of all, thanks a lot for nextest. It's super clean.

Our fork and implementation is here: https://github.com/comper-io/nextest

I'm perfectly fine to maintain our own fork of this, as it might not fit the project's goals. However, if it's interesting, we'd be happy to try and contribute a properly engineered version of this.

[sunshowers]

Thanks. This is really interesting stuff. At Oxide we serialize seeds to disk instead and spin up services from the seed in every test.

I would definitely be interested in taking some version of this, if you'd like to contribute it.

Some notes:

• What is the contract here? Is the server process part of the system under test, or does it act as an oracle that's assumed to be correct? My worry is that with a single shared server process across all tests, you'll end up in situations where one test corrupts in-process state in the server in a way that shows up in other tests. (Having an on-disk seed that each test spins up a process from makes this less painful overall, though there's certainly the startup cost.)
• I'm a bit concerned about the probe. I see what it's trying to do (effectively acting as a wait condition) but I feel like a better approach would be that the server writes to a pipe that nextest waits on. (That would be more general than HTTP, for example.)
• How does process lifecycle management work? What if the server process dies in the middle of test execution? Should we no longer spawn any tests? Should we try and restart the server?
• Is there prior art in other test runners or languages? Would love to know if you did a survey/wrote some notes about approaches and alternatives.

Thanks!

[jtwaleson]

Thanks @sunshowers !

Adding my thoughts on your points below. However, I have some reservations with adding this to nextest apart from the issues you raised.

1. It seems like it doesn't completely match the "pure" way that nextest was designed for. Mostly the strict process isolation.
2. I don't know if there's a broader need for this? No one else commented or upvoted so far. I don't know how to test the waters with other users. If only we (@comper) are using it, it'll just be feature, config & documentation creep for the project.

This is just based on my gut feeling, so as the author & maintainer, tell me what you think. Now on to your points:

What is the contract here? Is the server process part of the system under test, or does it act as an oracle that's assumed to be correct? My worry is that with a single shared server process across all tests, you'll end up in situations where one test corrupts in-process state in the server in a way that shows up in other tests. (Having an on-disk seed that each test spins up a process from makes this less painful overall, though there's certainly the startup cost.)

It'll require deliberate reviews for sure, but I feel that is the case with most optimized test suites. For example, we have some flaky tests which sometimes will crash or corrupt the server. It's a trade-off between CI speed and test isolation.

I'm a bit concerned about the probe. I see what it's trying to do (effectively acting as a wait condition) but I feel like a better approach would be that the server writes to a pipe that nextest waits on. (That would be more general than HTTP, for example.)

Sure, that makes sense! It'll be a bit more complicated as we currently run exec at the end of our server setup script, so the server itself would have to be modified to write to a pipe, or we use a more complicated setup script.

How does process lifecycle management work? What if the server process dies in the middle of test execution? Should we no longer spawn any tests? Should we try and restart the server?

I would treat that as a hard error, so it would cancel all further tests (so always fail fast).

Is there prior art in other test runners or languages? Would love to know if you did a survey/wrote some notes about approaches and alternatives.

In pytest (which I have some experience with) there are fixtures per dir / class / test, which are similar. Other frameworks (JUnit, Go testing, Jest, Vitest) typically have global setup / teardown methods. I like my current implementation better, because each test can declare what it depends on, so running a couple of unit tests will not trigger the whole server setup/teardown.

[sunshowers]

Re your last point, what I'm curious is prior art for test-run-scoped processes specifically. I'm particularly curious about how the process lifecycle gets handled.

I guess one option is, if you have shared-memory tests, to do all initialization in-memory.

[jtwaleson]

https://github.com/pytest-dev/pytest-xprocess this does something similar, but does not handle cleanup out of the box. It just ensures the server is up. Other than that, I did not really find anything. There's https://testcontainers.com/ that was suggested to me because they handle the lifecycle neatly, but they spin up their services per test-suite execution, not per individual test.

We don't really have shared-memory tests, but we need to test our app in combination with postgres. We have client <--ws--> app <-- pg listen/notify --> postgres, and want to test this end to end.

Hope this helps!

[sunshowers]

Oh, I might have misunderstood. Do you run a separate process for every test or share one across the test suite?

[jtwaleson]

No worries. Each test can declare which server it needs. At the first test that needs it, the specified server is launched. Others re-use the same process. When the last test that needs the server has finished, the server is terminated.

[jlizen]

I see use cases inside Amazon that call for this as well. I'll see if any engineers can comment with more detail on their specific usage, but to summarize:

We frequently have integration tests that rely on secondary servers running out-of-process that are somewhat heavyweight (slow to start up, might be java, might even be a remote server that is spun up dynamically). One case I know of is a full ~90 seconds to start up. The servers might be full-on mocks, or they might be somewhat stateful, but teams are generally willing to accept cross-test resource sharing rather than having a process-per-test.

Currently teams use the setup scripts to spin these up. But, it is unfortunate because it makes all unit tests wait on the setup script even if they don't use the server. And then, waiting 90 seconds only to have a unit test fail that doesn't even use the server, feels bad :)

I haven't looked through @jtwaleson 's fork in depth, but the model of running in a sidecar and lazily spinning it up on first invocation, seems like it would fit our usage well. I'll look at it more deeply, as well as our internal usage, to get a better idea of how well that maps over.

Something else to consider (as later scope) might be some amount of cross-invocation lifecycle management that allows both spinning up and subsequently attaching to an existing process. I know that's a whole can of design worms and not strictly required for nextest to handle, but might be nice ^^

[sunshowers]

Thanks @jlizen — looking forward to more feedback on usage and direction here. Having two or more real users to design against makes this a lot more compelling to maintain in nextest overall.

[teor2345]

At subspace, they start up a WASM runtime for some tests, or several nodes to simulate various blockchain

Making server startup lazy would help other tests run faster, and also maybe help with memory issues or timeouts in constrained environments (or on busy CI). There's a bunch of bugs about this, for example:

• Fix XDM tests which occasionally timeout in CI autonomys/subspace#3562
• domain-client-operator receipt tests fail with missing or unexpected receipts autonomys/subspace#3385
• test_domain_node_starting_check sometimes fails in CI with: "The system cannot find the path specified." autonomys/subspace#3532

Some of them could be due to race conditions, too, which might be easier to debug by delaying some server startups.

Also, lazy logging startup could help make their tests run under cargo test and cargo nextest:
https://github.com/autonomys/subspace/blob/7818df416740a7f75cc822047114914b1bbe3f62/shared/subspace-process/src/lib.rs#L52