Repository navigation
Simplifying NUnit execution: shared lifecycle, clear ownership and coordinated releases #1960
Replies: 4 comments 2 replies
|
@CharliePoole @manfred-brands @OsirisTerje — a further refinement of the session proposal: could we distinguish the logical responsibility map from the physical processes and communication boundaries? For ordinary remote execution with an isolated worker, one possible physical arrangement is: runner process → remote host process → isolated test process The engine and provider can live inside the runner, the launcher inside the remote host, and the agent and framework inside the test process. Their separate responsibilities do not each require another executable, protocol or serialization stage. That process count is an example, not the simplification goal. The goal is one common lifecycle with the fewest physical boundaries needed by the execution capabilities. Embedded and dynamically deployed describe independent properties: embedded describes the runtime environment, while dynamic deployment describes how the host and test package arrive. A backend can be both. The effective componentization and sequencing remain the same: acquire the target, stage the host/package as needed, start execution, obtain an agent, discover/run, release resources and verify cleanup. A resident host can reuse already-provisioned resources within that same lifecycle. What may differ is the physical placement of the roles in processes, application domains or objects, and the isolation/termination capabilities offered by the runtime. New embedded backends should use the same logical lifecycle without importing unnecessary desktop-style processes or relays. That suggests reviewing the deployment coordinator as part of the integration discussion. Our experimental #1968 has a separate coordinator executable, which is useful for exercising the lifecycle, but that need not determine the final process model. Where practical, its shared lifecycle code could be a library used by an existing process. A bootstrap helper may still be justified by deployment, authentication or lifetime requirements; it should not automatically become a relay for every test event and result. The simplification goals would be:
Could we add a process-and-transport view alongside the responsibility map, identify the purpose of each actual boundary, and use that to decide coordinator placement and the next small integration step? This would build on the existing provider architecture rather than assume a new public interface or an immediate redesign of the current PRs. Prepared by Neil Colvin with assistance from OpenAI Codex. |
|
@nunit/engine-team Although this is an engine team discussion, I'm treating it as a broader one since @oznetmaster refers to it as serving the adapter, Console and TestCentric equally. That's OK, since all the right people are here and there's no other common place for discussion on GitHub. TL;DR The original post is very broad and perhaps too broad for practical, useful discussion. Let's talk about Agents, the Agent server and how both of them affect the adapter. Apologies for my own verbosity. The build and release simplification issue, #1945 is about simplification of structure within the engine and was undertaken because some of the new features being added had already introduced a greater level of complexity. We're still working on that. I believe @oznetmaster wants to have a similar discussion here, but at a higher level, across our various projects. For me, a concrete breakdown of components works well, so I'll start by trying to apply the responsibility map to the components we have... Front end We (nunit organization) have two front ends: the Console Runner and NUnitLite, which is bundled with the framework. TestCentric is another. The responsibilities listed appear to be what we already do. I've always referred to these as "Runners" which is potentially confusing since the engine also contains classes referred to as runners, so "Front end" makes sense in the broader context of this discussion. Microsoft also provides several front ends, but I think they must be discussed separately, since the responsibilities they assume are beyond our control. Engine We have one engine, the NUnit engine. Although it's built together with the console runner it's logically separate and only accessed through interfaces. At one time we had a goal of making the engine a replaceable component but back when Chris Maddock was the engine lead we decided to abandon that and always pair the console runner with a particular engine. TestCentric originally had it's own engine, but my team abandoned that as well. So... one engine is what we expect to have, although some front ends might not actually use it and I don't believe that use of it should be required. Provider/backend Up to now, we have not had any separate components of this type. The server that we're currently working on will be the first one. The engine contains classes called providers and even a remote provider, intended to communicate with the actual remote server. Some of the terminogy could change of course. Personally, I think back end is slightly confusing because it means a lot of things in different contexts. For example, in the GUI project we refer to the NUnit engine as our back end. Agent/frameworkdriver In V3, agents and framework drivers are conceptually part of the engine. We changed that for V4, making agents pluggable extensions. Framework drivers are now a part of the agent. For agents we (engine team) develop, the framework driver is a part of the common So, after laying that out, I have to ask myself "What is there to discuss?" I'll answer in WRT the front ends, I think there's not much to say. I'm open to contradiction, of course, but I think paring down what we need to talk about can make for a simpler conversation and the role of the front ends is well established. The engine internals are pretty clear. The external interfaces are driven by clients. That is, when the console or the adapter need something from the engine it doesn't yet supply, a new interface is provided or an old one modified. Similarly, third-party runner authors make requests and if they are accepted. the changes are made. Downstream, the framework can make changes that require some action on the part of the engine. This can include interface changes, of course, but that has happened only once over the past years to my knowledge. More common is the introduction of new named settings, which have to pass from the runner, through the engine, to the framework, crossing one or two transmission boundaries. These are usually discovered when something doesn't work, leading to a bug report, which eventually makes its way to the engine. I'd be interested in improving this interface. The server that @oznetmaster is creating is one area where there's a lot to talk about. The engine has to interface with it directly so the discussion has been within the engine team up to now. If the adapter expected to use it, then it would be a big part -- I think the major part -- of this discussion. However, up to now, the adapter has not used agents or processes to execute tests, so I'd ask @OsirisTerje to let us know if a broader discussion should be started. The role of agents is also something we have not yet discussed widely. That's another area I do agree could use a wider discussion. The new in-process agent will restore the in-process capability we removed from the console runner in the early betas and will most likely be used by the adapter. Remote processes, of course, will involve interaction with the new Server being developed. I wrote the above as something of a devil's advocate, for which I apologize, but I think that discussions can suffer from premature generalization just as software can. In my view, we should start with agents, the remote server and especially the potential effect of both on the adapter. A further word about the adapter... Adding our times working on the engine together, @OsirisTerje and I have worked with Microsoft on the subject of the adapter over a period of decades. At best, we have at times been able to get clear information from them about how to work within the interfaces that they define. At worst.... well, we have had to reverse engineer or perform experiments to deduce how things work. I an remember only one occasion where an actual change was made as a result of the needs we expressed. I think we both have come to consider Microsoft interfaces and responsibilities as a condition we have to accept rather than something that is up for discussion. One result of this is that the adapter, although a very important part of this discussion, may end up being an exception to any responsibility matrix we try to establish. Work on the adapter is subject to sever constraints. As always, I'm open to discussion on any of this, but the above is my starting point. |
Agent interfaces and gaps for discussion 1960Draft for review · 18 November 2026 @CharliePoole @OsirisTerje Here is a concrete comparison of the existing interfaces and the remaining integration work. The common provider contract already covers all three agent types; this suggests completing and validating those paths before introducing another general interface. Source scope: engine main
The shared surfaces are the provider contract, agent information, agent/runner interfaces, stop operations and exit event data. The concrete remote behavior is in the provider, proxy, server stop dispatch and backend-path selection. I suggest resolving three practical points next:
Main currently pins Engine.Api beta.101.4; the remote branch pins beta.101.2. The branch therefore needs alignment before it is treated as a current-main implementation. Server deployment can remain optional composition around acquisition, allowing session-deployed and independently installed servers within the same provider contract. This table proposes no change to project governance or mandatory process boundaries. Prepared jointly by Neil Colvin and OpenAI Codex. |
|
@oznetmaster My goal in jumping in with a comment, as I told you privately, was to get this discussion moving. You addressed it initially to @manfred-brands, @OsirisTerje and myself last week and there were no responses. I believe that's because the scope is somewhat abstract and definitely very large. This is my motive for wanting to pare it down. I strongly believe that getting an actual discussion in which others participate will eventually lead us to cover everything you want to see covered. The first comment indicates agreement with the suggestion of reducing the scope of the discussion. I had suggested "Agents, the Agent server and how both of them affect the adapter." You also suggest a three-part breakdown: Agent acquisition, Agent ownership and release and Cancellation and Completion. This sounds even narrower but is OK with me. As I said, i think it will take us everywhere we need to go. One note... we have not yet finalized the public engine API. Issue #770, introduced by Chris Maddock, was replaced by #1631. The latter appeared resolved in beta.1, so I closed it. This issue calls for having a separate interface for extension developers so that engine internals are not needlessly exposed by the primary engine API. Some of the changes we have made toward developing beta.6 have undone the earlier work, particularly as regards providers. That's OK for the development phase we are in but will need to be fixed prior to beta.6. I think this discussion could be helpful for that purpose. Your second is a bit unclear to me. Are you intending the table to be a description of what is currently implemented or a proposal of how the discussion should resolve the issues you are raising? My comments below assume they are intended as discussion starters...
@OsirisTerje @manfred-brands The two of you were invited to this discussion by @oznetmaster. I'm asking that you please respond at least minimally to indicate whether you want to have it. I'm going to stay out until I hear from one or both you. Charlie |
Uh oh!
There was an error while loading. Please reload this page.
@CharliePoole, @manfred-brands, @OsirisTerje — our recent engine, remote-execution and adapter work suggests an opportunity to simplify how we develop and maintain capabilities spanning several NUnit components.
Charlie’s build and release simplification epic, #1945, already addresses substantial infrastructure work. This discussion would complement that effort by considering execution contracts and responsibility across the engine, providers, agents and front ends.
The existing architecture has supported our remote implementation well. However, implementing and validating discovery, cancellation, results and cleanup has required considerable discussion across component boundaries. Could we establish a shared execution model that makes those responsibilities easier to understand and maintain?
A possible organizing concept is a test execution session, covering environment acquisition, package preparation, discovery, execution, events, cancellation and cleanup. This is initially a way to describe the lifecycle and its ownership. Whether it requires a new public interface should follow from examining the existing contracts.
A starting responsibility map might be:
Each session would have an identifiable coordinator, while resources would retain explicit owners. Releasing a borrowed agent, terminating an owned worker and restoring a remote environment are different operations; the contracts should make those differences clear.
Cancellation is a useful example. Requesting cooperative cancellation and terminating an execution environment are distinct capabilities. We could define their semantics once, including what callers can expect when either capability is unavailable. Front ends would request the appropriate operation; the engine and backend would carry it out within the constraints of their host. In particular, an adapter cannot assume it owns the test-platform process.
Completion reporting also needs a shared contract. Test-result completion and confirmed execution quiescence are different facts. A synthesized terminal event must not, by itself, authorize removal of a package that may still be executing.
For remote execution, I would include package staging and cleanup within this lifecycle. A session-deployed server and an independently installed server could both supply the same execution capability. Users should be able to initiate testing through one workflow without separately maintaining a server unless their environment requires that arrangement. Our implementation experience is recorded in #1909.
This should serve the adapter, Console and TestCentric equally. Shared acceptance scenarios for discovery, identity, filtering, cancellation, connection loss and cleanup could establish the common behavior, with front-end integration tests covering the differences.
There is a related packaging question: which components genuinely need independent versioning and releases? Coordinated development and release tooling could reduce maintenance for closely coupled components while preserving useful public extension boundaries. We should evaluate that alongside #1945 before considering repository consolidation.
Would a short, shared execution-lifecycle document and responsibility map be useful? We could use it to assess existing interfaces and guide focused changes, building on the provider work already underway. I would particularly welcome adapter and TestCentric input before assuming the same contract meets every front end’s needs.
Prepared by Neil Colvin with assistance from OpenAI Codex.
All reactions