RFC: consolidate integration test harnesses

[sanity]

Summary

Over the last few months we accidentally accumulated multiple integration / system-test harnesses:

Harness	Location	Language	Primary Scope	Strengths	Current Gaps / Notes
freenet_test macro + TestContext	crates/freenet-macros, crates/core/src/test_utils.rs	Rust (in-tree)	freenet-core integration tests (connectivity.rs, operations.rs, etc.)	Built into core repo, async aware, produces rich diagnostics/events	Lives inside freenet-core so other repos (River, apps) can’t reuse it; still maintains its own node bootstrap pipeline separate from newer crates
freenet-test-network crate	~/code/freenet/freenet-test-network	Rust crate	Generic gateway/peer launcher; used by new diagnose_connectivity.rs, River tests, ping diagnostics	Repo-agnostic, exposes builder API, preserves temp dirs/logs, easy to script	Exists out-of-tree, so freenet-core still has parallel logic (macro) and existing harnesses haven’t migrated; feature coverage not yet on par with TestContext (event aggregation, TestLogger, etc.)
Ubertest	crates/core/tests/ubertest.rs	Rust + external riverctl	End-to-end River scenario on freenet-core	Exercises real app flows, validates River compatibility	Duplicates config/build logic, requires manual deps, hasn’t adopted the crate/macro
Gateway Test Framework	freenet-testing-tools/gateway-testing	Python	Release/pre-release test suite, extended soak runs	Battle-tested, produces markdown/log bundles, supports multi-room/stability modes	Entirely separate stack (Python + scripts), hard to keep in sync with Rust harness improvements
Ping app harness	apps/freenet-ping/app/tests/run_app.rs	Rust	Manual ping scenarios (partial connectivity, diagnostics)	Deep instrumentation of ping flows, integrates with app build	Reimplements node start/diagnostics, only usable from ping repo
River CLI tests	river/cli/tests/message_flow.rs	Rust	River message flow over a freenet-core checkout	Runs against actual River client binaries	Maintains another copy of cluster start/teardown logic

This fragments fixes (like Nacho’s request on PR #2081) and causes new harnesses to pop up whenever the existing ones don’t fit (“the macro lives in freenet-core so River can’t use it”, “gateway-testing is Python”, etc.).

Proposal / Discussion Starters

1. Identify a single reusable foundation – likely freenet-test-network, since it can be imported by any crate. Can we teach the freenet_test macro/TestContext to use it under the hood so core tests migrate without losing diagnostics? Once that happens, we could drop the bespoke bootstrap logic from ubertest, ping, River, etc.
2. Decide what survives:
   • Keep freenet-test-network as the canonical launcher?
   • Migrate the macro/TestContext to wrap it (and keep the nice ergonomics like ctx.node("peer1"))?
   • Treat gateway-testing Python as “release-only” and avoid spinning up new Python stacks?
3. Migration path – e.g. prioritize porting Ubertest, ping manual tests, and River message-flow tests onto the shared crate, then delete their inline launchers.
4. Feature gaps – list anything the shared harness must support (event aggregation, TestLogger, River-specific hooks, artifact dumps) before consolidation.

Comments/thoughts welcome on:

• Which harness should be the canonical one?
• What blockers do River/ping teams have with adopting it?
• Whether gateway-testing stays for release soak once equivalent Rust coverage exists.

Let’s use this issue to track the decision + migration plan so future PRs don’t add yet another bespoke harness.

[github-actions]

🤖 Auto-labeled

Applied labels:

• T-tracking (90% confidence)
• E-hard (85% confidence)
• A-developer-xp (95% confidence)
• S-needs-design (90% confidence)

Reasoning: This is an RFC/meta issue proposing consolidation and a migration plan across multiple repos and test harnesses, so it functions as a tracking/meta-design discussion (T-tracking) and requires architectural design (S-needs-design). Consolidating harnesses across repos is complex and touches developer tooling and CI, so effort is assessed as E-hard and area as A-developer-xp.

Previous labels: none

If these labels are incorrect, please update them. This helps improve the auto-labeling system.

[iduartgomez]

All external applications can import freenet as a dependency (and will be, probably), so the optimal location should be here where we can also use it in-tree IMO.

I am almost sure 90% of the code overlaps so we can probably merge it into https://github.com/freenet/freenet-core/blob/main/crates/core/src/test_utils.rs (the module can be expanded if necessary to be multipage source pages).

I know the answer of LLMs is to always generate more code, since is what they are trained for, but that is not always the right answer. Having a common framework will allow us to improve it over time, easier to maintain and have a common understanding.

I would like to distinguish two things: one thing is having scripts for driving integration or end to end tests etc. which is fine, but having multiple tests harness should be frozen upon without very good reason (even if we need to support multiple use cases, the code can probably live in a single place).

[sanity]

@iduartgomez I agree we need to consolidate—but I'd still prefer an external crate like freenet-test-network because of the decoupling benefits: when I need to test an older worktree or experimental branch, I have to cherry-pick the latest harness changes into that branch if it's in-tree. Having the harness as a separate crate lets me point it at whatever freenet-core binary or branch I'm debugging without touching that branch's source.

All external applications can import freenet as a dependency

River only depends on freenet-stdlib today—it doesn't depend on freenet-core IIRC. So for River tests, an external harness crate makes more sense than pulling in all of freenet-core as a dev-dependency just to access test_utils.

Proposal: We commit to freenet-test-network as the canonical harness, migrate any missing features (like event aggregation) into it, and deprecate the other 5. That gives us consolidation while preserving the decoupling benefits.

[iduartgomez]

You can still have it as a separate crate and in this repo, or do we need repo proliferation? (I already regret having freenet-stdlib in a separate repo , would rather have a monorepo, we had a submodule for that but was removed cause the friction.)

[sanity]

I'm sure you're right that using a different repo for freenet-stdlib was a mistake (which we can correct), but I think there is a good argument for freenet-test-network being its own repo because it decouples the test harness from specific freenet-core commits and branches. Consider scenarios like:

1. Bisecting bugs: Testing 20 commits to find a regression requires the SAME harness across all commits. In-repo means each commit has different test code.

2. Parallel development: I often have 3-4 worktrees on different branches. When I improve the harness (better logging, new assertions), I want those improvements available across all branches immediately—if it's in the main repo then I have to cherry-pick changes and it gets messy fast.

3. Regression testing: "Did commit X introduce this behavior or has it always existed?" needs identical test code at different points in history. The harness should be a stable measurement tool that evolves independently from the code-under-test.

[iduartgomez]

If you take a look at the test utils in crates/core/src/test_utils.rs you will see we cannot decouple from freenet_core completely since it uses facilities provided by core itself to aggregate multiple events from multiple nodes. The only way to do that would be to decouple it and syncronize via schemas and deserializing events etc. There is something built-in cause we have an aggregator which works over websockets so maybe that could be re-used for the external test harness, but is not going to be as convenient to use and we may have to duplicate schema definitions which is insanity and defeats the whole pourpouse of having schemas, or depend on freenet_core anyway.

Let's start porting the macro and the code in test_utils there and see how that pans out. I don't thinkall the external tests harness etc. will work well with higher density networks and it will become unmanageable goign through logs etc. without being able to gather information in a structured way. That's why we had the whole network_monitor (leveraging precisely the event collection etc.).

I am skeptical this extenral harness and scripts will scale to more complex scenarios etc. so maybe we need separate frameworks after all.

[sanity]

@iduartgomez I think there is a good argument for keeping both test_utils.rs + the freenet_test macro given they have access to freenet-core internals, and freenet-test-network outside of freenet-core making it easier to test different commits and branches and pull in other components to test with like River.

How about if we go from 6 harnesses down to 2:

Keep:

1. test_utils.rs + freenet_test macro (in freenet-core) - in-process testing with internal API access
2. freenet-test-network (external) - process-based testing for cross-version work and external apps

Remove:

• Python gateway-testing (clearly obsolete)
• Ping app harness (should use freenet-test-network)

Refactor or Remove:

• Ubertest - This is a specific test suite (not a harness) that validates River works on freenet-core. Both tests are currently ignored. It could either:
  • Be refactored to use the freenet_test macro (eliminate duplicated setup code)
  • Be moved to River's test suite using freenet-test-network
  • Be removed if River's own tests cover this

@netsirius - thoughts on ubertest? Should we migrate it to use the macro, or is this better tested with freenet-test-network like here?

Does that consolidation path make sense?