Skip to content

Consolidate integration tests into single bin per crate#662

Merged
jmqd merged 2 commits intomainfrom
jm/consolidate-integration-tests
Apr 10, 2026
Merged

Consolidate integration tests into single bin per crate#662
jmqd merged 2 commits intomainfrom
jm/consolidate-integration-tests

Conversation

@jmqd
Copy link
Copy Markdown
Contributor

@jmqd jmqd commented Apr 9, 2026

Currently, each file in tests/ produces a separate binary and link step. This adds tests/main.rs entry points for each crate and disables auto-discovery so each crate compiles one test binary instead.

Crate Before After
etl-api 11 binaries 1
etl-destinations 5 binaries 1
etl 6 binaries 1
  • Insta snapshot files renamed to match the new binary name (main__ prefix).
  • Tests remain individually addressable, e.g. cargo test -p etl-api --test main -- destinations::.

Measured impact (clean builds, back-to-back, same machine)

Metric Before After Change
Incremental --tests build 39.6s 26.0s -34%
Integration test disk usage 2.9 GB 484 MB -83%
Total executables 35 16 -54%
Target directory 14 GB 11 GB -21%
Full clean build (lib + tests) 3m 35s 3m 10s -12%

@jmqd jmqd force-pushed the jm/consolidate-integration-tests branch from e3708ed to a2382af Compare April 9, 2026 18:24
@coveralls
Copy link
Copy Markdown

coveralls commented Apr 9, 2026

Coverage Status

No base build to compare — jm/consolidate-integration-tests into main

@jmqd jmqd marked this pull request as ready for review April 10, 2026 02:36
@jmqd jmqd requested a review from a team as a code owner April 10, 2026 02:36
@jmqd jmqd changed the title Consolidate integration tests into single binaries per crate Consolidate integration tests into single bin per crate Apr 10, 2026
Copy link
Copy Markdown
Contributor

@farazdagi farazdagi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question: since we are producing only one binary (per sub-project), won't it hinder parallel test execution with nextest?

@jmqd
Copy link
Copy Markdown
Contributor Author

jmqd commented Apr 10, 2026

One question: since we are producing only one binary (per sub-project), won't it hinder parallel test execution with nextest?

AFAIU, nextest does process-per-test parallelism:

nextest might also be overall a bit faster with these changes as well (less binaries to query), although I haven't tested that.

Each file in tests/ produces a separate binary and link step.
Add tests/main.rs entry points and disable auto-discovery so
each crate compiles one test binary instead:

  etl-api:          11 binaries -> 1
  etl-destinations:  5 binaries -> 1
  etl:               6 binaries -> 1

Metric                          Before    After   Change
Incremental --tests build       45.5s     25.7s   -43%
Integration test binaries       22        3       -86%
Test binary disk (integ)        3.1 GB    484 MB  -84%
Clean build                     2m 44s    2m 45s  ~same
@jmqd jmqd force-pushed the jm/consolidate-integration-tests branch from a2382af to 37ae7bc Compare April 10, 2026 06:37
@farazdagi farazdagi self-requested a review April 10, 2026 10:18
@jmqd jmqd merged commit 296e920 into main Apr 10, 2026
13 checks passed
@jmqd jmqd deleted the jm/consolidate-integration-tests branch April 10, 2026 10:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants