Skip to content

Use nextest for CI test jobs#22544

Draft
Dandandan wants to merge 1 commit into
apache:mainfrom
Dandandan:codex/ci-nextest
Draft

Use nextest for CI test jobs#22544
Dandandan wants to merge 1 commit into
apache:mainfrom
Dandandan:codex/ci-nextest

Conversation

@Dandandan
Copy link
Copy Markdown
Contributor

@Dandandan Dandandan commented May 26, 2026

Which issue does this PR close?

N/A.

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Yes:

Are there any user-facing changes?

No. This only changes CI test execution.

@github-actions github-actions Bot added the development-process Related to development process of DataFusion label May 26, 2026
@Omega359
Copy link
Copy Markdown
Contributor

Omega359 commented May 26, 2026

I think I tried this in the past ... do the ci machines have enough cores and memory to actually do this effectively?

@2010YOUY01
Copy link
Copy Markdown
Contributor

Thank you for working on this!

I’m wondering if you’ve tried connecting the sqllogictest harness to nextest. This might need some work, but it could make tests easier to run.

@Omega359
Copy link
Copy Markdown
Contributor

Thank you for working on this!

I’m wondering if you’ve tried connecting the sqllogictest harness to nextest. This might need some work, but it could make tests easier to run.

I think @alamb had tried originally to get it hooked up in #13045 but was blocked by a nextest behaviour.

@alamb
Copy link
Copy Markdown
Contributor

alamb commented May 27, 2026

I think @alamb had tried originally to get it hooked up in #13045 but was blocked by a nextest behaviour.

As I recall, what is needed is to mimic the output of cargo test 's default runner to list the tests that are run. This might be straightforward with claude /codex now

- name: Install nextest
uses: taiki-e/install-action@213ccc1a076163c093f914550b94feb90fab916d # v2.79.2
with:
tool: nextest
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a nice way to avoid having to rebuild nextest each run

@Dandandan
Copy link
Copy Markdown
Contributor Author

I think @alamb had tried originally to get it hooked up in #13045 but was blocked by a nextest behaviour.

As I recall, what is needed is to mimic the output of cargo test 's default runner to list the tests that are run. This might be straightforward with claude /codex now

We can do this in a separate pass I think.

@Dandandan Dandandan force-pushed the codex/ci-nextest branch 2 times, most recently from 0bb8416 to 2e64940 Compare May 28, 2026 06:41
@Dandandan
Copy link
Copy Markdown
Contributor Author

Thank you for working on this!

I’m wondering if you’ve tried connecting the sqllogictest harness to nextest. This might need some work, but it could make tests easier to run.

How would it make tests easier to run? Or do you mean that it could use nextest for all?

@Dandandan
Copy link
Copy Markdown
Contributor Author

Dandandan commented May 28, 2026

I think I tried this in the past ... do the ci machines have enough cores and memory to actually do this effectively?

They have 4 cores it seems, so it should help.

AFAIK the biggest speedup with nextest comes from running tests in parallel across binaries, instead of waiting for each binary to finish sequentially.

@Dandandan Dandandan marked this pull request as ready for review May 28, 2026 07:20
@Dandandan Dandandan marked this pull request as draft May 28, 2026 07:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

development-process Related to development process of DataFusion

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants