Auto start testcontainers for `datafusion-cli` #16644

blaginin · 2025-07-01T18:55:18Z

Which issue does this PR close?

Closes Auto run docker containers needed for tests #15092

Rationale for this change

Right now, running integration tests requires manual setup. Once tests are complete, you also need to remember to clean them up...

What changes are included in this PR?

Automatically start and configure containers on start and delete them on completion

Are these changes tested?

Are there any user-facing changes?

No

blaginin · 2025-07-01T23:30:44Z

fyi @Omega359 -- as you brought test containers initially 👀

Omega359 · 2025-07-02T14:04:08Z

Cargo.toml

@@ -173,6 +173,8 @@ rstest = "0.25.0"
 serde_json = "1"
 sqlparser = { version = "0.55.0", default-features = false, features = ["std", "visitor"] }
 tempfile = "3"
+testcontainers = { version = "0.24", features = ["default"] }
+testcontainers-modules = { version = "0.12" }


fyi, I am not sure why but locally I am unable to properly run minio with version 0.12, I have this in my Cargo.toml:

# version 0.12.0 seems to cause minio to hang testcontainers-modules = { version = "=0.11.6", features = ["postgres", "minio", "blocking"] }

unable to properly run minio

can you please post the error?

It just hangs locally (NOT this code, but my own tests). I think it's because of something related to waiting for minio to come up - I'm not doing the mc commands like you are.

Omega359 · 2025-07-02T14:13:39Z

Do we need teardown code to shutdown the container after the run in a dtor annotated method ?

blaginin · 2025-07-02T14:14:48Z

I don't think so because:

Containers implement Drop. As soon as they go out of scope, the underlying docker container is removed.

Omega359 · 2025-07-02T14:21:49Z

Ah. I'm used to scala version mostly where if a test fails the container sticks around. I have to clean them up every so often when I'm doing the test/fix iteration cycle :(

Omega359 · 2025-07-02T14:22:35Z

I haven't run this myself but looking over the changes I see nothing problematic .. I think this is a good improvement!

# Conflicts: # Cargo.toml

blaginin · 2025-07-05T17:09:19Z

@findepi hey 👋 FYI as you suggested the change - in case you want to approve / merge 👀

alamb · 2025-07-12T10:40:07Z

This looks great -- I am testing it out now

alamb

Thank you @blaginin -- I tried this locally on my mac and on a hosted ubuntu cloud VM and it seems to work great

The only thing I think would be good to figure out is if we can get the error when docker is not running to be more visible

@Omega359 did you ever figure out the problem on your machine? Given that this won't affect anyone unless they are running with INTEGRATION_TESTS I think we can merge it and perhaps debug on main

alamb · 2025-07-12T10:51:49Z

datafusion-cli/CONTRIBUTING.md

-export AWS_SECRET_ACCESS_KEY=TEST-DataFusionPassword
-export AWS_ENDPOINT=http://127.0.0.1:9000
-export AWS_ALLOW_HTTP=true
+TEST_STORAGE_INTEGRATION=1 cargo test


When I first ran this command without docker running, several commands failed

Once I started docker it worked great

TEST_STORAGE_INTEGRATION=1 cargo test ---- test_aws_options stdout ---- thread 'test_aws_options' panicked at datafusion-cli/tests/cli_integration.rs:63:10: Failed to start MinIO container: Client(CreateContainer(HyperLegacyError { err: hyper_util::client::legacy::Error(Connect, Os { code: 61, kind: ConnectionRefused, message: "Connection refused" }) })) note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace ---- test_cli stdout ---- thread 'test_cli' panicked at datafusion-cli/tests/cli_integration.rs:63:10: Failed to start MinIO container: Client(CreateContainer(HyperLegacyError { err: hyper_util::client::legacy::Error(Connect, Os { code: 61, kind: ConnectionRefused, message: "Connection refused" }) }))

Note it didn't show any errors about starting the container in the logs / output

if ! docker info > /dev/null 2>&1; then echo "This script requires docker to be running. Please start docker and try again." exit 1 fi

???

Thanks, updated in 4deb0eb

alamb · 2025-07-12T10:52:16Z

datafusion-cli/tests/cli_integration.rs

@@ -35,6 +40,67 @@ fn make_settings() -> Settings {
    settings
 }

+async fn setup_minio_container() -> ContainerAsync<minio::MinIO> {
+    const MINIO_ROOT_USER: &str = "TEST-DataFusionLogin";


I think it is very nice that these environment variables get setup via the test harness now rather than having to be setup outside

datafusion-cli/CONTRIBUTING.md

* Add rust CI cache * comment save if for now * Add cache to depscache * Unify the containers * linux-build-lib to use its own cache * rollback datafusion-cli for now (blocked by #16644) * Fix CLI comment * Load cache for `linux-datafusion-common-features` * Setup more caches * Cache for clippy * comment clippy save * Remove caches that aren't working * Save if * Save if

Omega359 · 2025-07-12T19:54:35Z

@Omega359 did you ever figure out the problem on your machine? Given that this won't affect anyone unless they are running with INTEGRATION_TESTS I think we can merge it and perhaps debug on main

The problem with minio? No, unfortunately not. How I'm starting and waiting for minio to come up in my test code is different than how it's handled in this PR and I don't think this PR will have the same issue that I'm seeing.

blaginin · 2025-07-13T13:32:02Z

There's a minio alternative which actually uses datafusion internally 😱 https://github.com/rustfs/rustfs

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

…ontainers-minio

blaginin · 2025-07-14T19:45:16Z

There's a minio alternative which actually uses datafusion internally 😱 https://github.com/rustfs/rustfs

It's in active development but will switch after the first stable release 📝

blaginin added 3 commits June 27, 2025 19:55

WIP: Testcontainers minio

5021bc5

Merge branch 'main' into testcontainers-minio

6e189f0

Fix wait conditions & update docs

9f987fd

blaginin self-assigned this Jul 1, 2025

github-actions bot added the sqllogictest SQL Logic Tests (.slt) label Jul 1, 2025

blaginin added 2 commits July 1, 2025 19:56

Prettier

ed8c85e

Remove CI step + fmt

6dba785

github-actions bot added the development-process Related to development process of DataFusion label Jul 1, 2025

blaginin added 2 commits July 1, 2025 23:06

Unused imports

3b41443

Remove minio output

42ca948

blaginin marked this pull request as ready for review July 1, 2025 23:29

Omega359 reviewed Jul 2, 2025

View reviewed changes

Merge branch 'main' into testcontainers-minio

4cf2dcc

# Conflicts: # Cargo.toml

blaginin added a commit to blaginin/datafusion that referenced this pull request Jul 7, 2025

rollback datafusion-cli for now (blocked by apache#16644)

46484e4

blaginin added 3 commits July 7, 2025 21:42

Merge branch 'main' into testcontainers-minio

f454866

Clippy

fa9d170

Merge branch 'main' into testcontainers-minio

ca16255

alamb approved these changes Jul 12, 2025

View reviewed changes

blaginin and others added 2 commits July 14, 2025 20:32

Merge branch 'main' into testcontainers-minio

2198c46

Update datafusion-cli/CONTRIBUTING.md

bcddd99

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

blaginin and others added 4 commits July 14, 2025 20:33

Update datafusion-cli/CONTRIBUTING.md

d941ae6

Co-authored-by: Andrew Lamb <andrew@nerdnetworks.org>

Merge remote-tracking branch 'origin/testcontainers-minio' into testc…

aebb86c

…ontainers-minio

Some typos

bf7f857

Better error when docker isn't running

4deb0eb

Auto start testcontainers for datafusion-cli #16644

Are you sure you want to change the base?

Auto start testcontainers for datafusion-cli #16644

Conversation

blaginin commented Jul 1, 2025

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

blaginin commented Jul 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Omega359 commented Jul 2, 2025

Uh oh!

blaginin commented Jul 2, 2025

Uh oh!

Omega359 commented Jul 2, 2025

Uh oh!

Omega359 commented Jul 2, 2025

Uh oh!

blaginin commented Jul 5, 2025

Uh oh!

alamb commented Jul 12, 2025

Uh oh!

alamb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Omega359 commented Jul 12, 2025

Uh oh!

blaginin commented Jul 13, 2025

Uh oh!

blaginin commented Jul 14, 2025

Uh oh!

Uh oh!

Auto start testcontainers for `datafusion-cli` #16644

Auto start testcontainers for `datafusion-cli` #16644

alamb left a comment •

edited

Loading