test: add integration test with minio

[abyssnlp]

Description

Adds integration tests with s3-compatible MinIO.

• docker/docker-compose.yaml spins up the minio container
• docker/copy_tables.sh creates the bucket; unzips, copies and creates the tables in the minio store
• The environment variables for the integration test are supplied via integration_test/env.toml
• Adds running integration tests on pull requests via ci.yml

Notes:

• We can extract the logic to run the integration tests in the Makefile or as a separate script
• The integration test takes around ~3 minutes to complete
• The integration test is only for the hudi-rs rust support via datafusion

This closes #81 .

How are the changes test-covered

• N/A
• Automated tests (unit and/or integration tests)
• Manual tests
  • Details are described below

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 8 lines in your changes missing coverage. Please review.

Project coverage is 86.82%. Comparing base (3359e10) to head (5a747fd).
Report is 51 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/tests/src/lib.rs	0.00%	4 Missing ⚠️
integration_test/tests/hudi_test.rs	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #112      +/-   ##
==========================================
- Coverage   87.42%   86.82%   -0.61%     
==========================================
  Files          14       15       +1     
  Lines         700      736      +36     
==========================================
+ Hits          612      639      +27     
- Misses         88       97       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[xushiyan]

@abyssnlp don't think the ci job is triggered. probably format issue.

[abyssnlp]

@xushiyan The CI is failing due to this error:
dtolnay/rust-toolchain@1.75.0 is not allowed to be used in apache/hudi-rs. I used this action to setup the rust toolchain. Is there a way to whitelist this action or should I use an alternative?

[abyssnlp]

I've removed the external action since GHA runner has rustup already.

[abyssnlp]

@xushiyan All tests (including integration) are green. The labeler is failing due to permissions. I've made the small fix but let me know if this should be done in a different PR.

[abyssnlp]

@xushiyan Addressed all comments in a28debc.

I've reused the TestTable enum but I had to add an extra method to get the s3 path as we unzip and load the tables into minio for the integration tests so we don't need to extract them.

[xushiyan]

thanks for putting this up! Some high-level feedback

• How we plan for docker/: we can position it as a sandbox where people can easily spin up some e2e setup and run something, we may want to add more query engines later, so integration test can be part of it, serving as a validation to make sure all components are working.
• Right now the docker setup is just providing minio, and the integ test is running natively and connecting to minio container. To align with the plan above, we should move the test into a container itself, which installs crates from the code repo.
• The integ test should also cover a python test case by pip install hudi from the repo

[xushiyan]

can this script be incorporated into Dockerfile? it's just a few commands after all. we want to keep things more consolidated

[xushiyan]

why create a lib ? it's just to run an application so bin works better? we don't expect another crate to depend on integ test

[xushiyan]

is this still needed given there is s3.env ? can those 2 be consolidated?

[xushiyan]

@abyssnlp are you planning to resume this work?

[abyssnlp]

Hi @xushiyan,

I've been busier than usual so couldn't get around to this earlier. I've addressed your comments in 5a747fd.

• The integration tests now run in a container
• I'll add a test for python as well later

Let me know what you think.

[xushiyan]

Ok, due to time constraint, I'll push this to the next release.

[xushiyan]

re-worked in #226