Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(core): add Alluxio e2e test #3573

Merged
merged 1 commit into from
Nov 13, 2023
Merged

Conversation

hoslo
Copy link
Contributor

@hoslo hoslo commented Nov 13, 2023

base on PR: #3548 , we need to add E2E test for Alluxio.

.env.example Outdated Show resolved Hide resolved
@hoslo hoslo force-pushed the alluxio-e2e-test branch 3 times, most recently from 6d22521 to c6d305d Compare November 13, 2023 04:10
Copy link
Member

@Zheaoli Zheaoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MOST LGTM
BTW, we need add health check in docker compose

@hoslo
Copy link
Contributor Author

hoslo commented Nov 13, 2023

MOST LGTM BTW, we need add health check in docker compose

Excuse me? How can I fix this CI error

@Xuanwo
Copy link
Member

Xuanwo commented Nov 13, 2023

Let's address the core's failure first.

thread 'main' panicked at core/src/raw/tests/utils.rs:68:45:
must succeed: Unsupported (permanent) at  => scheme is not enabled or supported

Context:
   scheme: alluxio

It's looks alluxio is not added in via_map at https://github.com/apache/incubator-opendal/blob/main/core/src/types/operator/builder.rs#L154?

@suyanhanx

This comment was marked as resolved.

@suyanhanx

This comment was marked as resolved.

@suyanhanx
Copy link
Member

@Xuanwo
Copy link
Member

Xuanwo commented Nov 13, 2023

The following tests are failure:

failures:
    behavior::test_blocking_write_with_special_chars
    behavior::test_blocking_create_dir_existing
    behavior::test_blocking_stat_not_exist
    behavior::test_blocking_stat_with_special_chars
    behavior::test_blocking_read_not_exist
    behavior::test_check
    behavior::test_blocking_delete_file
    behavior::test_list_dir
    behavior::test_list_dir_with_metakey
    behavior::test_blocking_remove_one_file
    behavior::test_list_non_exist_dir
    behavior::test_list_sub_dir
    behavior::test_list_dir_with_metakey_complete
    behavior::test_scan_root
    behavior::test_list_nested_dir
    behavior::test_scan
    behavior::test_rename_non_existing_source
    behavior::test_rename_file
    behavior::test_remove_all
    behavior::test_rename_nested
    behavior::test_create_dir_existing
    behavior::test_write_with_special_chars
    behavior::test_rename_overwrite
    behavior::test_stat_not_exist
    behavior::test_stat_with_special_chars
    behavior::test_read_not_exist
    behavior::test_read_with_special_chars
    behavior::test_delete_file
    behavior::test_delete_with_special_chars
    behavior::test_delete_not_existing
    behavior::test_writer_abort
    behavior::test_remove_one_file
    behavior::test_list_rich_dir
    behavior::test_delete_stream

You can reproduce those tests locally by OPENDAL_TEST=alluxio cargo test behaivor --features services-alluxio

@hoslo
Copy link
Contributor Author

hoslo commented Nov 13, 2023

It seems that these error contexts don't have the correct scheme set and can be updated together.

https://github.com/apache/incubator-opendal/blob/bec1e9db5effb99fe1d369941f6e04036fb2ec99/core/src/services/alluxio/backend.rs#L151-L166

fixed

The following tests are failure:

failures:
    behavior::test_blocking_write_with_special_chars
    behavior::test_blocking_create_dir_existing
    behavior::test_blocking_stat_not_exist
    behavior::test_blocking_stat_with_special_chars
    behavior::test_blocking_read_not_exist
    behavior::test_check
    behavior::test_blocking_delete_file
    behavior::test_list_dir
    behavior::test_list_dir_with_metakey
    behavior::test_blocking_remove_one_file
    behavior::test_list_non_exist_dir
    behavior::test_list_sub_dir
    behavior::test_list_dir_with_metakey_complete
    behavior::test_scan_root
    behavior::test_list_nested_dir
    behavior::test_scan
    behavior::test_rename_non_existing_source
    behavior::test_rename_file
    behavior::test_remove_all
    behavior::test_rename_nested
    behavior::test_create_dir_existing
    behavior::test_write_with_special_chars
    behavior::test_rename_overwrite
    behavior::test_stat_not_exist
    behavior::test_stat_with_special_chars
    behavior::test_read_not_exist
    behavior::test_read_with_special_chars
    behavior::test_delete_file
    behavior::test_delete_with_special_chars
    behavior::test_delete_not_existing
    behavior::test_writer_abort
    behavior::test_remove_one_file
    behavior::test_list_rich_dir
    behavior::test_delete_stream

You can reproduce those tests locally by OPENDAL_TEST=alluxio cargo test behaivor --features services-alluxio

thanks, I'll fix them.

@hoslo hoslo force-pushed the alluxio-e2e-test branch 3 times, most recently from c4df8d5 to 30d3677 Compare November 13, 2023 11:06
core/src/services/alluxio/core.rs Outdated Show resolved Hide resolved
core/src/services/alluxio/core.rs Show resolved Hide resolved
core/src/services/alluxio/error.rs Show resolved Hide resolved
Copy link
Member

@Xuanwo Xuanwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot!

@Xuanwo Xuanwo dismissed Zheaoli’s stale review November 13, 2023 16:00

Already addressed.

@Xuanwo Xuanwo merged commit 47c47b7 into apache:main Nov 13, 2023
177 checks passed
@hoslo hoslo deleted the alluxio-e2e-test branch November 27, 2023 06:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants