Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve][Connector-v2] The hive connector support multiple filesystem #6648

Merged
merged 6 commits into from
Apr 18, 2024

Conversation

dailai
Copy link
Contributor

@dailai dailai commented Apr 6, 2024

Purpose of this pull request

Hive connector support multiple filesystem for issue #6641

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@dailai dailai force-pushed the support-multiple-file-system-for-hive branch 2 times, most recently from 361b673 to f6db3d8 Compare April 6, 2024 11:17
@hailin0
Copy link
Contributor

hailin0 commented Apr 7, 2024

cc @EricJoy2048

@EricJoy2048
Copy link
Member

cc @EricJoy2048

Thanks, I will have a look.

@EricJoy2048
Copy link
Member

Please add e2e test for this pr.

@dailai
Copy link
Contributor Author

dailai commented Apr 7, 2024

Please add e2e test for this pr.

The hive connector has no e2e tests before. And there doesn't seem to be any mirrors that can support this e2e test. I can provide e2e test cases, but I don't have test environments for cos, oss, and s3.

@EricJoy2048
Copy link
Member

Please add e2e test for this pr.

The hive connector has no e2e tests before. And there doesn't seem to be any mirrors that can support this e2e test. I can provide e2e test cases, but I don't have test environments for cos, oss, and s3.

You can add e2e test case and then test in your local and upload a screenshot of successful testing. Before you push code, you can add @Disable to disable the e2e.

@dailai dailai force-pushed the support-multiple-file-system-for-hive branch from 2d797b4 to fd1292a Compare April 10, 2024 01:56
@dailai
Copy link
Contributor Author

dailai commented Apr 10, 2024

HDFS:
hive_on_hdfs
hive_on_hdfs_data

@dailai dailai force-pushed the support-multiple-file-system-for-hive branch 3 times, most recently from e42d697 to 048c7ae Compare April 12, 2024 03:16
@dailai
Copy link
Contributor Author

dailai commented Apr 12, 2024

EMR(Hive on s3):

  1. write:
截屏2024-04-12 11 21 44 截屏2024-04-12 11 22 02 截屏2024-04-12 11 26 11
  1. read:
截屏2024-04-12 11 22 56 截屏2024-04-12 11 23 13

@dailai dailai requested a review from EricJoy2048 April 12, 2024 03:29
@dailai dailai force-pushed the support-multiple-file-system-for-hive branch from 048c7ae to 25b49be Compare April 12, 2024 05:19
@dailai
Copy link
Contributor Author

dailai commented Apr 16, 2024

EMR(Hive on oss):

  1. write:
截屏2024-04-16 16 30 13 截屏2024-04-16 16 30 44 截屏2024-04-16 16 31 23
  1. read:
截屏2024-04-16 16 35 25 截屏2024-04-16 16 35 43

@dailai dailai force-pushed the support-multiple-file-system-for-hive branch from 88ac9d4 to 2b0a92a Compare April 17, 2024 02:38
Copy link
Member

@EricJoy2048 EricJoy2048 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, Please waiting for CI complete.

@hailin0 hailin0 merged commit 8a4c01f into apache:dev Apr 18, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Connector-V2] The hive connector should support cos、oss、s3 file system.
3 participants