New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[air] Add test for remote_storage with real hdfs backend. #31940

Merged

xwjiang2010 merged 18 commits into ray-project:master from xwjiang2010:hdfs

Jan 31, 2023

Contributor

xwjiang2010 commented Jan 25, 2023 •

edited

Signed-off-by: xwjiang2010 xwjiang2010@gmail.com

Why are these changes needed?

Fix get_file_and_path logic in remote_storage to take into account hdfs uri with namenode case.
set up a hdfs environment in CI to test integration like this.
dataset tests can also potentially benefit from this.

Related issue number

Closes #31673

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(


          [air] Add test for remote_storage with real hdfs backend.

d73cccc

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

Contributor Author

xwjiang2010 commented Jan 25, 2023

Let me see if the ci runs fine...


          typo

6b80da8

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

Contributor Author

xwjiang2010 commented Jan 25, 2023

Maybe related is https://github.com/ray-project/ray/pull/30611/files. Still clarifying..

xwjiang2010 added 4 commits

January 25, 2023 16:01


          typo

752b68d

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>


          try a different syntax.

f6320c1

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>


          change install-hdfs.sh permission.

65ea3b1

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>


          -hdfs in air tests.

0045d45

update ssh-kengen command.

fix a few typos.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

xwjiang2010 assigned krfricke

xwjiang2010 added 2 commits

January 26, 2023 09:53


          test_env=

51c6b44

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>


          cat hdfs_env

ad5960b

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

xwjiang2010 force-pushed the hdfs branch from 89e190f to ad5960b Compare

January 26, 2023 22:12

xwjiang2010 added 4 commits

January 26, 2023 15:04


          move PATH as well to a separate file.

4c29342

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>


          setting env vars in test only.

f274d51

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>


          fix import

2681b1a

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

fix

e0dda1d

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

krfricke requested changes

View reviewed changes

Contributor

krfricke left a comment

I have a few questions regarding the install-hdfs.sh script. I'm also happy to take a look!

Let's put the test setup into a fixture so we can re-use it across tests

ci/env/install-hdfs.sh Show resolved Hide resolved

python/ray/air/tests/test_remote_storage_hdfs.py Outdated Show resolved Hide resolved

python/ray/air/tests/test_remote_storage_hdfs.py Outdated Show resolved Hide resolved

krfricke added the @author-action-required label

xwjiang2010 added 3 commits

January 28, 2023 07:19


          address comments.

be2fd7c

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

nit

aac1231

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>


          fix fixture

ed05d2e

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

xwjiang2010 removed the @author-action-required label

krfricke approved these changes

View reviewed changes

Contributor

krfricke left a comment

Looks good to me! Just minor naming nits

ci/env/install-hdfs.sh Show resolved Hide resolved

python/ray/air/tests/test_remote_storage_hdfs.py Outdated Show resolved Hide resolved

python/ray/air/tests/test_remote_storage_hdfs.py Outdated Show resolved Hide resolved

python/ray/air/tests/test_remote_storage_hdfs.py Outdated Show resolved Hide resolved

.buildkite/pipeline.build.yml Outdated Show resolved Hide resolved

xwjiang2010 and others added 3 commits

January 30, 2023 11:25


          address comments

7bfbd24

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>


          address comments

50d29f1

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>


          Merge branch 'ray-project:master' into hdfs

Contributor Author

xwjiang2010 commented Jan 31, 2023

https://buildkite.com/ray-project/oss-ci-build-pr/builds/10913#018608ac-a154-477f-ab78-ad9acce3501c
hdfs test is passing. Failing tests are not related.

xwjiang2010 merged commit 5cf61f0 into ray-project:master

xwjiang2010 deleted the hdfs branch

January 31, 2023 17:45

clarng pushed a commit to clarng/ray that referenced this pull request


          [air] Add test for remote_storage with real hdfs backend. (ray-projec…

e7f128b

…t#31940)

* [air] Add test for remote_storage with real hdfs backend.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* try a different syntax.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* change `install-hdfs.sh` permission.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* -hdfs in air tests.

update ssh-kengen command.

fix a few typos.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* test_env=

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* cat hdfs_env

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* move `PATH` as well to a separate file.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* setting env vars in test only.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix import

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* nit

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix fixture

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

---------

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

cadedaniel pushed a commit to cadedaniel/ray that referenced this pull request


          [air] Add test for remote_storage with real hdfs backend. (ray-projec…

f4b33f2

…t#31940)

* [air] Add test for remote_storage with real hdfs backend.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* try a different syntax.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* change `install-hdfs.sh` permission.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* -hdfs in air tests.

update ssh-kengen command.

fix a few typos.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* test_env=

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* cat hdfs_env

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* move `PATH` as well to a separate file.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* setting env vars in test only.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix import

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* nit

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix fixture

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

---------

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

edoakes pushed a commit to edoakes/ray that referenced this pull request


          [air] Add test for remote_storage with real hdfs backend. (ray-projec…

97a4b91

…t#31940)

* [air] Add test for remote_storage with real hdfs backend.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* try a different syntax.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* change `install-hdfs.sh` permission.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* -hdfs in air tests.

update ssh-kengen command.

fix a few typos.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* test_env=

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* cat hdfs_env

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* move `PATH` as well to a separate file.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* setting env vars in test only.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix import

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* nit

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix fixture

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

---------

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>
Signed-off-by: Edward Oakes <ed.nmi.oakes@gmail.com>

scottsun94 pushed a commit to scottsun94/ray that referenced this pull request


          [air] Add test for remote_storage with real hdfs backend. (ray-projec…

e74b337

…t#31940)

* [air] Add test for remote_storage with real hdfs backend.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* try a different syntax.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* change `install-hdfs.sh` permission.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* -hdfs in air tests.

update ssh-kengen command.

fix a few typos.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* test_env=

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* cat hdfs_env

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* move `PATH` as well to a separate file.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* setting env vars in test only.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix import

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* nit

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix fixture

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

---------

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

cassidylaidlaw pushed a commit to cassidylaidlaw/ray that referenced this pull request


          [air] Add test for remote_storage with real hdfs backend. (ray-projec…

3bf29ab

…t#31940)

* [air] Add test for remote_storage with real hdfs backend.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* typo

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* try a different syntax.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* change `install-hdfs.sh` permission.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* -hdfs in air tests.

update ssh-kengen command.

fix a few typos.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* test_env=

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* cat hdfs_env

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* move `PATH` as well to a separate file.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* setting env vars in test only.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix import

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments.

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* nit

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* fix fixture

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

* address comments

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

---------

Signed-off-by: xwjiang2010 <xwjiang2010@gmail.com>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment