Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Functional download/upload tests with xrootd containers #2311

Closed
bari12 opened this issue Mar 12, 2019 · 1 comment · Fixed by #4165
Closed

Functional download/upload tests with xrootd containers #2311

bari12 opened this issue Mar 12, 2019 · 1 comment · Fixed by #4165
Assignees
@maany
Labels
enhancement Testing
Milestone
1.25.0

Comments

@bari12
Copy link
Member

bari12 commented Mar 12, 2019

Motivation

At the moment we only have very limited upload/download tests either using mock calls or direct posix protocols. One possibility to expand these to actualy functional tests would be to use xrootd docker containers and start up two xrootd storages in the test environment. This would allow us to test upload/download also with different protocols, such as gfal, which would also help enormously to validate gfal for python 3.6.
@nikmagini nikmagini self-assigned this Mar 25, 2019
maany added a commit to maany/rucio that referenced this issue Nov 26, 2020
@maany
rucio#2311
72b7b1f
maany added a commit to maany/rucio that referenced this issue Nov 26, 2020
@maany
rucio#2311 return lfn_dict if already valid, in file: lib.rucio.commo…
9294d0e 
…n.make_valid_did
maany added a commit to maany/rucio that referenced this issue Nov 26, 2020
@maany
rucio#2311 set attributes for xrd test containers and misc workarounds
0adcf58
maany added a commit to maany/rucio that referenced this issue Nov 26, 2020
@maany
rucio#2311 rucio server upload/download tests
bb182d8
maany added a commit to maany/rucio that referenced this issue Nov 26, 2020
@maany
rucio#2311 stop containers in gh action workflow
06353f0
maany added a commit to maany/rucio that referenced this issue Nov 26, 2020
@maany
rucio#2311 port xrd tests to python3
bebedc3
maany added a commit to maany/rucio that referenced this issue Nov 26, 2020
@maany
rucio#2311 python3 changes for xrootd.py, fix arguments for methods i…
3bfb8f9 
…n rsemanager.py
maany added a commit to maany/rucio that referenced this issue Nov 26, 2020
@maany
rucio#2311 update gh workflow
e77b045
maany added a commit to maany/rucio that referenced this issue Nov 26, 2020
@maany
rucio#2311 fix syntax in gh workflow
6983d32
maany added a commit to maany/rucio that referenced this issue Nov 26, 2020
@maany
cleanup rucio#2311
0d9e9ba
@maany maany linked a pull request Nov 26, 2020 that will close this issue
Testing:upload/download and xrd integration tests #2311 #4165
Merged
maany added a commit to maany/rucio that referenced this issue Nov 26, 2020
@maany
Testing: fix ci errors rucio#2311
3b0808e
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
d45ad55
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
6c0583c
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
a426471
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
20c3b80
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
e10315d
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
dd6991c
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
c104317
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
de4a3bd
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
6c0619b
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
d166b9a
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
9abc9c5
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
e64505d
maany added a commit to maany/rucio that referenced this issue Dec 3, 2020
@maany
Testing: build images rucio#2311
b49809e
maany added a commit to maany/rucio that referenced this issue Jan 21, 2021
@maany
Testing: cleanup GH workflows rucio#2311
930c883
maany added a commit to maany/rucio that referenced this issue Jan 21, 2021
@maany
Testing: cleanup GH workflows rucio#2311
f466690
@rcarpa
Copy link
Contributor

rcarpa commented Feb 12, 2021

I'm working on increasing the coverage of the tests in test_download.py. I need better tests to be able to work more confidently on #4153 . Let me know if I risk to do duplicate work.

rcarpa added a commit to rcarpa/rucio that referenced this issue Feb 16, 2021
@rcarpa
tests: improve download client test coverage; rucio#2311
cfb1bea 
- check the status of different results to ensure that files are
  correctly overwritten or not
- check the resulting path with/without no_subdir set
- add test for downloading from an archive
- implicitly test XRD download because it is used in the archive test
- add tests for downloading multiple files via a wildcard
- implicitly test some upload logic
rcarpa added a commit to rcarpa/rucio that referenced this issue Feb 16, 2021
@rcarpa
Testing: improve download client test coverage; rucio#2311
bb15d5f 
- check the status of different results to ensure that files are
  correctly overwritten or not
- check the resulting path with/without no_subdir set
- add test for downloading from an archive
- implicitly test XRD download because it is used in the archive test
- add tests for downloading multiple files via a wildcard
- implicitly test some upload logic
rcarpa added a commit to rcarpa/rucio that referenced this issue Feb 16, 2021
@rcarpa
Testing: improve download client test coverage; rucio#2311
f6a17a5 
- check the status of different results to ensure that files are
  correctly overwritten or not
- check the resulting path with/without no_subdir set
- add test for downloading from an archive
- implicitly test XRD download because it is used in the archive test
- add tests for downloading multiple files via a wildcard
- implicitly test some upload logic
rcarpa added a commit to rcarpa/rucio that referenced this issue Feb 16, 2021
@rcarpa
Testing: improve download client test coverage; rucio#2311
8bd607c 
- check the status of different results to ensure that files are
  correctly overwritten or not
- check the resulting path with/without no_subdir set
- add test for downloading from an archive
- implicitly test XRD download because it is used in the archive test
- add tests for downloading multiple files via a wildcard
- implicitly test some upload logic
rcarpa added a commit to rcarpa/rucio that referenced this issue Feb 16, 2021
@rcarpa
Testing: improve download client test coverage; rucio#2311
89703f4 
- check the status of different results to ensure that files are
  correctly overwritten or not
- check the resulting path with/without no_subdir set
- add test for downloading from an archive
- implicitly test XRD download because it is used in the archive test
- add tests for downloading multiple files via a wildcard
- implicitly test some upload logic
- some verifications that trace_copy_out works as expected
- ensure that checksum validation is enforced (or ignored if option set)
rcarpa added a commit to rcarpa/rucio that referenced this issue Feb 16, 2021
@rcarpa
Testing: improve download client test coverage; rucio#2311
92ca838 
- check the status of different results to ensure that files are
  correctly overwritten or not
- check the resulting path with/without no_subdir set
- add test for downloading from an archive
- implicitly test XRD download because it is used in the archive test
- add tests for downloading multiple files via a wildcard
- implicitly test some upload logic
- some verifications that trace_copy_out works as expected
- ensure that checksum validation is enforced (or ignored if option set)
rcarpa added a commit to rcarpa/rucio that referenced this issue Feb 17, 2021
@rcarpa
Testing: improve download client test coverage; rucio#2311
6436eb8 
- check the status of different results to ensure that files are
  correctly overwritten or not
- check the resulting path with/without no_subdir set
- add test for downloading from an archive
- implicitly test XRD download because it is used in the archive test
- add tests for downloading multiple files via a wildcard
- implicitly test some upload logic
- some verifications that trace_copy_out works as expected
- ensure that checksum validation is enforced (or ignored if option set)
bari12 added a commit that referenced this issue Feb 22, 2021
@bari12
Merge pull request #4340 from rcarpa/patch-2311-more_download_tests
13c4108 
Testing: improve download client test coverage; #2311
bari12 pushed a commit that referenced this issue Feb 22, 2021
@rcarpa @bari12
Testing: improve download client test coverage; #2311
02b340f 
- check the status of different results to ensure that files are
  correctly overwritten or not
- check the resulting path with/without no_subdir set
- add test for downloading from an archive
- implicitly test XRD download because it is used in the archive test
- add tests for downloading multiple files via a wildcard
- implicitly test some upload logic
- some verifications that trace_copy_out works as expected
- ensure that checksum validation is enforced (or ignored if option set)
bari12 pushed a commit that referenced this issue Feb 22, 2021
@maany
Testing:upload/download and xrd integration tests #2311 (#4165)
dd149ad 
* #2311 
* #2311 return lfn_dict if already valid, in file: lib.rucio.common.make_valid_did
* #2311 set attributes for xrd test containers and misc workarounds
* #2311 rucio server upload/download tests
* #2311 stop containers in gh action workflow
* #2311 port xrd tests to python3
* #2311 python3 changes for xrootd.py, fix arguments for methods in rsemanager.py
* cleanup #2311
* Testing: fix ci errors #2311
* Testing: build and cache rucio-dev images #2311
* Testing: enable protocol tests only in CI #2311
* Testing: imagecaching for image used by integration tests #2311
* Testing: filter matrix in e2e test workflow #2311
* Testing: filter matrix in autotest, build image in e2e tests #2311
* Testing: Use jq to extract e2e test matrix #2311
* Testing: change image building and caching in e2e workflow #2311
* Testing: fix matrix for autotests #2311
* Testing: fix jq filtering for matrix in autotest workflow #2311
* Testing: change pytest args #2311
* Testing: revert autotest workflow #2311
* Testing: refactor build_images and autotests #2311
* Testing: refactor build_images, add new matrix #2311
* Testing: add integration_test image identifier #2311
* Testing: remove integration_test suite from autotest matrix #2311
* Testing: cleanup debug code in build_images.py #2311
* Testing: cleanup debug code in build_images.py #2311
* Testing: debug autotests #2311
* Testing: fix integration-tests #2311
* Testing: use rucio/containers for e2e tests #2311
* Testing: remove integration_test suite from autotest matrix #2311
* Testing: cleanup GH workflows #2311
* Testing: cleanup GH workflows #2311
@bari12 bari12 added this to the 1.25.0 "Rat-Donkey" milestone Feb 22, 2021
rcarpa added a commit to rcarpa/rucio that referenced this issue Mar 1, 2021
@rcarpa
Testing: Activate XRD archive download test on integration. rucio#2311
d32b21d 
The dependency commit was merged.
@bari12 bari12 mentioned this issue Mar 10, 2021
Closed
rcarpa added a commit to rcarpa/rucio that referenced this issue Mar 16, 2021
@rcarpa
Testing: Activate XRD archive download test on integration. rucio#2311
bf8a6ff 
The dependency commit was merged.
rcarpa added a commit to rcarpa/rucio that referenced this issue Mar 19, 2021
@rcarpa
Testing: Activate XRD archive download test on integration. rucio#2311
bd711ce 
The dependency commit was merged.
rcarpa added a commit to rcarpa/rucio that referenced this issue Mar 19, 2021
@rcarpa
Testing: Activate XRD archive download test on integration. rucio#2311
4dd0f1a 
The dependency commit was merged.
rcarpa added a commit to rcarpa/rucio that referenced this issue Mar 19, 2021
@rcarpa
Testing: mark protocol xrootd tests noparallel. rucio#2311
89cace9 
Each test creates a directory and then removes it. One test can remove
the directory while another test needs it.
rcarpa added a commit to rcarpa/rucio that referenced this issue Mar 19, 2021
@rcarpa
Testing: use main Dockerfile for integration tests. rucio#2311
391c5f3 
The *_py3 one was removed.
rcarpa added a commit to rcarpa/rucio that referenced this issue Mar 22, 2021
@rcarpa
Testing: Activate XRD archive download test on integration. rucio#2311
4f81c8f 
The dependency commit was merged.
rcarpa added a commit to rcarpa/rucio that referenced this issue Mar 22, 2021
@rcarpa
Testing: mark protocol xrootd tests noparallel. rucio#2311
a76e875 
Each test creates a directory and then removes it. One test can remove
the directory while another test needs it.
rcarpa added a commit to rcarpa/rucio that referenced this issue Apr 26, 2021
@rcarpa
Testing: Activate XRD archive download test on integration. rucio#2311
a9b4d86 
The dependency commit was merged.
rcarpa added a commit to rcarpa/rucio that referenced this issue Apr 26, 2021
@rcarpa
Testing: mark protocol xrootd tests noparallel. rucio#2311
9450de2 
Each test creates a directory and then removes it. One test can remove
the directory while another test needs it.
rcarpa added a commit to rcarpa/rucio that referenced this issue Apr 26, 2021
@rcarpa
Testing: Activate XRD archive download test on integration. rucio#2311
2e4bcf4 
The dependency commit was merged.
rcarpa added a commit to rcarpa/rucio that referenced this issue Apr 26, 2021
@rcarpa
Testing: mark protocol xrootd tests noparallel. rucio#2311
e5ff279 
Each test creates a directory and then removes it. One test can remove
the directory while another test needs it.
bari12 pushed a commit that referenced this issue Apr 27, 2021
@rcarpa
Testing: Re-activate integration tests. #2311 (#4578)
a053913 
* Testing: Activate XRD archive download test on integration. #2311

The dependency commit was merged.

* Testing: search rse-repository.json and rse_account.cfg in config dirs

otherwise tests fail when not run from the project base dir

Integration tests got silently disabled because the script which runs
tests was updated to "cd" into a different directory.

* Testing: mark protocol xrootd tests noparallel. #2311

Each test creates a directory and then removes it. One test can remove
the directory while another test needs it.
bari12 pushed a commit that referenced this issue Apr 27, 2021
@rcarpa @bari12
Testing: Re-activate integration tests. #2311 (#4578)
0003068 
* Testing: Activate XRD archive download test on integration. #2311

The dependency commit was merged.

* Testing: search rse-repository.json and rse_account.cfg in config dirs

otherwise tests fail when not run from the project base dir

Integration tests got silently disabled because the script which runs
tests was updated to "cd" into a different directory.

* Testing: mark protocol xrootd tests noparallel. #2311

Each test creates a directory and then removes it. One test can remove
the directory while another test needs it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maany maany

Labels
enhancement Testing
Projects
None yet
Milestone
1.25.0
Development

Successfully merging a pull request may close this issue.

Testing:upload/download and xrd integration tests #2311 maany/rucio
4 participants
@bari12 @rcarpa @nikmagini @maany