Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rucio get doesn't show summary anymore #5250

Closed
cserf opened this issue Feb 17, 2022 · 3 comments · Fixed by #5498
Closed

rucio get doesn't show summary anymore #5250

cserf opened this issue Feb 17, 2022 · 3 comments · Fixed by #5498
Assignees
@joeldierkes
Labels
bug Clients patch
Milestone
1.28.4

Comments

@cserf
Copy link
Contributor

cserf commented Feb 17, 2022

Motivation

CLI : rucio get doesn't show summary anymore
@bari12
Copy link
Member

bari12 commented Feb 18, 2022

Adding some more info from DAST

2022-01-26 18:26:01,532 INFO    Processing 1 item(s) for input
2022-01-26 18:26:02,546 INFO    No preferred protocol impl in rucio.cfg: No section: 'download'
2022-01-26 18:26:02,546 INFO    No preferred protocol impl in rucio.cfg: No section: 'download'
2022-01-26 18:26:02,546 INFO    No preferred protocol impl in rucio.cfg: No section: 'download'
2022-01-26 18:26:02,546 INFO    No preferred protocol impl in rucio.cfg: No section: 'download'
2022-01-26 18:26:02,546 INFO    No preferred protocol impl in rucio.cfg: No section: 'download'
2022-01-26 18:26:02,546 INFO    No preferred protocol impl in rucio.cfg: No section: 'download'
2022-01-26 18:26:02,546 INFO    No preferred protocol impl in rucio.cfg: No section: 'download'
2022-01-26 18:26:02,547 INFO    No preferred protocol impl in rucio.cfg: No section: 'download'
2022-01-26 18:26:02,547 INFO    No preferred protocol impl in rucio.cfg: No section: 'download'
2022-01-26 18:26:02,547 INFO    No preferred protocol impl in rucio.cfg: No section: 'download'
2022-01-26 18:26:02,547 INFO    No preferred protocol impl in rucio.cfg: No section: 'download'
2022-01-26 18:26:02,553 INFO    Using 5 threads to download 11 files
2022-01-26 18:26:02,553 INFO    Thread 0/5: Preparing download of user.XXX:user.XXX.27765327._000001.out-skim.root
2022-01-26 18:26:13,056 INFO    Thread 1/5: Preparing download of user.XXX:user.XXX.27765327._000002.out-skim.root
2022-01-26 18:26:13,057 INFO    Thread 2/5: Preparing download of user.XXX:user.XXX.27765327._000003.out-skim.root
2022-01-26 18:26:15,036 INFO    Thread 3/5: Preparing download of user.XXX:user.XXX.27765327._000004.out-skim.root
2022-01-26 18:26:37,719 INFO    Thread 0/5: File exists already locally: user.XXX:user.XXX.27765327._000001.out-skim.root
2022-01-26 18:26:37,841 INFO    Thread 4/5: Preparing download of user.XXX:user.XXX.27765327._000005.out-skim.root
2022-01-26 18:28:02,082 INFO    Thread 2/5: Trying to download with root and timeout of 62429s from UKI-LT2-QMUL_SCRATCHDISK: user.XXX:user.XXX.27765327._000003.out-skim.root
2022-01-26 18:28:14,522 INFO    Thread 2/5: Using PFN: [root://xrootd01.esc.qmul.ac.uk:1094//atlas/atlasscratchdisk/rucio/user/XXX/cf/1c/user.XXX.27765327._000003.out-skim.root]()
2022-01-26 18:29:11,262 INFO    Thread 4/5: File exists already locally: user.XXX:user.XXX.27765327._000005.out-skim.root
2022-01-26 18:29:19,224 INFO    Thread 0/5: Preparing download of user.XXX:user.XXX.27765327._000007.out-skim.root
2022-01-26 18:29:41,042 INFO    Thread 3/5: File exists already locally: user.XXX:user.XXX.27765327._000004.out-skim.root
2022-01-26 18:30:00,068 WARNING Thread 2/5: Download attempt failed. Try 1/2
2022-01-26 18:30:22,768 INFO    Thread 1/5: File exists already locally: user.XXX:user.XXX.27765327._000002.out-skim.root
2022-01-26 18:30:23,031 INFO    Thread 3/5: Preparing download of user.XXX:user.XXX.27765327._000008.out-skim.root
2022-01-26 18:31:30,560 INFO    Thread 4/5: Preparing download of user.XXX:user.XXX.27765327._000009.out-skim.root
2022-01-26 18:33:21,323 INFO    Thread 0/5: File exists already locally: user.XXX:user.XXX.27765327._000007.out-skim.root
2022-01-26 18:34:20,002 INFO    Thread 3/5: File exists already locally: user.XXX:user.XXX.27765327._000008.out-skim.root
2022-01-26 18:34:20,067 WARNING Thread 2/5: Download attempt failed. Try 2/2
2022-01-26 18:34:20,068 INFO    Thread 1/5: Preparing download of user.XXX:user.XXX.27765327._000010.out-skim.root
2022-01-26 18:35:01,948 INFO    Thread 4/5: File exists already locally: user.XXX:user.XXX.27765327._000009.out-skim.root
2022-01-26 18:35:01,951 INFO    Thread 0/5: Preparing download of user.XXX:user.XXX.27765327._000011.out-skim.root
2022-01-26 18:35:19,288 INFO    Thread 2/5: Trying to download with davs and timeout of 62429s from UKI-LT2-QMUL_SCRATCHDISK: user.XXX:user.XXX.27765327._000003.out-skim.root
2022-01-26 18:35:19,354 INFO    Thread 0/5: File exists already locally: user.XXX:user.XXX.27765327._000011.out-skim.root
2022-01-26 18:35:31,937 INFO    Thread 2/5: Using PFN: [davs://se03.esc.qmul.ac.uk:8443/webdav/atlas/atlasscratchdisk/rucio/user/XXX/cf/1c/user.XXX.27765327._000003.out-skim.root]()
2022-01-26 18:35:32,382 INFO    Thread 3/5: Preparing download of user.XXX:user.XXX.27765327._000012.out-skim.root
2022-01-26 18:35:41,709 WARNING Thread 2/5: Download attempt failed. Try 1/2
2022-01-26 18:35:43,363 INFO    Thread 3/5: File exists already locally: user.XXX:user.XXX.27765327._000012.out-skim.root
2022-01-26 18:35:53,766 INFO    Thread 1/5: File exists already locally: user.XXX:user.XXX.27765327._000010.out-skim.root
2022-01-26 18:35:53,817 WARNING Thread 2/5: Download attempt failed. Try 2/2
2022-01-26 18:35:53,977 INFO    Thread 2/5: Trying to download with storm and timeout of 62429s from UKI-LT2-QMUL_SCRATCHDISK: user.XXX:user.XXX.27765327._000003.out-skim.root
2022-01-26 18:35:54,104 INFO    Thread 2/5: Using PFN: [storm://se03.esc.qmul.ac.uk:8443/mnt/lustre_3/storm_3/atlas/atlasscratchdisk/rucio/user/XXX/cf/1c/user.XXX.27765327._000003.out-skim.root]()
2022-01-26 18:35:54,104 ERROR   Thread 2/5: Failed to download item
2022-01-26 18:35:54,105 ERROR   An unknown exception occurred.
Details: 11 items were in the input queue but only 10 are in the output queue

but no summary at the very end. The info messages are a bit different though; it gives the exact pfns which are used for the download, which is good.

@bari12
Copy link
Member

bari12 commented Feb 18, 2022

@cserf Did you have a look on this yet in respect to the old ticket?
@joeldierkes can you please have a look on this too.

@joeldierkes joeldierkes self-assigned this Feb 22, 2022
@joeldierkes
Copy link
Contributor

Okay, I had a look. At which version did the rucio download command print a summary when the number of items in the input and output queue are different?
From my POV, this never happened, which would make this Issue a feature request and not a bug.

History in the repository:
https://github.com/rucio/rucio/blob/master/lib/rucio/client/downloadclient.py#L357-L358
Prior to that:
https://github.com/rucio/rucio/blame/80aaae8b9db41755ee4fd8646e275e58f25ed028/lib/rucio/client/downloadclient.py#L347-L348

The summary can't be printed if the return statement is not reached, which means that the summary was never printed.

I also was not able to recreate the bug, which makes it harder to debug.

joeldierkes pushed a commit to joeldierkes/rucio that referenced this issue Apr 25, 2022
Clients: Return information on download exception rucio#5250
fd81b1f 
The different download functions in the `DownloadClient` return lists containing
the files and the states of the corresponding downloads. These states also
include failed states.

If an unknown exception is thrown in the `_download_item` method, the
corresponding file, which should be downloaded, is not present in the output
list. This leads to two problems; both can be solved by adding the file with the
download status `FAILED` to the output list:

- The number of files in the input queue and output queue is different, which
  throws an error. This is easily resolved by setting the file download status
  to `FAILED`. An error is later thrown.

- The client does not print the summary in case there is no items in the output
  queue. By adding the file to the output queue this problem is resolved.
joeldierkes pushed a commit to joeldierkes/rucio that referenced this issue Apr 25, 2022
Clients: Return information on download exception rucio#5250
eaf7634 
The different download functions in the `DownloadClient` return lists containing
the files and the states of the corresponding downloads. These states also
include failed states.

If an unknown exception is thrown in the `_download_item` method, the
corresponding file, which should be downloaded, is not present in the output
list. This leads to two problems; both can be solved by adding the file with the
download status `FAILED` to the output list:

- The number of files in the input queue and output queue is different, which
  throws an error. This is easily resolved by setting the file download status
  to `FAILED`. An error is later thrown.

- The client does not print the summary in case there is no items in the output
  queue. By adding the file to the output queue this problem is resolved.
@joeldierkes joeldierkes linked a pull request Apr 25, 2022 that will close this issue
Clients: Return information on download exception #5250 #5498
Merged
joeldierkes pushed a commit to joeldierkes/rucio that referenced this issue May 10, 2022
Clients: Change the download client state to an enum rucio#5250
fa501ba 
The client state uses strings to store the state. This has the problem that the
evalutation, if a client state is wrong, is done at runtime. Also, it uses the
exception name as state. This is inconsistent, because all available states
should be well defined.

This commit uses an enum for the download client state. This enables checks at
runtime, since all usable states are defined. The values of the enum ensure the
serialization to strings.

It also changes the usage from an exception name as state to the `FAILING`
state. The exception itself is added in the body already.
bari12 added a commit that referenced this issue May 12, 2022
@bari12
Merge pull request #5498 from joeldierkes/patch-5250-CLI___rucio_get_…
00bf7e2 
…doesn_t_show_summary_anymore

Clients: Return information on download exception #5250
bari12 pushed a commit that referenced this issue May 12, 2022
@bari12
Clients: Return information on download exception #5250
2d8c851 
The different download functions in the `DownloadClient` return lists containing
the files and the states of the corresponding downloads. These states also
include failed states.

If an unknown exception is thrown in the `_download_item` method, the
corresponding file, which should be downloaded, is not present in the output
list. This leads to two problems; both can be solved by adding the file with the
download status `FAILED` to the output list:

- The number of files in the input queue and output queue is different, which
  throws an error. This is easily resolved by setting the file download status
  to `FAILED`. An error is later thrown.

- The client does not print the summary in case there is no items in the output
  queue. By adding the file to the output queue this problem is resolved.
@bari12 bari12 added this to the 1.28.4 milestone May 12, 2022
@bari12 bari12 changed the title CLI : rucio get doesn't show summary anymore rucio get doesn't show summary anymore May 12, 2022
bari12 added a commit that referenced this issue May 16, 2022
@bari12
Merge pull request #5536 from joeldierkes/feature-5250-CLI_rucio_get_…
4bb2c21 
…doesn_t_show_summary_anymore

Clients: Change the download client state to an enum #5250
bari12 pushed a commit that referenced this issue May 16, 2022
@bari12
Clients: Change the download client state to an enum #5250
2658140 
The client state uses strings to store the state. This has the problem that the
evalutation, if a client state is wrong, is done at runtime. Also, it uses the
exception name as state. This is inconsistent, because all available states
should be well defined.

This commit uses an enum for the download client state. This enables checks at
runtime, since all usable states are defined. The values of the enum ensure the
serialization to strings.

It also changes the usage from an exception name as state to the `FAILING`
state. The exception itself is added in the body already.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@joeldierkes joeldierkes

Labels
bug Clients patch
Projects
None yet
Milestone
1.28.4
Development

Successfully merging a pull request may close this issue.

Clients: Return information on download exception #5250 joeldierkes/rucio
3 participants
@bari12 @joeldierkes @cserf