Client raising the wrong error

[bari12]

Motivation

From Stephane:

when I introduce a typo in the RSE name, the error message which I expect 'RSE does not exist' is very complicated :

-bash-4.1$ rucio -v download --rse MWt2_DATADISK mc16_13TeV:HITS.17430965._017098.pool.root.1
2019-03-26 10:12:16,014 INFO    Processing 1 item(s) for input
2019-03-26 10:12:16,014 DEBUG   Processing item mc16_13TeV:HITS.17430965._017098.pool.root.1
2019-03-26 10:12:16,015 DEBUG   RSE-Expression: (MWt2_DATADISK)&istape=False
2019-03-26 10:12:16,015 DEBUG   Splitted DID: mc16_13TeV:HITS.17430965._017098.pool.root.1
2019-03-26 10:12:16,055 DEBUG   1 DIDs after processing input
2019-03-26 10:12:16,056 DEBUG   Processing: {'resolve_archives': True, 'name': 'HITS.17430965._017098.pool.root.1', 'did': 'mc16_13TeV:HITS.17430965._017098.pool.root.1', 'rse': '(MWt2_DATADISK)&istape=False', 'no_subdir': False, 'nrandom': None, 'transfer_timeout': 3600, 'scope': 'mc16_13TeV', 'type': u'FILE', 'force_scheme': None, 'base_dir': '.'}
2019-03-26 10:12:16,080 DEBUG   <?xml version="1.0" encoding="UTF-8"?>
<metalink xmlns="urn:ietf:params:xml:ns:metalink">

2019-03-26 10:12:16,092 DEBUG   Traceback (most recent call last):
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.18.5/bin/rucio", line 159, in new_funct
    return function(*args, **kwargs)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.18.5/bin/rucio", line 968, in download
    result = download_client.download_dids(items, args.ndownloader, trace_pattern)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.18.5/lib/python2.7/site-packages/rucio/client/downloadclient.py", line 291, in download_dids
    input_items = self._prepare_items_for_download(items)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.18.5/lib/python2.7/site-packages/rucio/client/downloadclient.py", line 914, in _prepare_items_for_download
    files_with_pfns = self._parse_list_replica_metalink(metalink_str)
  File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.18.5/lib/python2.7/site-packages/rucio/client/downloadclient.py", line 996, in _parse_list_replica_metalink
    raise error
ExpatError: no element found: line 3, column 0

2019-03-26 10:12:16,092 ERROR   no element found: line 3, column 0
2019-03-26 10:12:16,093 ERROR   Strange error: No section: 'policy' ```

This should raise RSENotFound.

[sjezequel]

Hello,
could you also make sure that the message is clear in this case (copy on TAPE which cannot be accessed with 'rucio download'). In the previous case, it is just that the RSE does not exist.

lappsl6b.in2p3.fr> rucio -v download --rse RAL-LCG2_MCTAPE mc16_13TeV:AOD.15164692._000021.pool.root.1
2019-03-26 17:13:27,262 INFO Processing 1 item(s) for input
2019-03-26 17:13:27,263 DEBUG Processing item mc16_13TeV:AOD.15164692._000021.pool.root.1
2019-03-26 17:13:27,263 DEBUG RSE-Expression: (RAL-LCG2_MCTAPE)&istape=False
2019-03-26 17:13:27,264 DEBUG Splitted DID: mc16_13TeV:AOD.15164692._000021.pool.root.1
2019-03-26 17:13:27,361 DEBUG 1 DIDs after processing input
2019-03-26 17:13:27,361 DEBUG Processing: {'resolve_archives': True, 'name': 'AOD.15164692._000021.pool.root.1', 'did': 'mc16_13TeV:AOD.15164692._000021.pool.root.1', 'rse': '(RAL-LCG2_MCTAPE)&istape=False', 'no_subdir': False, 'nrandom': None, 'transfer_timeout': 3600, 'scope': 'mc16_13TeV', 'type': u'FILE', 'force_scheme': None, 'base_dir': '.'}
2019-03-26 17:13:27,478 DEBUG

2019-03-26 17:13:27,484 DEBUG Traceback (most recent call last):
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.18.5/bin/rucio", line 159, in new_funct
return function(*args, **kwargs)
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.18.5/bin/rucio", line 968, in download
result = download_client.download_dids(items, args.ndownloader, trace_pattern)
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.18.5/lib/python2.7/site-packages/rucio/client/downloadclient.py", line 291, in download_dids
input_items = self._prepare_items_for_download(items)
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.18.5/lib/python2.7/site-packages/rucio/client/downloadclient.py", line 914, in _prepare_items_for_download
files_with_pfns = self._parse_list_replica_metalink(metalink_str)
File "/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase/x86_64/rucio-clients/1.18.5/lib/python2.7/site-packages/rucio/client/downloadclient.py", line 996, in _parse_list_replica_metalink
raise error
ExpatError: no element found: line 3, column 0

2019-03-26 17:13:27,484 ERROR no element found: line 3, column 0
2019-03-26 17:13:27,489 ERROR Strange error: No section: 'policy'
lappsl6b.in2p3.fr> rucio list-file-replicas mc16_13TeV:AOD.15164692._000021.pool.root.1
+------------+----------------------------------+------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| SCOPE | NAME | FILESIZE | ADLER32 | RSE: REPLICA |
|------------+----------------------------------+------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| mc16_13TeV | AOD.15164692._000021.pool.root.1 | 5.798 GB | 197562c0 | BNL-OSG2_DATADISK: srm://dcsrm.usatlas.bnl.gov:8443/srm/managerv2?SFN=/pnfs/usatlas.bnl.gov/BNLT0D1/rucio/mc16_13TeV/56/7e/AOD.15164692._000021.pool.root.1 |
| mc16_13TeV | AOD.15164692._000021.pool.root.1 | 5.798 GB | 197562c0 | RAL-LCG2_MCTAPE: srm://srm-atlas.gridpp.rl.ac.uk:8443/srm/managerv2?SFN=/castor/ads.rl.ac.uk/prod/atlas/simRaw/atlasmctape/mc16_13TeV/AOD/e5814_e5984_s3126_r10724_r10726/mc16_13TeV.345120.PowhegPy8EG_NNLOPS_nnlo_30_ggH125_tautaul13l7.merge.AOD.e5814_e5984_s3126_r10724_r10726_tid15164692_00/AOD.15164692._000021.pool.root.1 |
| mc16_13TeV | AOD.15164692._000021.pool.root.1 | 5.798 GB | 197562c0 | MWT2_DATADISK: srm://uct2-dc1.uchicago.edu:8443/srm/managerv2?SFN=/pnfs/uchicago.edu/atlasdatadisk/rucio/mc16_13TeV/56/7e/AOD.15164692._000021.pool.root.1 |
+------------+----------------------------------+------------+-----------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

[bari12]

I think we could add an extra message, if the selected RSE is skipped because it is a tape RSE. That shouldn't be too difficult to add.
@TWAtGH Line 209 in the downloadclient should also be rewritten similar to line 1156.

[TWAtGH]

Hey,
the error message for a metalink parsing error did change with 1.19.
The error here is that the metalink returned by list_replicas is incomplete (I guess an exception in the REST frontend).

If list_replicas doesnt't return any replicas for a file the message is something like 'No sources found'. This message should be enough @bari12 no?
In the case of this ticket we could print a more user friendly message like 'Error in server response'.

@bari12 we already discussed in a Rucio meeting that the code of line 209 can be removed #2021

[TWAtGH]

BTW regarding 'this should raise RSENotFound'. I'm not sure about that. It's currently not really expected that list_replicas raises exceptions for reasons like this. Would need to check this.

[mlassnig]

Problem confirmed in list_replicas REST frontend. Metalink starts streaming, then later the exception is raised, causing the client to think everything is in order.