Implement file dataset=/a/b/c site=XXX run=123 query using Rucio APIs #30
Comments
|
Almost. I think what you want to do is what this does: https://github.com/rucio/rucio/blob/0246888ceeb8cc12387c6aaffd398921b31da10e/lib/rucio/client/replicaclient.py#L117 You can pass either a container or a block and get all the file replicas, or if you pass an RSE it will give just data at that RSE. Then you probably need to filter out what Rucio gives you for the files which matched the run in your example. Of course, you could query file by file or provide a list of files, but that may be less efficient or involve transferring more data. The code shows you how to build the REST query. |
|
Eric, I still need your assistance with this as I'm getting different errors from Rucio server. So if I correctly depict replicaclient.py codebase you pointed out I came up with the following plain curl call: Here I tried two URLs: So, as you know I really need plain URL example in order to proceed with this request. Please guide me as necessary. |
|
It’s list/replicas, not list/cms/replicas
I don’t know why exactly, but your JSON is throwing an error here:
https://github.com/rucio/rucio/blob/2ff6f17c7fda45524be8e644cb85c1ed568b0bcd/lib/rucio/web/rest/webpy/v1/replica.py#L345 <https://github.com/rucio/rucio/blob/2ff6f17c7fda45524be8e644cb85c1ed568b0bcd/lib/rucio/web/rest/webpy/v1/replica.py#L345>
In fact if I check your JSON, I get:
>> a = loads('{"dids": ["scope":"cms", "name":"/JetHT/Run2018A-TkAlMinBias-12Nov2019_UL2018-v2/ALCARECO"], "domain": "all"}')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting , delimiter: line 1 column 18 (char 17)
>> a = loads('{"dids": ["scope": "cms", "name": "/JetHT/Run2018A-TkAlMinBias-12Nov2019_UL2018-v2/ALCARECO"], "domain": "all"}')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib64/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib64/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib64/python2.7/json/decoder.py", line 382, in raw_decode
obj, end = self.scan_once(s, idx)
ValueError: Expecting , delimiter: line 1 column 18 (char 17)
… On Jan 20, 2021, at 1:36 PM, Valentin Kuznetsov ***@***.***> wrote:
Eric, I still need your assistance with this as I'm getting different errors from Rucio server. So if I correctly depict replicaclient.py codebase you pointed out I came up with the following plain curl call:
#!/bin/bash
opt="-s -L -k --key $HOME/.globus/userkey.pem --cert $HOME/.globus/usercert.pem"
token=`curl $opt -v https://cms-rucio-auth.cern.ch/auth/x509 2>&1 | grep "X-Rucio-Auth-Token:" | sed -e "s,< X-Rucio-Auth-Token: ,,g"`
echo "$token"
dataset=/JetHT/Run2018A-TkAlMinBias-12Nov2019_UL2018-v2/ALCARECO
curl $opt -H "X-Rucio-Auth-Token: $token" -X POST -d '{"dids": ["scope":"cms", "name":"$dataset"], "domain": "all"}' "http://cms-rucio.cern.ch/replicas/cms/list"
Here I tried two URLs: http://cms-rucio.cern.ch/replicas/cms/list which returns internal server error, but I'm not sure if /cms should be part of URL since it does not like the case from replicaclient.py code. So I tried w/o it, i.e. http://cms-rucio.cern.ch/replicas/list which gives me a different error {"ExceptionMessage": "Cannot decode json parameter list", "ExceptionClass": "ValueError"}.
So, as you know I really need plain URL example in order to proceed with this request. Please guide me as necessary.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dmwm_das2go_issues_30-23issuecomment-2D763883603&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=mPJRZORsuklurZcOG_N0LaipVo0nQBBR5OXSTO6H6tA&s=Ety5LV7ga95YhISjLQWsmYskJWIyC141CzhFR6xUQjU&e=>, or unsubscribe <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAMYJLRNXV7FDOZRZWDKLIDS24WCHANCNFSM4WHJSYOQ&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=EHaoB-POFWGrYFvPXoj1bQ&m=mPJRZORsuklurZcOG_N0LaipVo0nQBBR5OXSTO6H6tA&s=RWNV1w6aREYlMhQl6RkoGkvaoL-QIfeJw2g_Qb4elqo&e=>.
|
|
Eric, thanks for spotting json problem. I managed to get the output with the following sequence of steps: The output looks like this now: which I hope would be sufficient for this use-case. I'll proceed with implementing necessary bits in DAS codebase. |
|
Done. The new release on cmsweb is upgraded and new dasgoclient PR is here cms-sw/cmsdist#6584 If you need a binary version of dasgoclient before it will be updated on cvmfs please take it from here: The new version is and your query looks like this: Closing the issue. |
Originally the support for
query was done through DBS and Phedex APIs. First, we resolved list of blocks for a given dataset. Then, we find files for a given set of blocks and run number, and finally filter files using Phedex fileReplicas API to select files on a given site.
Now, we need to implement the same logic using DBS and Rucio APIs. The question is do we have similar to fileReplicas Rucio API to select files only for a given site or should we find another route in Rucio to accommodate this workflow.
@ericvaandering could you please comment on this?
The text was updated successfully, but these errors were encountered: