Transfer with xroot using POSC fails for token with storage.modify #7129
Comments
|
Petr, just so I understand: After #6952 was resolved, you updated to 8.2.13 and this worked. Now you updated from 8.2.13 to 8.2.20 with no intervening updates, and this no longer works. Correct? If so, then the only change I can see which was made that might be affecting this would be the resolve path (symlinks) change from 8.2.17 and 8.2.19. You can see this is touched from the output: I don't, however, currently understand why that messes up the RESTRICTIONS. How difficult would it be for you to try 8.2.16 and 8.2.17 and compare the results? If that is problematic for you, then I will try to see if I can set up my testbed to use tokens with your structure. Al |
|
I just downgraded my testbed (I probably never tested "gfal-copy" upload with 8.2.13) |
|
Ah, well that's progress, thanks. Dumb question: you are sure that when you upgrade dCache that your gPlazma cell(s) are also running 8.2.13? That is where the change was made. |
|
Yes, my testbed host all services on one node => one dCache version. |
|
OK, just double-checking. I will have to try to reproduce your setup on my testbed. Might you post your gPlazma token settings so I can try to mirror them? Thanks again, Al |
|
My layout file is following (minus unrelated protocol & pools) |
|
Great, thanks again Petr |
|
Petr, I think I'm going to need to see your gplazma.conf and also your authorization conf relevant to VO wlcg as well ... sorry. |
|
If you want to move this to RT to protect private data, please do. |
|
My current dCache testbed configuration |
|
👍
…________________________________________________
Albert L. Rossi
Senior Software Developer
Scientific Computing Division, Scientific Data Services, Distributed Data Development
WH 566
Fermi National Accelerator Laboratory
Batavia, IL 60510
(630) 840-3023
________________________________
From: vokac ***@***.***>
Sent: Thursday, April 20, 2023 9:10 AM
To: dCache/dcache ***@***.***>
Cc: Albert Rossi ***@***.***>; Comment ***@***.***>
Subject: Re: [dCache/dcache] Transfer with xroot using POSC fails for token with storage.modify (Issue #7129)
My current dCache testbed configuration<https://vokac.web.cern.ch/vokac/tmp/dcache.farm.particle.cz-etc-dcache.tar.gz>
—
Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_dCache_dcache_issues_7129-23issuecomment-2D1516400437&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=6m0JxkOeU6oVbtvyR6gSGuA4JV4hKf-AJVLa10yf7Nr5diUQjFrM9Z68z0VNKm7A&s=95m003HflA2McBQXobfBdhDqltENXIqRdJv6rU5QurA&e=>, or unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AA6NBHBVBOVIYMR7AHKTBT3XCE7T5ANCNFSM6AAAAAAXEX2JDQ&d=DwMCaQ&c=gRgGjJ3BkIsb5y6s49QqsA&r=60rQ0HHqHmEY1P6VSdyuTQ&m=6m0JxkOeU6oVbtvyR6gSGuA4JV4hKf-AJVLa10yf7Nr5diUQjFrM9Z68z0VNKm7A&s=wWiTAujq9x-6beeUT0tfBykl8Nrx91h2VcMCSxhGc9g&e=>.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
OK. Mirroring minimally your configuration.
Now, without POSC: and with POSC result: So @vokac Petr, what have I not caught that is particular to your setup? Do you see any glaring differences? Thanks, Al |
|
Good news - with your configuration everything works for me, but when I return to my more fancy configuration it stops working with |
|
Thanks. The investigation will be very useful one way or the other. |
|
Finally I found that by using
in your configuration transfers with POSC starts to fail. Why setting different root of namespace leads to different behavior for xroots transfers with / without POSC? |
|
Petr, I can't seem to reproduce this error. Changing to: Doing -P: You can see by the timestamp that the upload dir was created, the file written there, and then moved: Without -P also works: Looks like it is something else ... ? |
|
I don't understand what's wrong with my configuration: $ rpm -qa dcache
dcache-8.2.21.ceb2175-1.noarch
$ cat /etc/dcache/dcache.conf | grep -v ^# | grep -v '^ *$'
dcache.java.memory.heap = 2048m
dcache.java.memory.direct = 2048m
dcache.layout = mylayout-tokens
dcache.default-retention-policy=REPLICA
dcache.default-access-latency=ONLINE
dcache.authn.crl-mode = IF_VALID
pool.wait-for-files = ${path}/data
pool.tags = hostname=${host.fqdn}
dcache.upload-directory=/upload
$ cat /etc/dcache/layouts/mylayout-tokens.conf | grep -v ^# | grep -v '^ *$'
[centralDomain]
dcache.broker.scheme = core
[centralDomain/zookeeper]
[centralDomain/pnfsmanager]
[centralDomain/cleaner]
[centralDomain/poolmanager]
[centralDomain/spacemanager]
[centralDomain/pinmanager]
[centralDomain/billing]
[centralDomain/admin]
[centralDomain/gplazma]
gplazma.configuration.file=/etc/dcache/gplazma-tokens.conf
gplazma.oidc.provider!demo=https://demo.scitokens.org -profile=wlcg -prefix=/demo
gplazma.oidc.provider!wlcg=https://wlcg.cloud.cnaf.infn.it/ -profile=wlcg -prefix=/wlcg
gplazma.oidc.provider!atlas=https://atlas-auth.web.cern.ch/ -profile=wlcg -prefix=/atlas
gplazma.oidc.audience-targets=https://wlcg.cern.ch/jwt/v1/any https://dcache.farm.particle.cz roots://dcache.farm.particle.cz:1094 https://demo.scitokens.org
gplazma.authzdb.file=/etc/dcache/storage-authzdb
[doorsDomain]
[doorsDomain/webdav]
webdav.authn.protocol = https
webdav.net.port = 443
[doorsDomain/xrootd]
xrootd.authz.write-paths = /
[poolsDomain1]
[poolsDomain1/pool]
pool.name=pool1
pool.path=/srv/dcache/pool-1
pool.size=50G
pool.wait-for-files=${pool.path}/data
[poolsDomain2]
[poolsDomain2/pool]
pool.name=pool2
pool.path=/srv/dcache/pool-2
pool.size=50G
pool.wait-for-files=${pool.path}/data
[poolsDomain3]
[poolsDomain3/pool]
pool.name=pool3
pool.path=/srv/dcache/pool-3
pool.size=50G
pool.wait-for-files=${pool.path}/data
$ cat /etc/dcache/gplazma-tokens.conf | grep -v ^# | grep -v '^ *$'
auth sufficient oidc
map sufficient multimap gplazma.multimap.file=/etc/dcache/multimap.conf
map optional authzdb
session sufficient authzdb
$ cat /etc/dcache/storage-authzdb | grep -v ^# | grep -v '^ *$'
version 2.1
authorize wlcg read-write 1199 1199 / / /
authorize demo read-write 1199 1199 / / /
authorize atlas read-write 2199 2199 / / /
$ cat /etc/dcache/multimap.conf | grep -v ^# | grep -v '^ *$'
op:demo username:demo uid:1199 gid:1199,true
op:wlcg username:wlcg uid:1199 gid:1199,true
op:atlas username:atlas uid:2199 gid:2199,truebecause transfer with this slightly weird token (WLCG and SCITOKEN version in one token): {
"wlcg.ver": "1.0",
"sub": "58280cfd-ed7f-4954-90c7-cfde610cb963",
"aud": "https://wlcg.cern.ch/jwt/v1/any",
"scope": "storage.modify:/",
"iss": "https://demo.scitokens.org",
"exp": 1683030837,
"iat": 1683030237,
"nbf": 1683030237,
"jti": "1a65b297-c950-4053-8315-866af498f565",
"client_id": "a6d4bb06-a7ef-40c8-92de-bc5b5e140609",
"ver": "scitoken:2.0"
}still fails with $ xrdcopy -P /etc/services roots://dcache.farm.particle.cz:1094//demo/x
[0B/0B][100%][==================================================][0B/s]
Run: [ERROR] Server responded with an error: [3010] Restriction MultiTargetedRestriction[Authorisation{allowing [LIST, MANAGE, UPLOAD, DELETE, READ_METADATA, UPDATE_METADATA] on /demo}] denied activity UPLOAD on /upload/1/57e02248-6257-4739-b514-0775ca0a7ca6/x (destination)
$ xrdcopy /etc/services roots://dcache.farm.particle.cz:1094//demo/x
[654.6kB/654.6kB][100%][==================================================][654.6kB/s]
$ xrdfs roots://dcache.farm.particle.cz rm /demo/x
rm /demo/x : [SUCCESS] Same results also with normal WLCG JWT ATLAS tokens |
|
Wow. OK, let me start from scratch here, use a single domain and exactly the same branch. Maybe there is a bug in 8.2.21 that is not in other branches. I'll report back. |
|
Indeed. Using 8.2.21, 9.0.4 and master, with your layout and conf, I get your error: Really scratching my head on this one. Will continue to investigate ... |
|
I'm happy that you are able to reproduce this behavior (I just hope that it is not just some simple typo in my config file, because in the past such mistakes already caused me quite a lot of headache) |
|
Actually, there was a bug in the code and for some reason my original testing did not uncover it. posting a fix. (It looks like it was your original observation concerning the user home/root). Would you be willing to try out an 8.2 version of the patch? |
|
sure |
|
Please give this a try. https://drive.google.com/file/d/1ztj51YhIjolSX-0v7CD_C7pCwJ0--igB/view?usp=share_link |
|
I can confirm that copy with Now finally also gfal-copy succeeds, so far everything OK for $ gfal-copy /etc/services roots://dcache.farm.particle.cz:1094//atlas/atlasscratchdisk/SAM/x
Copying file:///etc/services [DONE] after 0s
$ gfal-rm roots://dcache.farm.particle.cz:1094//atlas/atlasscratchdisk/SAM/x
roots://dcache.farm.particle.cz:1094//atlas/atlasscratchdisk/SAM/x DELETED
$ xrdcopy -P /etc/services roots://dcache.farm.particle.cz:1094//atlas/atlasscratchdisk/SAM/x
[654.6kB/654.6kB][100%][==================================================][654.6kB/s]
$ xrdfs roots://dcache.farm.particle.cz rm /atlas/atlasscratchdisk/SAM/x
rm /atlas/atlasscratchdisk/SAM/x : [SUCCESS]
$ xrdcopy /etc/services roots://dcache.farm.particle.cz:1094//atlas/atlasscratchdisk/SAM/x
[654.6kB/654.6kB][100%][==================================================][654.6kB/s]
$ xrdfs roots://dcache.farm.particle.cz rm /atlas/atlasscratchdisk/SAM/x
rm /atlas/atlasscratchdisk/SAM/x : [SUCCESS] ATLAS token {
"wlcg.ver": "1.0",
"sub": "b41bd224-951e-47b9-8f86-c234e491d8b4",
"aud": "https://wlcg.cern.ch/jwt/v1/any",
"nbf": 1683120158,
"scope": "storage.modify:/atlasscratchdisk/SAM/ storage.read:/atlasscratchdisk/SAM/",
"iss": "https://atlas-auth.web.cern.ch/",
"exp": 1683123758,
"iat": 1683120158,
"jti": "b809d0b0-1216-4037-b7f8-0c3e2d1aea7c",
"client_id": "aa69c0b7-c8ea-41be-a8bb-614f3fc82541"
} |
|
Great, thanks again Petr. |
alrossi
added a commit
to alrossi/dcache-server
that referenced
this issue
May 3, 2023
https://rb.dcache.org/r/13855/ master@40dc86223b1f664f6dad5eff4ec222fae5631d3e was intended to allow upload (for xrootd, POSC) when there are Multitargeted Restrictions from tokens. However, as dCache#7129 (comment) `Transfer with xroot using POSC fails for token with storage.modify` indicates, there is a bug which is erroneously excluding user roots equivalent to '/'. Modification: Remove the conditional excluding '/'. Result: POSC works with tokens when user root is '/'. Target: master Request: 9.0 Request: 8.2 Request: 8.1 Request: 8.0 Request: 7.2 Closes: dCache#7129 Requires-notes: yes Patch: https://rb.dcache.org/r/13973/ Acked-by: Dmitry
alrossi
added a commit
to alrossi/dcache-server
that referenced
this issue
May 3, 2023
https://rb.dcache.org/r/13855/ master@40dc86223b1f664f6dad5eff4ec222fae5631d3e was intended to allow upload (for xrootd, POSC) when there are Multitargeted Restrictions from tokens. However, as dCache#7129 (comment) `Transfer with xroot using POSC fails for token with storage.modify` indicates, there is a bug which is erroneously excluding user roots equivalent to '/'. Modification: Remove the conditional excluding '/'. Result: POSC works with tokens when user root is '/'. Target: master Request: 9.0 Request: 8.2 Request: 8.1 Request: 8.0 Request: 7.2 Closes: dCache#7129 Requires-notes: yes Patch: https://rb.dcache.org/r/13973/ Acked-by: Dmitry
alrossi
added a commit
to alrossi/dcache-server
that referenced
this issue
May 3, 2023
https://rb.dcache.org/r/13855/ master@40dc86223b1f664f6dad5eff4ec222fae5631d3e was intended to allow upload (for xrootd, POSC) when there are Multitargeted Restrictions from tokens. However, as dCache#7129 (comment) `Transfer with xroot using POSC fails for token with storage.modify` indicates, there is a bug which is erroneously excluding user roots equivalent to '/'. Modification: Remove the conditional excluding '/'. Result: POSC works with tokens when user root is '/'. Target: master Request: 9.0 Request: 8.2 Request: 8.1 Request: 8.0 Request: 7.2 Closes: dCache#7129 Requires-notes: yes Patch: https://rb.dcache.org/r/13973/ Acked-by: Dmitry
alrossi
added a commit
to alrossi/dcache-server
that referenced
this issue
May 3, 2023
https://rb.dcache.org/r/13855/ master@40dc86223b1f664f6dad5eff4ec222fae5631d3e was intended to allow upload (for xrootd, POSC) when there are Multitargeted Restrictions from tokens. However, as dCache#7129 (comment) `Transfer with xroot using POSC fails for token with storage.modify` indicates, there is a bug which is erroneously excluding user roots equivalent to '/'. Modification: Remove the conditional excluding '/'. Result: POSC works with tokens when user root is '/'. Target: master Request: 9.0 Request: 8.2 Request: 8.1 Request: 8.0 Request: 7.2 Closes: dCache#7129 Requires-notes: yes Patch: https://rb.dcache.org/r/13973/ Acked-by: Dmitry
Merged
alrossi
added a commit
to alrossi/dcache-server
that referenced
this issue
May 3, 2023
https://rb.dcache.org/r/13855/ master@40dc86223b1f664f6dad5eff4ec222fae5631d3e was intended to allow upload (for xrootd, POSC) when there are Multitargeted Restrictions from tokens. However, as dCache#7129 (comment) `Transfer with xroot using POSC fails for token with storage.modify` indicates, there is a bug which is erroneously excluding user roots equivalent to '/'. Modification: Remove the conditional excluding '/'. Result: POSC works with tokens when user root is '/'. Target: master Request: 9.0 Request: 8.2 Request: 8.1 Request: 8.0 Request: 7.2 Closes: dCache#7129 Requires-notes: yes Patch: https://rb.dcache.org/r/13973/ Acked-by: Dmitry
This was referenced May 3, 2023
Merged
Merged
Merged
Merged
lemora
pushed a commit
that referenced
this issue
May 12, 2023
https://rb.dcache.org/r/13855/ master@40dc86223b1f664f6dad5eff4ec222fae5631d3e was intended to allow upload (for xrootd, POSC) when there are Multitargeted Restrictions from tokens. However, as #7129 (comment) `Transfer with xroot using POSC fails for token with storage.modify` indicates, there is a bug which is erroneously excluding user roots equivalent to '/'. Modification: Remove the conditional excluding '/'. Result: POSC works with tokens when user root is '/'. Target: master Request: 9.0 Request: 8.2 Request: 8.1 Request: 8.0 Request: 7.2 Closes: #7129 Requires-notes: yes Patch: https://rb.dcache.org/r/13973/ Acked-by: Dmitry
lemora
pushed a commit
that referenced
this issue
May 12, 2023
https://rb.dcache.org/r/13855/ master@40dc86223b1f664f6dad5eff4ec222fae5631d3e was intended to allow upload (for xrootd, POSC) when there are Multitargeted Restrictions from tokens. However, as #7129 (comment) `Transfer with xroot using POSC fails for token with storage.modify` indicates, there is a bug which is erroneously excluding user roots equivalent to '/'. Modification: Remove the conditional excluding '/'. Result: POSC works with tokens when user root is '/'. Target: master Request: 9.0 Request: 8.2 Request: 8.1 Request: 8.0 Request: 7.2 Closes: #7129 Requires-notes: yes Patch: https://rb.dcache.org/r/13973/ Acked-by: Dmitry
lemora
pushed a commit
that referenced
this issue
May 12, 2023
https://rb.dcache.org/r/13855/ master@40dc86223b1f664f6dad5eff4ec222fae5631d3e was intended to allow upload (for xrootd, POSC) when there are Multitargeted Restrictions from tokens. However, as #7129 (comment) `Transfer with xroot using POSC fails for token with storage.modify` indicates, there is a bug which is erroneously excluding user roots equivalent to '/'. Modification: Remove the conditional excluding '/'. Result: POSC works with tokens when user root is '/'. Target: master Request: 9.0 Request: 8.2 Request: 8.1 Request: 8.0 Request: 7.2 Closes: #7129 Requires-notes: yes Patch: https://rb.dcache.org/r/13973/ Acked-by: Dmitry
lemora
pushed a commit
that referenced
this issue
May 12, 2023
https://rb.dcache.org/r/13855/ master@40dc86223b1f664f6dad5eff4ec222fae5631d3e was intended to allow upload (for xrootd, POSC) when there are Multitargeted Restrictions from tokens. However, as #7129 (comment) `Transfer with xroot using POSC fails for token with storage.modify` indicates, there is a bug which is erroneously excluding user roots equivalent to '/'. Modification: Remove the conditional excluding '/'. Result: POSC works with tokens when user root is '/'. Target: master Request: 9.0 Request: 8.2 Request: 8.1 Request: 8.0 Request: 7.2 Closes: #7129 Requires-notes: yes Patch: https://rb.dcache.org/r/13973/ Acked-by: Dmitry
lemora
pushed a commit
that referenced
this issue
May 12, 2023
https://rb.dcache.org/r/13855/ master@40dc86223b1f664f6dad5eff4ec222fae5631d3e was intended to allow upload (for xrootd, POSC) when there are Multitargeted Restrictions from tokens. However, as #7129 (comment) `Transfer with xroot using POSC fails for token with storage.modify` indicates, there is a bug which is erroneously excluding user roots equivalent to '/'. Modification: Remove the conditional excluding '/'. Result: POSC works with tokens when user root is '/'. Target: master Request: 9.0 Request: 8.2 Request: 8.1 Request: 8.0 Request: 7.2 Closes: #7129 Requires-notes: yes Patch: https://rb.dcache.org/r/13973/ Acked-by: Dmitry
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With latest dCache 8.2.20 I'm getting same error as discussed in #6952 (xroot with POSC transfers using scope based authorization fails), e.g.
for WLCG JWT token
{ "wlcg.ver": "1.0", "sub": "58280cfd-ed7f-4954-90c7-cfde610cb963", "aud": "https://wlcg.cern.ch/jwt/v1/any", "nbf": 1681947649, "scope": "storage.modify:/", "iss": "https://wlcg.cloud.cnaf.infn.it/", "exp": 1681951249, "iat": 1681947649, "jti": "c1cd1fb7-7819-4c06-aaf3-8a3fb390d80d", "client_id": "a6d4bb06-a7ef-40c8-92de-bc5b5e140609" }Our dCache journal show following details
Transfers using xroot with POSC works with group based authorization /
wlcg.groups.The text was updated successfully, but these errors were encountered: