Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pluggable LFN2PFN algorithms #570

Closed
bbockelm opened this issue Jan 31, 2018 · 3 comments
Closed

Pluggable LFN2PFN algorithms #570

bbockelm opened this issue Jan 31, 2018 · 3 comments
Assignees
@bbockelm
Labels
Core & Internals feature Transfers
Milestone
1.15.0

Comments

@bbockelm
Copy link
Contributor

Previously, we introduced a mechanism to specify the algorithm used for LFN2PFN at the RSE. The options now are default and identity. The default algorithm creates subdirectories based on hashes of the LFN, while identity is the identity function: LFN is used directly.

As new communities come in, each has their own twist on the LFN2PFN mapping. Currently, there's no good way to add community-specific code for the mapping. I propose we:

  • In the Rucio server configuration, provide a module that is loaded on startup which can register additional plugins (mapping the algorithm name to a Python callable).
  • Continue to allow the algorithm to be set per-RSE as an attribute.
  • In the Rucio server configuration, specify the default algorithm to use if not specified as an attribute.

So, Rucio could continue to offer one or two stock algorithms -- but as experiments mature, this can become part of their policies.
@bari12
Copy link
Member

bari12 commented Jan 31, 2018

Fully agree with all Brian wrote 😄(This also hooks in a little bit with the way we import external policies. eventually we should have a better mechanism to import the policies, without the need of them being part of the rucio package - but lets have a separate issue for this)

Just one side note, as I looked at the code recently. For the site2site transfers a different method (construct_surl) is used to build the PFN for a site2site transferred replica. This has to be changed and the method should also inherit the algorithm from the lfn2pfn function of the respective policy.

@bbockelm bbockelm self-assigned this Feb 2, 2018
@bbockelm
Copy link
Contributor Author

bbockelm commented Feb 2, 2018

Ok, I have a work-in-progress branch for this -- will be testing this out for LIGO (where this is most urgent).

bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 4, 2018
@bbockelm
Transfers: Refactor lfn2pfn path translation to a separate class. ruc…
e0134d1 
…io#570
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 4, 2018
@bbockelm
Transfers: Add tests for LFN2PFN plugins. rucio#570
1c3cb80
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 5, 2018
@bbockelm
Transfers: Address flake8 issues. rucio#570
155afac
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 5, 2018
@bbockelm
Transfers: Refactor lfn2pfn path translation to a separate class. ruc…
307d6d3 
…io#570
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 5, 2018
@bbockelm
Transfers: Add tests for LFN2PFN plugins. rucio#570
3df95ef
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 5, 2018
@bbockelm
Transfers: Address flake8 issues. rucio#570
36e0a32
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 5, 2018
@bbockelm
Transfers: Fixup pylint suggestions. rucio#570
fb507db
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 5, 2018
@bbockelm
Transfers: Address flake8 issues generated by pylint changes. rucio#570
c836954
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 5, 2018
@bbockelm
Transfers: Fix unit test failure. rucio#570
487db95
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 5, 2018
@bbockelm
Transfers: Fix unit tests when there is no policy section. rucio#570
a86afe9
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 6, 2018
@bbockelm
Transfers: Fix flake8 issues. rucio#570
517dd68
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 6, 2018
@bbockelm
Transfers: Fixup unit tests. rucio#570
57832d3
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: Refactor lfn2pfn path translation to a separate class. ruc…
295ab31 
…io#570
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: Add tests for LFN2PFN plugins. rucio#570
9250652
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: Address flake8 issues. rucio#570
8a7807f
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: Fixup pylint suggestions. rucio#570
49c4c96
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: Address flake8 issues generated by pylint changes. rucio#570
2e159e4
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: Fix unit test failure. rucio#570
ab26e7b
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: Fix unit tests when there is no policy section. rucio#570
6494276
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: Fix flake8 issues. rucio#570
24b2de5
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: Fixup unit tests. rucio#570
4c82d9a
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: When debug is enabled, log full exception. rucio#570
412de6a
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: For unknown algorithsm, ask server-side for PFN. rucio#570
22b5b81
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: Add LFNS2PFNS RESTful API. rucio#570
43c4d11
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: Query for PFNs, not for replicas. rucio#570
c30e2f7 
Additionally adds python client API for retrieving a PFN.
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 8, 2018
@bbockelm
Transfers: fixup flake8 issuers. rucio#570
bb07003
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 9, 2018
@bbockelm
Transfers: Make sure we have sphinx-friendly function docs. rucio#570
e9ec835
bbockelm added a commit that referenced this issue Feb 9, 2018
@bbockelm
Merge pull request #585 from bbockelm/patch-570-transfers_pluggable_l…
aba7638 
…fn2pfn

Transfers: Implement pluggable LFN2PFN algorithms. Fixes #570
@bbockelm
Copy link
Contributor Author

Alright, I think this one is good to go!

@bari12 bari12 added this to the 1.15.0 milestone Feb 12, 2018
bbockelm added a commit to bbockelm/rucio that referenced this issue Feb 13, 2018
@bbockelm
clients: Make space tokens options. Fix rucio#575
a38a21b 
Outside ATLAS (particularly, for CMS), space tokens are optional at
some SRM endpoints.  Previously, for rucio#574, we fixed the conveyor
to handle empty space tokens.  Additionally, the fix for the client-side
of rucio#575 got mistakenly wrapped up in rucio#570 (pluggable LFN2PFN) and put
into `next`.

This is the missing piece for `master`.
This was referenced Feb 13, 2018
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bbockelm bbockelm

Labels
Core & Internals feature Transfers
Projects
None yet
Milestone
1.15.0
Development

No branches or pull requests
2 participants
@bari12 @bbockelm