Transfers: Implement pluggable LFN2PFN algorithms. Fixes #570

[bbockelm]

This adds the ability for a Rucio server configuration to specify a module that can register additional lfn2pfn algorithm implementations. The configuration file may also specify the default LFN2PFN algorithm (for example, replacing hash with identity or a VO-specific algorithm).

The callout algorithms receive a scope, LFN name, RSE name, RSE attributes, and protocol attributes. It should return the path portion of a PFN (not the full PFN).

The server instance's default can still be overridden by setting the lfn2pfn_algorithm attribute on the RSE.

This is the next version of #584.

[vingar]

Running flake8

lib/rucio/tests/test_rse_lfn2path.py:106:9: E306 expected 1 blank line before a nested definition, found 0

[bbockelm]

@mlassnig @vingar @bari12 - we can continue the conversation in this PR.

[vingar]

it can go in testing for me.
@bbockelm please fix the docsting of the functions

[vingar]

the docstring is incomplete . e.g.:

:param section
:param option

same comment for most of them

[bbockelm]

In Slack, @vingar asked about how this could impact current code:

1. LFN2PFN calls from the client to the server are now done with a new Rucio REST API.

   • This is used instead of listing replicas when the client needs the server to generate the PFN. This is true for the case where the RSE is non-deterministic or (new) if the client doesn't have the deterministic algorithm used.
   • On the server-side, LFN2PFN will invoke the deterministic algorithm (if it's a deterministic protocol) or do the non-deterministic lookup using list-replicas as before.

2. The new LFN2PFN calls are done if the deterministic algorithm listed in the RSE settings is not available on the client. This implies that the VO is responsible for potential mismatches (i.e., if I put an algorithm named ligo on the server, it better be the same ligo algorithm on the client -- or not present on the client at all).

3. Old clients can talk to new or old servers. New clients cannot talk to old servers for non-deterministic.

[mlassnig]

ok, even after going through it in detail i can't find something obviously wrong with this (except that the client and server should have the same version of the algorithm, which is fine). in any case we will see quickly if this breaks our integration testbed ;-) thanks @bbockelm !

[bbockelm]

Alright - merging. Thanks to everyone for sticking with me on this!