Conveyor breaks if no scheme is specified

[vingar]

Motivation

When no scheme is specified in the configuration, the conveyor fails to find a common scheme for the source and destination files, here gsiftp:

2019-03-27 08:51:47,386	3214	CRITICAL	Exception happened when trying to get transfer for request 5d9c6d99ebf04465ad3bef4d8078219c: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/rucio/core/transfer.py", line 624, in get_transfer_requests_and_source_replicas
    'schemes': __add_compatible_schemes(schemes=[matching_scheme[0]], allowed_schemes=current_schemes),
  File "/usr/local/lib/python2.7/dist-packages/rucio/core/transfer.py", line 923, in __add_compatible_schemes
    if scheme in allowed_schemes:
TypeError: argument of type 'NoneType' is not iterable

The respective rules stay in the STUCK state when asked for reevaluation:

2019-03-27 08:49:23,677	3213	INFO	rule_repairer[0/0]: Repairing rule 32d6775ead104b2c8be50cc946f01782
2019-03-27 08:49:23,749	3213	INFO	Rule 32d6775ead104b2c8be50cc946f01782 [0/0/1] state=STUCK

[bari12]

The conveyor should probably just stop starting, with an error, if no scheme is specified.

[vingar]

I would have expected to have the conveyor to figure out automatically which schemes/protocols should be used for third party transfers out of the ones available for the replicas: here it was gsiftp, davs, srm.

Note also here the non expected behaviour of the judge-repairer after the configuration has been corrected to use gsiftp.

[bari12]

It's also an option: If no scheme is given, just use all compatible ones.

For the judge-repairer I don't think anything has to be changed. The issue here is in the conveyor, as the request probably stays in it's state forever and never get's failed. Thus this is never reported back to the rule. For the judge it looks like that the request is still in submission.

[vingar]

One observation after such error: the request stays in then N, NO_SOURCES state and then disappeared. The judge-repairer keeps it in the S state forever.

[bari12]

Is it possible that there is no conveyor-finisher running? The finisher should process the NO_SOURCES ones and mark them as failed.

rucio/lib/rucio/daemons/conveyor/finisher.py

Line 113 in 39f66c0

reqs = request_core.get_next(request_type=[RequestType.TRANSFER, RequestType.STAGEIN, RequestType.STAGEOUT],

[vingar]

The finisher runs too.

[bari12]

Can you track if the replicas affected by this bug are actually handled by the finisher?
But in any way, this has to be fixed in the submitter, than the workflow through the finisher/repairer is fine again.

[vingar]

I'm seeing some messages in the finisher like requeing etc. Don't you have enough information to reproduce the problem and investigate more from your side ?

[bari12]

I only tested the NO_SOURCES one, and this works fine in my finisher. But I didn't test the full error yet. But yes, information is sufficient 👍