Default multihop tombstone is effectively None

[nsmith-]

Motivation

In

rucio/lib/rucio/core/transfer.py

Line 103 in d59c237

DEFAULT_MULTIHOP_TOMBSTONE_DELAY = datetime.timedelta(hours=2)

a DEFAULT_MULTIHOP_TOMBSTONE_DELAY is set, and then in

rucio/lib/rucio/core/transfer.py

Line 1440 in d59c237

default_tombstone_delay = core_config_get('transfers', 'multihop_tombstone_delay', default=DEFAULT_MULTIHOP_TOMBSTONE_DELAY, expiration_time=600)

the default value is passed if no runtime configuration is set. The problem is, all cache values are converted to string in

rucio/lib/rucio/core/config.py

Line 191 in d59c237

write_to_cache(value_key, str(value)) # Also write default to cache

which means that the actual default value ends up being the string '2:00:00', and later on in tombstone_from_delay we get a ValueError which is converted to None

rucio/lib/rucio/core/replica.py

Lines 1424 to 1428 in d59c237

	if not isinstance(tombstone_delay, timedelta):
	try:
	tombstone_delay = timedelta(seconds=int(tombstone_delay))
	except ValueError:
	return None

hence no tombstone unless transfers.multihop_tombstone_delay is explicitly set (or if the RSE has a multihop_tombstone_delay attribute)

Modification

Set

DEFAULT_MULTIHOP_TOMBSTONE_DELAY = int(datetime.timedelta(hours=2).total_seconds())

[rcarpa]

thank you for the investigation. I think the real problem is that we write the default to cache. Also, setting to none on conversion error isn't the best idea. @bari12 , WDYT?

[bari12]

Writing the default to cache is good in principle, this way we still minimise having to do database interaction in case the default is used. In principle dogpile supports writing the actual value (with type) to the cache and also retrieve it like that. I am not exactly sure why we cast it to string here. But I am a little bit concerned of changing this now and what side-effects this could result in.
For now I think @nsmith- suggestion is good. In a second step we should perhaps evaluate writing the right types to the cache.

[nsmith-]

A workaround is to set rucio-admin config set --section transfers --option multihop_tombstone_delay --value 7200