Multihop intermediate replica tombstone #4188
Comments
|
Somewhat related, I think the tombstone set on a replica that was made at the end of a transfer in https://github.com/rucio/rucio/blob/master/lib/rucio/daemons/conveyor/finisher.py#L462 should also be epoch. |
|
You could also argue since the replicas was used, for the multihop, it is a valid access. But I get your point that, since the data is migrated to tape, it is just less likely that a workflow will re-access it. |
|
See also #4436 |
|
Proposal: rucio/lib/rucio/core/replica.py Line 1900 in 398df65 list_and_mark_unlocked_replicas. I need to think what is the best solution.
|
|
Yes, as we have seen setting an epoch tombstone here would probably not work correctly, since it will remove the replica way to soon (See #4436) |
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 21, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 22, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 26, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 26, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 26, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 28, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 29, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 29, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 30, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 30, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 30, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 30, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 6, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 20, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 25, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Jun 15, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future. introduce multihop_transfer_delay configuration: To allow setting a default value for all RSEs used in multihops.
|
I'm confused, are you saying there is a race condition in the logic and you can configure how risky you want to be with respect to it? |
|
@nsmith- , the epoch tombstone is now default for intermediate replicas. Missing protections from deletion were added to allow that. We hope that we'll not have race conditions, but the machinery is big and fragile. The configuration parameter is there with 2 purposes: one is to allow keeping intermediate replicas for longer when it is administratively desired (high probability of re-usage of the replica). The second is the one you mentioned. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Motivation
From https://github.com/rucio/rucio/pull/3064/files#r343021930 it looks like, in addition to the transfer source replica protection, an additional protection is placed on files likely to be created by a multihop transfer in setting the tombstone to (now+2h). I think these intermediate replicas should have epoch tombstone so they are cleaned up immediately rather than flushing popular but unlocked replicas from the intermediate disk RSE.
Modification
There may be other uses of
update_replicas_statesbesides multihop that would hit this code, so I am not sure if we want to set the tombstone to epoch always. Perhaps somewhere else we can set this? Or we can add a flag to indicate the replica is being used for multihop?The text was updated successfully, but these errors were encountered: