Allow specification of a default tombstone when registering replicas #4491
Comments
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 20, 2021
Make it easy to use for simple test case initialization:
creating replicas/dataset/containers.
Rework the cleanup functions:
- split TemporaryFileFactory cleanup reduce chance of deadlocks in
tests and have the same behavior as in TemporaryRSEFactory (already
refactored earlier)
- only cleanup replicas which do actually exist. Query the database
instead of using list_replicas, because list_replicas also returns
archive constitutents. Trying to remove constituent replicas with
delete_replicas will fail.
- ignore rule locks in teardown
- also cleanup bad replicas
Add features:
- allow generating random dids. To avoid having to copy-paste
"did = {'scope': mock_scope, 'name': str(generate_uuid())}" all-over
the tests
- use the same uuid as part of all dids generating in a particular test
- allow to easily create containers and datasets
- allow to make rse without protocol
- allow to make srm rses
- pass vo argument on client creation
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 20, 2021
Make it easy to use for simple test case initialization:
creating replicas/dataset/containers.
Rework the cleanup functions:
- split TemporaryFileFactory cleanup reduce chance of deadlocks in
tests and have the same behavior as in TemporaryRSEFactory (already
refactored earlier)
- only cleanup replicas which do actually exist. Query the database
instead of using list_replicas, because list_replicas also returns
archive constitutents. Trying to remove constituent replicas with
delete_replicas will fail.
- ignore rule locks in teardown
- also cleanup bad replicas
Add features:
- allow generating random dids. To avoid having to copy-paste
"did = {'scope': mock_scope, 'name': str(generate_uuid())}" all-over
the tests
- use the same uuid as part of all dids generating in a particular test
- allow to easily create containers and datasets
- allow to make rse without protocol
- allow to make srm rses
- pass vo argument on client creation
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 21, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
bari12
pushed a commit
that referenced
this issue
Apr 22, 2021
* Testing: rework temp factories: #4385, #4491, #1431 Make it easy to use for simple test case initialization: creating replicas/dataset/containers. Rework the cleanup functions: - split TemporaryFileFactory cleanup reduce chance of deadlocks in tests and have the same behavior as in TemporaryRSEFactory (already refactored earlier) - only cleanup replicas which do actually exist. Query the database instead of using list_replicas, because list_replicas also returns archive constitutents. Trying to remove constituent replicas with delete_replicas will fail. - ignore rule locks in teardown - also cleanup bad replicas Add features: - allow generating random dids. To avoid having to copy-paste "did = {'scope': mock_scope, 'name': str(generate_uuid())}" all-over the tests - use the same uuid as part of all dids generating in a particular test - allow to easily create containers and datasets - allow to make rse without protocol - allow to make srm rses - pass vo argument on client creation * Testing: rename file_factory to did_factory It has grown to also create containers/datasets for easier test initialization. So the new name is better suited
bari12
pushed a commit
that referenced
this issue
Apr 22, 2021
* Testing: rework temp factories: #4385, #4491, #1431 Make it easy to use for simple test case initialization: creating replicas/dataset/containers. Rework the cleanup functions: - split TemporaryFileFactory cleanup reduce chance of deadlocks in tests and have the same behavior as in TemporaryRSEFactory (already refactored earlier) - only cleanup replicas which do actually exist. Query the database instead of using list_replicas, because list_replicas also returns archive constitutents. Trying to remove constituent replicas with delete_replicas will fail. - ignore rule locks in teardown - also cleanup bad replicas Add features: - allow generating random dids. To avoid having to copy-paste "did = {'scope': mock_scope, 'name': str(generate_uuid())}" all-over the tests - use the same uuid as part of all dids generating in a particular test - allow to easily create containers and datasets - allow to make rse without protocol - allow to make srm rses - pass vo argument on client creation * Testing: rename file_factory to did_factory It has grown to also create containers/datasets for easier test initialization. So the new name is better suited
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 22, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 26, 2021
- get_sum_count_being_deleted - update_replica_lock_counter - update_replicas_paths - list_unlocked_replicas - mark_unlocked_replicas
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 26, 2021
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 26, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 26, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 26, 2021
- get_sum_count_being_deleted - update_replica_lock_counter - update_replicas_paths - list_unlocked_replicas - mark_unlocked_replicas
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 26, 2021
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 26, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
Tests work correctly without this argument. Is it used by somebody in production? why would it be?
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
Allow to avoid un-necessary sleep in tests. 'break' instead of 'return', to correctly cleanup heartbeats.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 27, 2021
bari12
pushed a commit
that referenced
this issue
Apr 28, 2021
…4582) * Transfers: remove useless mock argument in submitter. #4491 Tests work correctly without this argument. Is it used by somebody in production? why would it be? * Transfers: fixup poller and finisher for tests. #4491 Allow to avoid un-necessary sleep in tests. 'break' instead of 'return', to correctly cleanup heartbeats. * Testing: add conveyor intermediate replica integration tests. #4491 Add an additional XRD container in tests. Make it connected to XRD3, to allow multihop transfers from XRD1. Call submitter, poller, finisher and reaper and verify at each stage that the lifecycle of intermediate replicas is correctly handled. * Testing: add multihop submitter test. #4491
bari12
pushed a commit
that referenced
this issue
Apr 28, 2021
…4582) * Transfers: remove useless mock argument in submitter. #4491 Tests work correctly without this argument. Is it used by somebody in production? why would it be? * Transfers: fixup poller and finisher for tests. #4491 Allow to avoid un-necessary sleep in tests. 'break' instead of 'return', to correctly cleanup heartbeats. * Testing: add conveyor intermediate replica integration tests. #4491 Add an additional XRD container in tests. Make it connected to XRD3, to allow multihop transfers from XRD1. Call submitter, poller, finisher and reaper and verify at each stage that the lifecycle of intermediate replicas is correctly handled. * Testing: add multihop submitter test. #4491
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 28, 2021
- get_sum_count_being_deleted - update_replica_lock_counter - update_replicas_paths - list_unlocked_replicas - mark_unlocked_replicas
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 28, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 29, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 29, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 30, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 30, 2021
- get_sum_count_being_deleted - update_replica_lock_counter - update_replicas_paths - list_unlocked_replicas - mark_unlocked_replicas
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 30, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 30, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
bari12
added a commit
that referenced
this issue
Apr 30, 2021
…licas Core & Internals: delete unused replicas functions. #4491
bari12
pushed a commit
that referenced
this issue
Apr 30, 2021
- get_sum_count_being_deleted - update_replica_lock_counter - update_replicas_paths - list_unlocked_replicas - mark_unlocked_replicas
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
Apr 30, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 6, 2021
…ucio#4188 Always set a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone - The other used to configure the multihop temporary replica tombstone As a tombstone will now be added when the intermediate multihop replica is created, remove the redundant protection which was touching the tombstone in the finisher. Add a submitter test which ensures that multihop sources are created; and that the multihop request are created with a tombstone.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 19, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 20, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
rcarpa
added a commit
to rcarpa/rucio
that referenced
this issue
May 25, 2021
…#4436 and rucio#4188 Allow setting a default tombstone on any replica creation. Introduce two rse attributes which allow to customize the behavior per rse: - One allows to define the normal replica creation tombstone (defaults to 0: "no tombstone") - The other used to configure the multihop temporary replica tombstone (defaults to 2 hours in the future) Use a negative value to set the epoch tombstone. Or 0 to explicitly set a null tombstone. Protect replicas which are used as sources from deletion even if their tombstone is set to epoch. Add an integration test which verifies this behavior. Add a submitter test ensuring that multihop sources are created; and that the temp multihop replicas are created with a tombstone. Remove the redundant behavior which touches the tombstone in the finisher. The 2 remaining protections for intermediate multihop replicas are now: 1) entries in the source table; 2) the default multihop tombstone delay of 2 hours into the future.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Motivation
Currently, replicas are registered without a default tombstone, a rule needs to touch them first for them to be eligible for deletion. This creates issues in certain workflows.
Modification
Allow the registration of replicas with a default tombstone. (Parameter name to be decided)
The text was updated successfully, but these errors were encountered: