Delete all pruned persisted info records #39219
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We prune persisted info records if they meet any of the following
criteria:
```clojure
(or (contains? (persisted-info/prunable-states) current-state)
(:archived card-info)
(not (:dataset card-info)))
```
But we only deleted the record when:
```clojure
(when (= "deletable" current-state)
(t2/delete! PersistedInfo :id (:id persisted-info)))
```
So any records that were in a "creating" state (persist a model, but
before it first gets persisted, make it not a model, or archive the
underlying model), we constantly pruned them but never removed the
persisted info record.
Leading to task results like:
```javascript
{"success": 21, "error": 0, "skipped": 0}
```
Because 21 things were queued up for pruning, were attempted to be
pruned, but the persisted info record never removed.
dpsutton
added
the
backport
|
snoe
approved these changes
Feb 27, 2024
|
@dpsutton Did you forget to add a milestone to the issue for this PR? When and where should I add a milestone? |
github-actions bot
pushed a commit
that referenced
this pull request
Feb 28, 2024
We prune persisted info records if they meet any of the following
criteria:
```clojure
(or (contains? (persisted-info/prunable-states) current-state)
(:archived card-info)
(not (:dataset card-info)))
```
But we only deleted the record when:
```clojure
(when (= "deletable" current-state)
(t2/delete! PersistedInfo :id (:id persisted-info)))
```
So any records that were in a "creating" state (persist a model, but
before it first gets persisted, make it not a model, or archive the
underlying model), we constantly pruned them but never removed the
persisted info record.
Leading to task results like:
```javascript
{"success": 21, "error": 0, "skipped": 0}
```
Because 21 things were queued up for pruning, were attempted to be
pruned, but the persisted info record never removed.
dpsutton
added a commit
that referenced
this pull request
Feb 28, 2024
If they are off, they are "eligible" models for persistence that we do not want to persist. And to remember that use choice to not persist them we need to keep the record around. This was handled previously, but #39219 was a bit too blunt. We had a bunch of records in a "creating" state that we weren't removing, so i deleted them regardless of state. But we want to make sure that the "off" ones are not deleted. On stats, a bunch of ones that were set to off are now slated to be persisted after the prune job deleted the persisted_info records.
dpsutton
added a commit
that referenced
this pull request
Feb 29, 2024
We prune persisted info records if they meet any of the following
criteria:
```clojure
(or (contains? (persisted-info/prunable-states) current-state)
(:archived card-info)
(not (:dataset card-info)))
```
But we only deleted the record when:
```clojure
(when (= "deletable" current-state)
(t2/delete! PersistedInfo :id (:id persisted-info)))
```
So any records that were in a "creating" state (persist a model, but
before it first gets persisted, make it not a model, or archive the
underlying model), we constantly pruned them but never removed the
persisted info record.
Leading to task results like:
```javascript
{"success": 21, "error": 0, "skipped": 0}
```
Because 21 things were queued up for pruning, were attempted to be
pruned, but the persisted info record never removed.
Co-authored-by: dpsutton <dan@dpsutton.com>
dpsutton
added a commit
that referenced
this pull request
Feb 29, 2024
* Don't delete "off" persisted_info records If they are off, they are "eligible" models for persistence that we do not want to persist. And to remember that use choice to not persist them we need to keep the record around. This was handled previously, but #39219 was a bit too blunt. We had a bunch of records in a "creating" state that we weren't removing, so i deleted them regardless of state. But we want to make sure that the "off" ones are not deleted. On stats, a bunch of ones that were set to off are now slated to be persisted after the prune job deleted the persisted_info records. * unused local. whoops
dpsutton
added a commit
that referenced
this pull request
Feb 29, 2024
* Don't delete "off" persisted_info records If they are off, they are "eligible" models for persistence that we do not want to persist. And to remember that use choice to not persist them we need to keep the record around. This was handled previously, but #39219 was a bit too blunt. We had a bunch of records in a "creating" state that we weren't removing, so i deleted them regardless of state. But we want to make sure that the "off" ones are not deleted. On stats, a bunch of ones that were set to off are now slated to be persisted after the prune job deleted the persisted_info records. * unused local. whoops
perivamsi
pushed a commit
that referenced
this pull request
Feb 29, 2024
* Don't delete "off" persisted_info records If they are off, they are "eligible" models for persistence that we do not want to persist. And to remember that use choice to not persist them we need to keep the record around. This was handled previously, but #39219 was a bit too blunt. We had a bunch of records in a "creating" state that we weren't removing, so i deleted them regardless of state. But we want to make sure that the "off" ones are not deleted. On stats, a bunch of ones that were set to off are now slated to be persisted after the prune job deleted the persisted_info records. * unused local. whoops Co-authored-by: dpsutton <dan@dpsutton.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We prune persisted info records if they meet any of the following criteria:
But we only deleted the record when:
So any records that were in a "creating" state (persist a model, but before it first gets persisted, make it not a model, or archive the underlying model), we constantly pruned them but never removed the persisted info record.
Leading to task results like:
Because 21 things were queued up for pruning, were attempted to be pruned, but the persisted info record never removed. And so we "remove" 21 persisted_info records every hour. But we actually aren't doing anything.
If the test looks a little weird it is to get output like: