-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improving Sandman Notifications (no deletion until user warned) #31
Comments
I'm not clear on its behaviour when there's already something in the db for the file. From your description it sounds like it always just replaces the entry with a new not-notified entry? But then you'd get multiple emails about the same thing? |
On testing, 2 unexpected things happened:
|
Corrections to the Above
This means that the diagram should be:
As described above, sandman sends all notifications in the DB marked unnotified, even if the sandman run doesn't run on a directory referred to by one of those notifications. |
Added: - Feature where files will only be deleted if they've been warned at least once - Warning period of sandman run interval for files that are to be deleted, but haven't been warned to anyone - Tests for connecting to postgres DB - Postgres instance within Travis - Tests for SQL string generation Modified: - Config tests using environment variables Removed: - DummyPersistence in testing (replaced with postgres)
Added: - Feature where files will only be deleted if they've been warned at least once - Warning period of sandman run interval for files that are to be deleted, but haven't been warned to anyone - Tests for connecting to postgres DB - Postgres instance within Travis - Tests for SQL string generation Modified: - Config tests using environment variables Removed: - DummyPersistence in testing (replaced with postgres)
Added: - Feature where files will only be deleted if they've been warned at least once - Warning period of sandman run interval for files that are to be deleted, but haven't been warned to anyone - Tests for connecting to postgres DB - Postgres instance within Travis - Tests for SQL string generation Modified: - Config tests using environment variables Removed: - DummyPersistence in testing (replaced with postgres)
Added: - Feature where files will only be deleted if they've been warned at least once - Warning period of sandman run interval for files that are to be deleted, but haven't been warned to anyone - Tests for connecting to postgres DB - Postgres instance within Travis - Tests for SQL string generation Modified: - Config tests using environment variables Removed: - DummyPersistence in testing (replaced with postgres)
Added: - Feature where files will only be deleted if they've been warned at least once - Warning period of sandman run interval for files that are to be deleted, but haven't been warned to anyone - Tests for connecting to postgres DB - Postgres instance within Travis - Tests for SQL string generation Modified: - Config tests using environment variables Removed: - DummyPersistence in testing (replaced with postgres)
Sandman Notifications
This is a summary to the logic behind Sandman's notifications.
Files
For every file:
(note: there is other logic here, i.e. for detecting staged files, corruptions etc., but I'm only summarising what we care about)
If it is, we check if it has passed the limbo threshold. If it has, we delete it. There are no notifications here. And we're done.
If it has:
If it hasn't:
for every passed warning threshold (based on the file age, doesn't touch DB):
Emails
For every stakeholder in the system:
Get anything with Deleted status, NOT notified to that stakeholder
For each warning threshold:
Send the emails.
Mark everything as notified.
Persisting
status
tablewarnings
tableTo mark something as notified, we add the status and the stakeholder to
notifications
table.Summary
It adds the required notifications to the DB marked as not notified. Then, it picks up all the non-notified stuff for that stakeholder, emails them, and then marks it notified.
Notes
Design
From the design doc:
There's been discussions about whether we needed state to be stored - simply, yes, because we need to know if:
As this is relational data (files -> different statuses -> statuses notified for various stakeholders), this DB is a good approach, even if the groups and group owners didn't need to be stored in it.
Timeline
This shows a timeline with two warning thresholds for a file and the deletion point. Above shows what will be sent if Sandman first sees that file at that point in the timeline. It'll send the most recent warning notification that is hasn't sent (stored as whether the stakeholder had been notified in the DB). If Sandman first sees a file after its deletion point, it'll just delete it without warning. (Note: it's just adding it to Limbo, it still gets its full time in Limbo).
This is Bad
We need to ensure at least one notification is sent, so when we come to check if the file can be deleted, we should query the database to see if there is a Warning state that has been notified to at least one stakeholder. If not, it will not move the file to limbo, and will add a
Warning
status with the time being the Sandman running interval (i.e. one day). This means, the user will get a one day warning without us having to modify the times of the file. The next time Sandman runs, it'll see that the file is past its threshold and that a notification has been sent, so it'll happily delete the file.Code Snippets
bin/sandman/sweep.py:323
This can have an extra check in to see if there is a warning marked as
notified: True
in the DB.bin/sandman/sweep.py:359
This is how we add warning notifications to be sent.
bin/sandman/sweep.py:131
Although we can filter out these notifications, currently it only does it by the configured notification periods, so we can't trivially add the notification period
sandman run interval
.This being said, we could (assuming we don't want this interval to be a typical warning), add it as an extra configuration option, and have something like
If the stakeholder doesn't have any notifcations for the run_interval time period, it'll just skip it, exactly the same as if the user doesn't have warnings for one of the normal warning time periods.
bin/sandman/sweep.py:108
This function is where the query is done: note the query is done with
notified: False
The text was updated successfully, but these errors were encountered: