Improve cleanup scripts

[haydenseitz]

update lock_and_purge shared function to do dbDelete in blocks of 1000 rows, similar to how the cleanup for syslog currently happens

DO NOT DELETE THE UNDERLYING TEXT

Please note

Please read this information carefully. You can run ./lnms dev:check to check your code before submitting.

• Have you followed our code guidelines?
• If my Pull Request does some changes/fixes/enhancements in the WebUI, I have inserted a screenshot of it.

Testers

If you would like to test this pull request then please run: ./scripts/github-apply <pr_id>, i.e ./scripts/github-apply 5926
After you are done testing, you can remove the changes with ./scripts/github-remove. If there are schema changes, you can ask on discord how to revert.

[librenms-bot]

This pull request has been mentioned on LibreNMS Community. There might be relevant details there:

https://community.librenms.org/t/polling-stops-every-day-during-daily-sh-run/15805/6

[haydenseitz]

updated lock_and_purge function. not sure how to implement with lock_and_purge_query, as if uses dbQuery instead of dbDelete.

Are there integration tests for the daily.sh script?

[murrant]

FYI $count is always numeric, you said it must be an int in the parameters.

[haydenseitz]

good catch, removed the check

[haydenseitz]

after running this a few days in production, I see the loop creates a sort of race condition on certain tables that are updated frequently (device_perf). Any of the SQL statements that use NOW() in the WHERE to delete old rows can potentially never return 0 rows, as there are new entries every time it runs.

I'm thinking I can update each of the SQL statements passed to the lock_and_purge function to either:

1. get the current time in PHP and use that in each the statement, then it's always looping over the same timestamp until all rows are deleted
2. update each statement to use CURDATE(), which will only delete records up to midnight of X days before, instead of actually X days before

I'm leaning toward option 2 - suggestions?

[murrant]

2 seems like the better choice, a fixed time instead of a rolling window.

[murrant]

Closing due to inactivity