DNN-39811 Purge schedule history much faster and in batches

[eugene-sea]

Fixes #3796

Summary

Subquery of DELETE is refactored in the following way: we get schedules with too many scheduling history items and a number of them to delete, then we get that number of first history items to delete for each scheduling ordered by StartDate. This resembles original logic but has a much more efficient execution plan. Note, we delete all purgeable scheduling history items instead of the top 5000. Deletes are performed in batches as the number of items to delete may be big but we want to reduce lock contention.

On production DB clone new version of SP is more than 300x faster.

[sleupold]

Using temp tables and table expressions is usually not very fast.
In my TurboScripts, I am using the following alternative;
CREATE PROCEDURE {databaseOwner}[{objectQualifier}PurgeScheduleHistory] AS BEGIN DELETE FROM {databaseOwner}[{objectQualifier}ScheduleHistory] WHERE ScheduleHistoryID IN (SELECT Top(50000) H.ScheduleHistoryID FROM (SELECT ScheduleHistoryID, ScheduleID, Row_Number() OVER (PARTITION BY ScheduleID ORDER BY ScheduleHistoryID) RN FROM {databaseOwner}[{objectQualifier}ScheduleHistory]) H JOIN {databaseOwner}[{objectQualifier}Schedule] S ON S.ScheduleID = H.ScheduleID WHERE RN > S.RetainHistoryNum AND S.RetainHistoryNum != -1 ORDER BY ScheduleHistoryID); END -- PROCEDURE GO
maybe you may compare performance to your solution

[eugene-sea]

@sleupold, below you could find results of the comparison. I have changed my query to select only top 50000 and delete in one operation.

Note, PR version of the SP deletes all obsolete items (not just top 50000) and deletes in batches to decrease lock contention (this increases SP time but it is not a problem in that case).

Your version is OK, but it does not delete all obsolete items.

[sleupold]

@eugene-sea,
so both central statements have about the same performance (in SQL Server, there is always a "warmup" and you shouldn't measure before a statement ran at least 3 times).
What is your need to purge all records? Usually, a need for purging more than 1000 records should only occur after the purge task had been disabled and needs to catch up. In this case, the host should set an appropriate frequency for this task.
Using your solution would be better for this specific situation, but puts unnecessary overhead to the server for all common executions,

[eugene-sea]

@sleupold, yes, I have run each statement more than 5 times before measurement.

On that production instance, history items are created with speed more than 5000 per hour. That is the reason obsolete items accumulate. Please see #3796.

You could create PR with your version if you wish but the current version of SP in the platform is unacceptable and should be updated.

With low number of items — deleting all will have no overhead and will complete in a fraction of second.

[mitchelsellers]

Just an update on this item, I'm doing some testing with large installations that I have to validate performance

[sleupold]

@mitchelsellers
You might want to test my alternative implementation (included in comment above) as well, and I'd be happy to submit a PR

[sleupold]

I'd add a host setting for batch size with default of 5000, this should be small enough for DNN on slow machines and might be increased, if necessary for bigger installations on powerful servers