Use raw sql to execute deletes

[Red-HAP]

Jira Ticket

COST-1490

Description

Use raw sql vs ORM to do deletes. This change takes the ORM query and uses django's own SQLDeleteCompiler to get the text of the SQL statement plus the parameters. This is then processed directly via connection and cursor objects without using django ORM to attempt to verify and handle the foreign keys.

Provider().delete() is now tuned up as well to call a func named cascade_delete that will walk ORM relations and execute SQL to delete related records bottom-up.

Testing

The resulting delete action should be no different from usual. But the memory usage should not expand to OOM state during delete operations for expired data or anything else that a db_cleaner would be called for.

If you run make docker-reinitdb-with-sources then run a django shell, you can watch the cascade delete work.
How-To-Do-It Lessons:

1. Navigate to top-of-repo
2. make docker-reinitdb-with-sources
3. make load-test-customer-data
4. You can check the relations from Provider to the line-item tables, if you want.
5. cd koku
6. export KOKU_LOG_LEVEL=DEBUG
7. python ./manage.py shell
8. from api.provider.models import Provider
9. p = Provider.objects.first() (or p = Provider.objects.get(pk=<UUID>) if you have identified a provider to use.)
10. p.delete()
11. You should see a flurry of log messages with Level and Cascade in them. With logging set to debug, you'll also see the formatted SQL statements that are being executed.
12. ctrl+d to exit the shell
13. ``unset KOKU_LOG_LEVEL` to clear debug logging

[codecov]

Codecov Report

Merging #2953 (adff7bf) into master (d8dc13c) will increase coverage by 0.0%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##           master   #2953    +/-   ##
=======================================
  Coverage    95.0%   95.1%            
=======================================
  Files         297     298     +1     
  Lines       23647   23759   +112     
  Branches     2632    2632            
=======================================
+ Hits        22475   22585   +110     
- Misses        703     704     +1     
- Partials      469     470     +1     

[dccurtis]

@Red-HAP Can this be extended to cover the flow for removing a provider here:

koku/koku/api/provider/provider_manager.py

Line 208 in 13b91d6

self.model.delete()

This is what is called when a source is destroyed and is the path in which we are suspecting the Django ORM being inefficient at managing the foreign key relationships while deleting.

[dccurtis]

Awesome work HAP!