Fix high memory consumption issues

[dralley]

closes: #7152
https://pulp.plan.io/issues/7152

[dralley]

Before and after with a single repository: https://imgur.com/a/mrqAJ2k

With 8 repositories and ~230k units, it maxes out around 600mb of RAM. That definitely would have run out of memory before in my experience, even with ~12gb allocated to the box.

I'm going to keep looking into this tomorrow because I want to make sure the tool I'm using isn't lying to me, since a lot of what we're doing is using C-based libraries that Python profiling tools can't always necessarily track. Also, it's a lot more difficult to track the memory usage of many different processes than one process, so I made a few hacks to execute the task without forking and disable parallel repo migration, so this isn't a very real-world setup. I'll see what I can do to verify this properly.
I'll also probably look into https://pulp.plan.io/issues/7183

[goosemania]

🎉 🐈 🥇