[MIG-699] Keep k8s clients around to avoid cost of re-generating them all the time

[djwhatle]

We should measure how long it takes to build a client. My guess is that with the amount of times we re-build clients we may be incurring a significant cost.

[djwhatle]

Wow. I knew this was an issue but not how much of an issue.

I just ran a migration that took 3 minutes, 4 seconds to complete.
Summing up the time spent on constructing a client, mig-controller spent 217,123 milliseconds creating clients.

Total Migration Time: 184 seconds
Client Creation Time: 217 seconds (parallel controllers)

That's 3.6 minutes of controller time creating clients (this is possible because there are multiple controllers) but wow. Keeping the clients in memory instead of recreating all over the place looks like it could have a huge impact on how quickly things could run.

[djwhatle]

This is pretty awesome. Running from my local machine: using a client we already created, migration time was decreased from 184 seconds to 70 seconds!

This means the migration only took 38% as long as originally.
Only 2 seconds were spent building clients instead of 217 seconds.

[djwhatle]

keepclient.txt

[djwhatle]

In-cluster results:

Not nearly as big of a difference, but definitely still a healthy improvement (10 seconds). It seems that the impact of this change may be related to latency. Running off my laptop it had a much bigger improvement. I suspect we would also see large improvement running across data centers, which is one of the performance areas for improvement we spotted.

Keep clients in memory (77 seconds)

Recreate clients (87 seconds)

[djwhatle]

Will be resolved by #1037