TO API POST /cdns/dnsseckeys/generate times out with large numbers of delivery services

[rawlinp]

I'm submitting a ...

• bug report

Traffic Control components affected ...

• Traffic Ops
• Traffic Vault

Current behavior:

With a large number of delivery services (e.g. 266), the POST /cdns/dnsseckeys/generate route times out and returns a 503 because the operation exceeds 1 minute.

Expected behavior:

The operation should succeed in less than 1 minute instead of timing out.

Minimal reproduction of the problem with instructions:

In a CDN with a large number of delivery services (e.g. 266), call this API, and observe that it times out after 1 minute and returns a 503.

Anything else:

This was observed on a VM w/ 8 CPUs at 2596.986 MHz, so the results might be different on baremetal or more powerful VMs. However, we should be able to run TO on low-performance VMs. I would propose that we parallelize this CPU-intensive operation so that we can take advantage of multicore servers.