Optimise API model discovery speed

[RobertLucian]

Closes #1663.

As an example, for #1592, it reduces the discovery time of models from ~300 seconds down to about 16 seconds (s3.search + model dir validation). 9 seconds are for s3.search and about 7 seconds are for validating the models. This is while factoring in the fact that there are ~10k paths with the specified prefix in cortex.yaml#predictor.models.dir.

• The validating functions can be further optimized.
• A profiling tool has been used for this.

---

checklist:

• run make test and make lint
• test manually (i.e. build/push all images, restart operator, and re-deploy APIs)

[RobertLucian]

A mod requested.