support auto-scale for inference job

[leigaoms]

1. Provide RestAPI interface;
2. JobManager deployment could auto-scale; framework launcher does not support yet

[coveralls]

Pull Request Test Coverage Report for Build 2997

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 92.893%

---

Totals
Change from base Build 2995:	0.0%
Covered Lines:	668
Relevant Lines:	709

---

💛 - Coveralls

[xudifsd]

Hope you can add a test case here to test this functionality(scale up/down). You may want to add branch like this to skip the test case if it's controller.

It can be run with parameters ./main.py --rest http://dltshub-int.redmond.corp.microsoft.com:5001 --vc platform --email xxx@microsoft.com --uid 1234 --config ~/dev/Deployment/Azure-EastUS-P40-Dev1/ please replace email and uid and config parameter when you run. With your email, uid get from identity table and config to backuped cluster config.

I want to make sure all test cases not broken whenever new feature added.

[Anbang-Hu]

Please do add a test case as @xudifsd commented.

[xudifsd]

I would prefer block_until_replica_count_is with a timeout parameter in test case. This will wait at least 30s.

[leigaoms]

The problem is if cluster does not have enough resource, the number of pod may never reach to desired_replicas. We can just make sure deployment.replica is correctly set, but the real replica also impacted by cluster status.

[xudifsd]

I think we can relax assumption about it should be a deployment here. We can count pods with label jobId=xxx here to check if it scaled up or down. Since controller doesn't create deployment.

[xudifsd]

Anyway, it's good to have a test case. I will modify this test case when I have time. You can merge this for now.