Sync from model config on agent startup

[yuzisun]

What this PR does / why we need it:
Model agent does not sync models from config when scaling to more replicas as there is no CREATE event generated when pods are started with initial mounted config.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #1203

Special notes for your reviewer:

• Fix s3 uri parse issue when there is only bucket
• Force syncing the model config on startup, if we delete and recreate the pod, the agent should sync the model config and download/load the exiting models from model config.

{"level":"info","ts":1605267738.2322607,"logger":"modelAgent","msg":"Initializing model agent with","config-dir":"/mnt/configs","model-dir":"/mnt/models"}
{"level":"info","ts":1605267738.2323687,"logger":"modelAgent","msg":"Initializing s3 client with ","endpoint":"http://minio-service:9000","region":"us-west-2"}
{"level":"info","ts":1605267738.2324173,"logger":"Watcher","msg":"Syncing model directory..","modelDir":"/mnt/models"}
{"level":"info","ts":1605267738.2325592,"logger":"modelWatcher","msg":"adding model","modelName":"cifar"}
{"level":"info","ts":1605267738.232946,"logger":"modelProcessor","msg":"worker is started for","model":"cifar"}
{"level":"info","ts":1605267738.233302,"logger":"modelProcessor","msg":"Downloading model","storageUri":"s3://torchscript/cifar"}
{"level":"info","ts":1605267738.2334309,"logger":"Downloader","msg":"Downloading to model dir","modelUri":"s3://torchscript/cifar","modelDir":"/mnt/models"}
{"level":"info","ts":1605267738.2335713,"logger":"modelAgent","msg":"Download model ","modelName":"cifar","storagUri":"s3://torchscript/cifar","modelDir":"/mnt/models"}
{"level":"info","ts":1605267738.2337146,"logger":"Watcher","msg":"Start to watch model config event"}
{"level":"info","ts":1605267738.2337751,"logger":"Watcher","msg":"Watching","modelConfig":"/mnt/data/models.json"}
{"level":"info","ts":1605267738.626316,"logger":"modelOnComplete","msg":"completion event for model","modelName":"cifar","inFlight":0}

Release note:

[yuzliu]

I thought we intentionally made this a blocking channel? Can @ifilonenko can confirm?

[yuzisun]

I do not see much reason to make it blocking channel, the problem is that if we push the events first and then start the puller the process just hangs there.

[ifilonenko]

/lgtm good catch on testing the scaling

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ifilonenko, yuzisun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [yuzisun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ifilonenko]

/lgtm