Add deployment status feature and cmdlet for deploying Yams applications

[nehmebilal]

Hi guys,

I finally found some time to work on the deployment status feature for YAMS. This feature makes it possible to get the status of the YAMS cluster for visualization and deployment monitoring purposes.

I originally thought about making the deployment status available in a DeploymentStatus.json blob but I quickly realized that race conditions between nodes trying to update this blob make it a bottleneck. Instead, I went with a blob per node. The deployment status blobs are published at

applications/status/clusters/clusterName/instances/instanceId

where instanceId is a text blob containing the deployment status of a particular node. The deployment status of a given node looks as follows:

{
  "Applications": [
    {
      "Id": "FooService",
      "Version": "1.0.1",
      "ClusterId": "FrontEnd",
      "InstanceId": "8d3f7411-ef85-485a-a119-33ffce5c6825",
      "UtcTimeStamp": "2017-10-13T02:48:17.0168224Z"
    }
  ]
}

This blob is published every time YAMS looks for updates (default is 5 seconds), even if there are no changes in the cluster. Basically the status of the application pool on the node will be flushed to the blob every time YAMS does a periodic check for application updates. In fact, If the deployment status file for a given node is not updated for a longer than expected period of time, it's an indication that the YAMS host is down or is having trouble reaching storage (basically the timestamp provided can be used to monitor the node health).

Consumers of the deployment status need to know the instance id in order to consume the status of a given node directly (this is easy to get for cloud services). If not, the BlobStorageDeploymentRepository exposes an Api to get the cluster deployment status; which lists all the blobs in the blob directory corresponding to the cluster and finds out the instances ids based on the blobs names (it also uses a TTL to ignore old leftover blobs). Unfortunately, blob storage does not have a TTL feature and YAMS will not do the cleanup for you at this point (which is only relevant when you redeploy the cluster or scale it down).

In addition to the deployment status, I added a PowerShell cmdlet that makes it easy to deploy YAMS with VSTS. Below is an example script that you can use in VSTS to deploy a service that uses two clusters, one is a front end (Web Api) and the other is an Orleans cluster.

Param(
  [string]$ConnectionString,
  [string]$Version,
  [string]$WorkingDir
)

Import-Module $WorkingDir\YamsPowershell\content\Etg.Yams.Powershell.dll

Install-Applications -ConnectionString $ConnectionString -AppsIds "FooService","FooService.Silo" -ClustersIds "FrontEnd","Orleans" -BinariesPath "$WorkingDir\drop\FooService\","$WorkingDir\drop\FooService.Silo" -Versions $Version,$Version -WaitForDeploymentsToComplete $true

This cmdlet takes advantage of the deployment status feature to allow the caller to optionally wait for the deployment to complete (i.e. all apps have been started). This is very useful to validate that the deployment went well and maybe follow it with some deployment verification tests that perform health checks and warm up the service.

I didn't get the chance yet to add extensive unit tests but the main features have been tested with end to end tests. I also updated the Yams Overview and added a tutorial of how to deploy YAMS with VSTS, which takes advantage of the deployment status feature and the powershell cmdlets.

Please share your thoughts and suggestions!

Thanks,
Nehme

Resolves #11 and partially #44