feat: add linux CSE tracing and metrics

[devigned]

Reason for Change:

It would help development if we were able to have more insight into the behavior of aks-engine.

This PR introduces a new command line tool for sending telemetry to Application Insights, apmz (docs in progress). apmz outputs each trace and metric to an intermediate file, then flushes them at the end. This behavior is only activated when the feature is turned on in the API model. When the feature is not active, there are still stub no-op bash functions available.

To query this data, you can run the following kql

customMetrics
| where customDimensions.os_type == "linux" and customDimensions.correlation_id == "41b61b03-ae2d-495d-a0f7-869afb024ed3"
| order by timestamp asc 

Issue Fixed:

Fixes #2248

Requirements:

• uses conventional commit messages
• includes documentation
• adds unit tests
• tested upgrade from previous version

Notes:

[marosset]

@devigned i opened a new PR with only the apimodel changes for enabling telemetry. Let's try and get that merged and we can work on enabling telemetry for linux and windows CSEs independently
#2417

[codecov]

Codecov Report

Merging #2415 into master will increase coverage by 0.06%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2415      +/-   ##
==========================================
+ Coverage   72.15%   72.22%   +0.06%     
==========================================
  Files         130      130              
  Lines       24486    24504      +18     
==========================================
+ Hits        17668    17698      +30     
+ Misses       5797     5785      -12     
  Partials     1021     1021

[marosset]

this PR is much cleaner now :)

[devigned]

/hold cancel

[devigned]

/azp run pr-e2e

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[devigned]

This PR is only waiting on getting ampz v0.4.0 into the prod storage bucket. Right now, it points to the staging bucket. Once I update it to point at the prod bucket, I'll remove the WIP tag.

@jchauncey, let me know if you need any additional info to approve the prod deploy jobs.

[devigned]

@marosset, please give this another look when you have a sec.

[marosset]

/lgtm

[acs-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: devigned, marosset

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [devigned,marosset]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[jackfrancis]

@palma21 FYI, AKS Engine CSE is now instrumented for App Insights telemetry. The time_metric func above is a wrapper around all CSE step executions, which is disabled by default, and is a no-op.

Just FYI.