Skip to content

Latest commit

 

History

History
155 lines (118 loc) · 9.05 KB

GitIntegration.md

File metadata and controls

155 lines (118 loc) · 9.05 KB
Raw

Using wavectl with git

One of the advanced features of wavectl is its git integration. It can be enabled using the --inGit,-g parameter to pull, push and create commands. wavectl gets several benefits via git integration that are not immediately obvious. Some effective use cases are as follows:

Build a git history of alerts, dashboards with pull

Wavefront already keeps the change history of alerts and dashboards. Those can be accessed via the browser gui. The powerful Wavefront v2 api also has endpoints to access the histories of alerts and dashboards. With git integration in wavectl we do not add a brand new feature on top of those. Instead, we attempt to make the retrieval of the histories easier via the command line mostly using existing mechanisms in git.

Passing --inGit to pull command initalizes a git repository in the pulled directory, if the directory gets created with the pull command.

  $ wavectl pull --inGit /tmp/GitIntegrationPull/alerts alert

A git repo is created in the newly created /tmp/GitIntegrationPull/alerts directory.

  $ git -C /tmp/GitIntegrationPull/alerts status
  On branch master
  nothing to commit, working tree clean

Each pull command creates a new commit with the changes pulled from the Wavefront server at that time.

  $ git -C /tmp/GitIntegrationPull/alerts log --oneline
  7c5147c Added files due to pull <resource> cmd:/Users/hbaba/box/src/skynet/wavectl/doc/bin/wavectl pull --inGit /tmp/GitIntegrationPull/alerts alert
  385967c Initial commit with the README.md file

If you execute a long running daemon executing periodic pulls from Wavefront, an extensive git history can be built. The git history will correspond to users' edits to alerts and dashboards.

 while true; do
    wavectl pull <someDir> alert
    wavectl pull <someDir> dashboard
    sleep 300
 done

Then, later on, if someone is interested understanding the changes to some alerts or dashboards, the investigation can be done with git show and log commands, which are widely known by programmers already.

For example:

When was an a particular alert created ?

  $ git -C /tmp/GitIntegrationPull/alerts log $(ls /tmp/GitIntegrationPull/alerts | sort | head -n 1)
  commit 7c5147c408a0117b66ef74a2ac8edf44a69e9685
  Author: Hakan Baba <you@example.com>
  Date:   Fri Feb 8 21:37:16 2019 -0800

      Added files due to pull <resource> cmd:/Users/hbaba/box/src/skynet/wavectl/doc/bin/wavectl pull --inGit /tmp/GitIntegrationPull/alerts alert

When were each alert snoozed ?

  $ git -C /tmp/GitIntegrationPull/alerts log -S snoozed
  commit 7c5147c408a0117b66ef74a2ac8edf44a69e9685
  Author: Hakan Baba <you@example.com>
  Date:   Fri Feb 8 21:37:16 2019 -0800

      Added files due to pull <resource> cmd:/Users/hbaba/box/src/skynet/wavectl/doc/bin/wavectl pull --inGit /tmp/GitIntegrationPull/alerts alert

NOTE: The updater id and the actual update time for each alert and dashboard can be retrieved using the history endpoing in Wavefront API. However, in the current wavectl implementation, the git commit messages do not contain either of them. In future wavectl releases we plan to improve the git integration of pull command to include the updater id and the update time.

Using git-diff to make safe local modifications to your alerts, dashboards with push

In the repetitive editing doc we have demonstrated an example using the sed command to search and replace strings on local files. After the modifications, the alert, dashboard json files were written to Wavefront.

It is always good practice to inspect the changes to alerts and dashboards before writing them back to Wavefront with the push command. The git integration for the push command can provide a diff of local modifications for the user to verify. Since the git integration will work on a local git repo, git-diff can be used for the diff generation.

Using the same example from the repetitive editing doc, we first pull the alerts matching a regular expression. But this time we use the git integration command line parameter --inGit,-g

  $ wavectl pull --inGit /tmp/GitIntegrationPush/alerts alert --match "proc\."

Then the local modifications are executed:

  $ find /tmp/GitIntegrationPush -type f | xargs sed -i -e 's/proc\./host.proc./g'

See the modifications to alerts that are going to be pushed to Wavefront:

  $ git -C /tmp/GitIntegrationPush/alerts diff HEAD
  diff --git a/1530723441304.alert b/1530723441304.alert
  index a1bb891..ec02df7 100644
  --- a/1530723441304.alert
  +++ b/1530723441304.alert
  @@ -1,7 +1,7 @@
   {
       "additionalInformation": "This alert tracks the used network bandwidth percentage for all the compute-* (compute-master and compute-node) machines. If the cpu utilization exceeds 80%, this alert fires.",
  -    "condition": "ts(proc.net.percent,server_type=\"compute-*\" and env=\"live\") > 80",
  -    "displayExpression": "ts(proc.net.percent,server_type=\"compute-*\" and env=\"live\")",
  +    "condition": "ts(host.proc.net.percent,server_type=\"compute-*\" and env=\"live\") > 80",
  +    "displayExpression": "ts(host.proc.net.percent,server_type=\"compute-*\" and env=\"live\")",
       "id": "1530723441304",
       "minutes": 2,
       "name": "Kubernetes - Node Network Utilization - HIGH (Prod)",
  diff --git a/1530723441442.alert b/1530723441442.alert
  index ad9e8ef..bb0bf57 100644
  --- a/1530723441442.alert
  +++ b/1530723441442.alert
  @@ -1,7 +1,7 @@
   {
       "additionalInformation": "This alert tracks the used cpu percentage for all the compute-* (compute-master and compute-node) machines. If the cpu utilization exceeds 80%, this alert fires.",
  -    "condition": "ts(proc.stat.cpu.percentage_used,server_type=\"compute-*\" and env=\"live\") > 80",
  -    "displayExpression": "ts(proc.stat.cpu.percentage_used,server_type=\"compute-*\" and env=\"dev\")",
  +    "condition": "ts(host.proc.stat.cpu.percentage_used,server_type=\"compute-*\" and env=\"live\") > 80",
  +    "displayExpression": "ts(host.proc.stat.cpu.percentage_used,server_type=\"compute-*\" and env=\"dev\")",
       "id": "1530723441442",
       "minutes": 2,
       "name": "Kubernetes - Node Cpu Utilization - HIGH (Prod)",
  diff --git a/1530723441589.alert b/1530723441589.alert
  index e7a0d7f..830d19f 100644
  --- a/1530723441589.alert
  +++ b/1530723441589.alert
  ...

Submit your changes to the local repo:

  $ git -C /tmp/GitIntegrationPush/alerts commit -a -m "proc. is replaced with host.proc."
  [master f33e513] proc. is replaced with host.proc.
   4 files changed, 21 insertions(+), 21 deletions(-)
   rewrite 1530723443146.alert (67%)

NOTE: If you are using git integration, wavectl will not let you push unless you have committed your changes to the repo. This behavior is like a safeguard to ensure that the user is fully aware of what he is writing to Wavefont via wavectl push. Asking the user to commit his local changes, serves to ensure that she has inspected the diff and is OK with the modifications.

Lastly, push your local modifications to Wavefront.

  $ wavectl push --inGit /tmp/GitIntegrationPush/alerts alert
  Replaced alert(s):
  ID               NAME                                                       STATUS    SEVERITY    
  1530723441304    Kubernetes - Node Network Utilization - HIGH (Prod)            WARN    
  1530723441442    Kubernetes - Node Cpu Utilization - HIGH (Prod)                WARN    
  1530723441589    Kubernetes - Node Memory Swap Utilization - HIGH (Prod)        WARN    
  1530723443146    Collections Dev High CPU                                       WARN

NOTE: Even with all the safeguards, if you have made a mistake and pushed to Wavefront, you can roll back the git repo to the previous commit. A command like git checkout HEAD^ will remove your most recent changes from local files. After that, you can re-execute the push command. That will update all alerts one more time and will set them to the previous state.