Skip to content

Commit

Permalink
Docs for Butler 8.3.2
Browse files Browse the repository at this point in the history
  • Loading branch information
mountaindude committed Jun 21, 2022
1 parent ce86fb6 commit 6893708
Show file tree
Hide file tree
Showing 158 changed files with 13,883 additions and 1,341 deletions.
10 changes: 5 additions & 5 deletions content/en/_index.html
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@
You or your users?
Who was it that stopped that important 3-hour reload 5 minutes before it was done?

Get notified when tasks fail or are stopped, with notifications sent to Slack, Microsoft Teams or email.
Get notified when tasks fail or are stopped, with notifications sent to Slack, Microsoft Teams, email, New Relic, Signl4 and other destinations.
Full support for HTML/Markdown formatting and 40+ template fields.

[Read more...](/docs/concepts/alert-emails/)
Expand Down Expand Up @@ -162,10 +162,10 @@

{{< blocks/section color="dark">}}

{{% blocks/feature icon="fa-lightbulb" title="Latest release: 8.1" %}}
📄 View script logs of failed reloads in New Relic.
🚨 Create New Relic events via Butler's REST API.
⚙️ Full set of task/app metadata for failed reloads available in New Relic.
{{% blocks/feature icon="fa-lightbulb" title="Latest release: 8.3" %}}
📈 What's better than getting reload failed alerts in New Relic?
📈📈📈 Getting those alerts (incl script logs) sent to multiple New Relic accounts, of course!.


Available on [GitHub](https://github.com/ptarmiganlabs/butler/releases).
{{% /blocks/feature %}}
Expand Down
37 changes: 37 additions & 0 deletions content/en/docs/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,43 @@ General guidance on how to do this is found [here](/docs/getting-started/upgrade

{{% pageinfo %}}

### What's new in version 8.3.1

*Bug Fixes*

* Add missing API endpoint docs to HTML/YAML/JSON API documents [#502](https://github.com/ptarmiganlabs/butler/issues/502)
* Fix broken macOS build

### What's new in version 8.3.0

NOTE: There is no macOS binary for this version.

*Features*

* Send Butler metrics and failed/aborted reloads as New Relic events and/or logs to zero, one or more New Relic accounts [#489](https://github.com/ptarmiganlabs/butler/issues/489)

*Other*

* Enforce code style across all files [#497](https://github.com/ptarmiganlabs/butler/issues/497)
* Update dependencies

### What's new in version 8.2.0

*Features*

* Add failed/aborted reload task and app tags as metadata for New Relic events and logs [#479](https://github.com/ptarmiganlabs/butler/issues/479)
* Add optional "from" option when sending test email [#486](https://github.com/ptarmiganlabs/butler/issues/486)

*Bug fixes*

* Add better debug logging around which email addresses are used when sending alert emails [#487](https://github.com/ptarmiganlabs/butler/issues/487)
* Back slash in script log breaks Slack and Teams messages [#485](https://github.com/ptarmiganlabs/butler/issues/485)
* Better debug logging when posting data to New Relic

*Other*

* **deps:** Updated dependencies

### What's new in version 8.1.0

*Features*
Expand Down
5 changes: 4 additions & 1 deletion content/en/docs/about/use cases/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,11 +43,14 @@ Butler offers advanced failed reload alerts via Slack, Teams, email and outgoing
Configurable templates means you can customize emails/Teams/Slack messages.

Sometimes you want a bit more structure though.
This is especially true when Sense is used in the enterprise, creating critical metrics and reports.
This is especially true when Sense is used in the enterprise.

Butler integrates with both [Signl4](https://www.signl4.com) and [New Relic](https://newrelic.com).
Both offer incident management features on both the web and via mobile clients.

Information about failed/aborted reloads can be sent to one or more New Relic accounts.
Tags for the reload task and associated app is sent to New Relic as metadata for the event/log entry that's created there.

## Use InfluxDB/Grafana or New Relic to track Butler memory usage

Butler can be configured to log its own memory usage to [InfluxDB](https://www.influxdata.com/products/), from where it can be visualised using [Grafana](https://grafana.com).
Expand Down
11 changes: 11 additions & 0 deletions content/en/docs/concepts/incident mgmt tools/new relic/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,19 @@ The [Getting started section](/docs/getting-started/setup/incident-mgmt-tools/ne

## New Relic screen shots

Sample New Relics dashboards are found below.

Please note that some of the charts (server RAM, CPU, user sessions etc) in these dashboards use data from Butler SOS.
Butler provides data for failed/aborted reloads tables and charts.

{{< imgproc new-relic-dashboard-1.png Resize "x640" >}} New Relic metrics & incident dashboard (light mode) {{< /imgproc >}}

{{< imgproc new-relic-senseops-1.png Resize "x640" >}} New Relic metrics & incident dashboard (dark mode, different time range than previous dashboard above). {{< /imgproc >}}

{{< imgproc new-relic-senseops-2.png Resize "x640" >}} Detailed information about failed reloads, including the last part of the script log, is available from within New Relic dashboards. {{< /imgproc >}}

{{< imgproc new-relic-senseops-3.png Resize "x640" >}} Butler SOS metrics in a dark mode New Relic dashboard. {{< /imgproc >}}

{{< imgproc new-relic-alert-overview-1.png Resize "x640" >}} New Relic incidents overview page (dark mode) {{< /imgproc >}}

{{< imgproc new-relic-alert-conditions-1.png Resize "x640" >}} New Relic alert conditions (dark mode){{< /imgproc >}}
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
64 changes: 37 additions & 27 deletions content/en/docs/examples/monitoring butler/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,47 +9,52 @@ description: >

Butler 4.0 adds several new features, one being the uptime messages that can be optionally posted to the logs. Each message tells how long Butler has been running and how much memory it is using right then.

This information can also, again optionally, be stored to [InfluxDB](https://docs.influxdata.com/influxdb/v1.8/). InfluxDB is a database for time-series data such as measurements.
Once in InfluxDB it's easy to create nice monitoring charts in [Grafana](https://grafana.com/) or similar tools.
This information can also, again optionally, be stored in [InfluxDB](https://docs.influxdata.com/influxdb/v1.8/) or [New Relic](https://newrelic.com).
InfluxDB is a database for time-series data such as measurements, while New Relic is enterprise grade, SaaS observability solution.

Why spend CPU cycles and disk space on this, you may ask?
Once in InfluxDB it's easy to create nice monitoring charts in [Grafana](https://grafana.com/) or similar tools. New Relic has their own built-in dashboard tool (but Grafana can actually load data from New Relic too!).

But hey - why spend CPU cycles and disk space on this?

Well, if you are serious about your Qlik Sense Enterprise environment, you should also be serious about your supporting tools and microservices, Butler included.

Even though Butler over the years has proven to be a very stable piece of software, there is always the risk of new features misbehaving, or new bugs appearing.
It's thus a good idea to monitor for example how much memory (RAM) tools like Butler use over time, and alert if things go the wrong way.
Even though Butler over the years has proven to be a very stable piece of software, there is always the risk of new features misbehaving or new bugs appearing.
It's thus a good idea to monitor for example how much memory (RAM) tools like Butler use over time and alert if things go wrong.

## Enable Butler's uptime monitor

Both he uptime monitor and the logging to InfluxDB must be enabled. Note that there are two settings for this. If your InfluxDB uses authentication you'll need to enable this too in Butler's config file.
Both he uptime monitor and the logging to desired destination (InfluxDB or New Relic) must be enabled. Note that there are two settings for this. If your InfluxDB uses authentication you'll need to enable this too in Butler's config file.

If you use New Relic to monitor your uptime metrics you must define the New Relic API key etc in the Butler config file's `Butler.thirdPartyToolsCredentialsnewRelic` settings.
If you use New Relic to monitor your uptime metrics you must first define the New Relic API credentials in the Butler config file's `Butler.thirdPartyToolsCredentials.newRelic` settings, then configure the uptime monitoring specifics in `Butler.uptimeMonitor.storeNewRelic`.

A snippet from a real-life Butler config file could look like this:
The uptime monitoring part of the config file could looks like this:

```yaml
# Uptime monitor
uptimeMonitor:
enable: true # Should uptime messages be written to the console and log files?
enable: false # Should uptime messages be written to the console and log files?
frequency: every 15 seconds # https://bunkat.github.io/later/parsers.html
logLevel: verbose # Starting at what log level should uptime messages be shown?
storeInInfluxdb:
enable: true
hostIP: 192.168.100.20
hostPort: 8086
storeInInfluxdb:
enable: false # Should Butler memory usage be logged to InfluxDB?
hostIP: <IP or host name> # Where is InfluxDB server located?
hostPort: 8086 # InfluxDB port
auth:
enable: false
# username: user_joe
# password: joesecret
dbName: butler
enable: false # Does InfluxDB require login?
username: user_joe
password: joesecret
dbName: butler # Name of database in InfluxDB to which Butler's data is written
instanceTag: DEV # Tag that can be used to differentiate data from multiple Butler instances
# Default retention policy that should be created in InfluxDB when Butler creates a new database there.
# Default retention policy that should be created in InfluxDB when Butler creates a new database there.
# Any data older than retention policy threshold will be purged from InfluxDB.
retentionPolicy:
name: 10d
duration: 10d
storeNewRelic:
enable: true
enable: false
destinationAccount:
- First NR account
- Second NR account
# There are different URLs depending on whther you have an EU or US region New Relic account.
# The available URLs are listed here: https://docs.newrelic.com/docs/accounts/accounts-billing/account-setup/choose-your-data-center/
# As of this writing the options for the New Relic metrics API are
Expand All @@ -75,9 +80,11 @@ A snippet from a real-life Butler config file could look like this:
value: prod
dynamic:
butlerVersion:
enable: true # Should the Butler version be included in the data sent to New Relic?
enable: true # Should the Butler version be included in the data sent to New Relic
```
## Creating a InfluxDB database
When starting Butler for the first time and InfluxDB is enabled, it will connect to InfluxDB and if needed create a new database called `butler`, together with a retention policy called `10d`:

![alt text](butler-new-influxdb-1.png "Starting Butler for the first time, InfluxDB database created")
Expand All @@ -86,17 +93,17 @@ Note that the only thing needed is a running InfluxDB instance. Butler creates t

## Hey data, are you there?

So far so good. Let's wait a few minutes and then verify that the InfluxDB database has received a few dataspoints. There should be data with 15 second intervals, to be precise.
So far so good. Let's wait a few minutes and then verify that New Relic and/or InfluxDB has received a few dataspoints. There should be data with 15 second intervals, to be precise.

Use the InfluxDB command line client to connect to InfluxDB and do a manual query:
Using the InfluxDB command line client to connect to InfluxDB we can do a manual query:

![alt text](butler-new-influxdb-2.png "Manual query of Butler data in InfluxDB")

Indeed, there are a few data points in InfluxDB. Butler's uptime monitor seems to be working.

## Butler + InfluxDB + Grafana = 🎉📈

Let's wrap up by creating a Grafana chart showing Butler memory use over time.
Grafana has excellent support for InfluxDB, it's therefore a good way to visualise Butler memory use over time.

To use the Grafana dashboard included in the Butler GitHub repository you first need to create a Grafana data source named `Butler ops metrics`, and point it to the InfluxDB database in which Butler stores its data.

Expand All @@ -106,18 +113,21 @@ If everything works you'll see something like this:

![alt text](butler-grafana-1.png "Butler memory metrics in Grafana")

Looks like Butler is using ca 70 MByte here. This is pretty normal, memory usage usually stays well below 100 MByte, even when Butler has been running for days, weeks and months.
Looks like Butler is using ca 70 MByte here. This is pretty normal, memory usage is usually around 100 MByte, even when Butler has been running for days, weeks and months. Exact memory usage will vary depending on which features are enabled.

## Butler + New Relic = 😎🌟

While InfluxDB combined with Grafana is hard to beat when it comes to flexibility and look'n'feel of the dashboards, New Relic is outstanding when it comes to ease of setup.
While InfluxDB combined with Grafana is hard to beat when it comes to flexibility and look'n'feel of the dashboards, New Relic is outstanding when it comes to ease of setup.

New Relic is a SaaS product which means you don't have to host neither databaes nor dashboard tool yourself.
It's all there within New Relic.

What about cost then?
Is New Relic expensive?
What about cost? Is New Relic expensive?

Well, if you have lots of metrics, log files etc New Relic can become quite expensive as they charge you based on how much data you send to New Relic.
But given that Butler will send *very little* data you are unlikely to ever reach the limit of New Relic's free tier.
There is thus a good chance you won't even have to pay for New Relic if you only use it to monitor Butler.

A New Relic chart showing Butler memory usage can look like this:

![alt text](/img/butler-memory-usage-new-relic-1.png "Butler memory usage in Grafana dashboard")
8 changes: 4 additions & 4 deletions content/en/docs/getting started/install/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Given the cross platform nature of Node.js (which is the language Butler is writ

It is therefore difficult to give detailed installation instructions for each possible installation scenario. This site thus tries explain how to get started with Butler in some of the most common scenarios.

Pre-built binaries are available for Windows, macOS and Linux. When using these there is no need to install Node.js, as the Node.js runtime is bundled into the various binaries.
Pre-built binaries are available for Windows and macOS. When using these there is no need to install Node.js, as the Node.js runtime is bundled into the binaries.

Using these binaries is the easiest - and thus recommended - way of using Butler.
...unless you want to use Docker, which is also a great option.
Expand All @@ -33,10 +33,9 @@ Using these binaries is the easiest - and thus recommended - way of using Butler

Sorry - there is no installer for Butler.

The pre-built binaries for Windows, macOS, Linux and Docker simply work as-is when combined with a properly set up configuration file.
The pre-built binaries for Windows, macOS and Docker simply work as-is when combined with a properly set up configuration file.

If you still want to run Butler as Node.js app you will first need to set up Node.js.
This isn't really that difficult, but still another step that has to be carried out.
If you still want to run Butler as Node.js app you will first need to install Node.js.

The instructions on the pages below should provide good guidance, if you still run into troubles you can always reach out via the [GitHub discussion forums](https://github.com/ptarmiganlabs/butler/discussions).

Expand All @@ -45,6 +44,7 @@ The instructions on the pages below should provide good guidance, if you still r
* A Butler executable of some kind. This would be one of the following
* A stand-alone binary for the operating system you plan to use
* A Docker image from which a Butler container can be created.
Images are available on [Docker Hub](https://hub.docker.com/r/ptarmiganlabs/butler/tags).
If you have a Docker or Kubernetes environment this is a great option.
* The Butler source code plus [Node.js](https://nodejs.org/en/) installed.
* A Butler config file adapted to your specific Qlik Sense environment
Expand Down
13 changes: 12 additions & 1 deletion content/en/docs/getting started/operations/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,16 +18,27 @@ How to start Butler varies depending on whether you run it as a standalone app,
<!-- TODO -->
Once Butler is running it's a good idea to also monitor it. Otherwise you stand the risk of not getting notified if Butler for some reason misbehaves.

But logs its own memory usage to the console and log files (if enabled), but can also send these metrics to a InfluxDB database and New Relic.

Butler will log its own memory usage to InfluxDB if

1. The config file's `Butler.uptimeMonitor.enable` and `Butler.uptimeMonitor.storeInInfluxdb.enable` properties are both set to `true`.
2. The remaining InfluxDB properties of the config file are correctly configured.

Assuming everything is correctly set up, you can then create a Grafana dashboard showing Butler's memory use over time.
Similarly, uptime metrics will be sent to New Relic if

1. The config file's `Butler.uptimeMonitor.enable` and `Butler.uptimeMonitor.storeNewRelic.enable` properties are both set to `true`.
2. The remaining New Relic properties of the config file are correctly configured.

Assuming everything is correctly set up, you can then create a Grafana dashboard showing Butler's memory use over time, using data from InfluxDB.
You can also set up alerts in Grafana if so desired, with notifications going to most IM tools and email.

A Grafana dashboard can look like this. Note that one of the available metrics (`external`) is not used in this particular dashboard. It's still logged to InfluxDB though.

![alt text](butler-memory-usage-grafana-1.png "Butler memory usage in Grafana dashboard")

There is a [sample Grafana dashboard](https://github.com/ptarmiganlabs/butler/tree/master/docs/grafana) in Butler's GitHub repo.

A New Relic graph covering the same information (but a different time range!) can look like this:

![alt text](/img/butler-memory-usage-new-relic-1.png "Butler memory usage in Grafana dashboard")
Loading

0 comments on commit 6893708

Please sign in to comment.