Skip to content

Latest commit

 

History

History
318 lines (256 loc) · 28.8 KB

README.md

File metadata and controls

318 lines (256 loc) · 28.8 KB
Raw

Prometheus BOSH Release

This is a BOSH release for Prometheus, Alertmanager, and Grafana. It also includes various Prometheus exporters and Grafana plugins.

The detailed list of included components and their maintenance status is available on VERSIONS.md

Questions? Pop in our Slack channel!

Table of Contents

Usage

Requirements

In order to use this BOSH release you will need:

Although not mandatory, it is recommended to deploy the node exporter addon in order to get system metrics.

Clone the repository

First, clone this repository into your workspace:

git clone https://github.com/bosh-prometheus/prometheus-boshrelease
cd prometheus-boshrelease
export BOSH_ENVIRONMENT=<name>

Then checkout to the release branch you want to use, so manifest files will be in synch with the release version:

git checkout v...

Basic deployment

To deploy a basic prometheus server with alertmanager and grafana (but no exporters) use the following command:

bosh -d prometheus deploy manifests/prometheus.yml \
  --vars-store tmp/deployment-vars.yml

Once deployed, look for the nginx instance IP address:

bosh -d prometheus instances

You can reach each component's web ui at:

  • alertmanager: http://<nginx-ip-address>:9093
  • grafana: http://<nginx-ip-address>:3000
  • prometheus: http://<nginx-ip-address>:9090

Credentials for each components can be located at the tmp/deployment-vars.yml file.

Using BOSH Service Discovery

If you want to use the BOSH Service Discovery in order to dynamically discover your exporters then add the monitor-bosh.yml op file by running the following command (filling the required variables with your own values):

bosh -d prometheus deploy manifests/prometheus.yml \
  --vars-store tmp/deployment-vars.yml \
  -o manifests/operators/monitor-bosh.yml \
  -v bosh_url= \
  -v bosh_username= \
  -v bosh_password= \
  --var-file bosh_ca_cert= \
  -v metrics_environment=

NOTE: metrics_environment is an arbitrary name to identify your environment (test, nyc-prod, ...)

If you have configured your bosh-deployment to use UAA user management (via the uaa.yml ops file) we recommend adding the add-bosh-exporter-uaa-clients.yml op file to your bosh-deployment and then adding the enable-bosh-uaa.yml ops file to the prometheus deployment by running the following command (filling the required variables with your own values):

bosh -d prometheus deploy manifests/prometheus.yml \
  --vars-store tmp/deployment-vars.yml \
  -o manifests/operators/monitor-bosh.yml \
  -o manifests/operators/enable-bosh-uaa.yml \
  -v bosh_url= \
  --var-file bosh_ca_cert= \
  -v metrics_environment=

In case you have configured manually an UAA client_id for the bosh_exporter (different from bosh_exporter), then run the following command instead:

bosh -d prometheus deploy manifests/prometheus.yml \
  --vars-store tmp/deployment-vars.yml \
  -o manifests/operators/monitor-bosh.yml \
  -o manifests/operators/enable-bosh-uaa.yml \
  -o manifests/operators/configure-bosh-exporter-uaa-client-id.yml \
  -v bosh_url= \
  -v uaa_bosh_exporter_client_id= \
  -v uaa_bosh_exporter_client_secret= \
  --var-file bosh_ca_cert= \
  -v metrics_environment=

Monitoring Cloud Foundry

If you want to monitor your Cloud Foundry platform, first update your cf-deployment adding the add-prometheus-uaa-clients.yml op file.

This will add the UAA clients required to gather information from the Cloud Foundry API and Firehose. Then add the monitor-cf.yml ops file by running the following command (filling the required variables with your own values):

bosh -d prometheus deploy manifests/prometheus.yml \
  --vars-store tmp/deployment-vars.yml \
  -o manifests/operators/monitor-bosh.yml \
  -v bosh_url= \
  -v bosh_username= \
  -v bosh_password= \
  --var-file bosh_ca_cert= \
  -v metrics_environment= \
  -o manifests/operators/monitor-cf.yml \
  -v metron_deployment_name= \
  -v system_domain= \
  -v uaa_clients_cf_exporter_secret= \
  -v loggregator_ca_name= \
  -v skip_ssl_verify=

NOTE: metron_deployment_name property should match the deployment property of your metron_agent or loggregator_agent jobs. Use:

NOTE: loggregator_ca_name property should match the full credhub path of loggregator_ca certificate variable, ex: /bosh-mydirector/cf/loggregator_ca.

NOTE: You can switch to legacy implementation of firehose_exporter and legacy cloud foundry dashboards by adding the following ops-files:

  • on prometheus deployment, adapt: 
    ...
-o manifests/operators/monitor-cf.yml \
-o manifests/operators/deprecated/monitor-cf-attic.yml \
-v uaa_clients_firehose_exporter_secret= \
-v traffic_controller_external_port= \
...
  • When using add-prometheus-uaa-clients.yml on cloud foundry deployment, adapt: 
    ...
-o manifests/operators/cf/add-prometheus-uaa-clients.yml
-o manifests/operators/deprecated/cf/add-prometheus-uaa-clients-attic.yml
...

This will switch deployment to firehose_exporter-attic, cloudfoundry_dashboards-attic and cloudfoundry_alerts-attic

Register Cloud Foundry routes

If you want to access alertmanager, grafana, and prometheus web ui's using your Cloud Foundry system domain instead of IP addresses, then you can register those routes inside your Cloud Foundry environment using the enable-cf-route-registrar.yml op file by running the following command (filling the required variables with your own values):

bosh -d prometheus deploy manifests/prometheus.yml \
  --vars-store tmp/deployment-vars.yml \
  ...
  -o manifests/operators/enable-cf-route-registrar.yml \
  -v system_domain= \
  -v cf_deployment_name=

The op file will register the following routes:

  • https://alertmanager.<cf system domain>
  • https://grafana.<cf system domain>
  • https://prometheus.<cf system domain>

Use UAA for Grafana authentication

If you want to allow users registered at your Cloud Foundry environment to access the Grafana dashboards (Viewer mode only), first update your cf-deployment adding the add-grafana-uaa-clients.yml op file. This will add the UAA client required by the Grafana-UAA integration.

Then add the enable-grafana-uaa.yml op file by running the following command (filling the required variables with your own values):

bosh -d prometheus deploy manifests/prometheus.yml \
  --vars-store tmp/deployment-vars.yml \
  ...
  -o manifests/operators/enable-grafana-uaa.yml \
  -v system_domain= \
  -v uaa_clients_grafana_secret= \
  --var-file uaa_ssl.ca= \
  --var-file uaa_ssl.certificate= \
  --var-file uaa_ssl.private_key=

Operations files

Additional operations files are located at the manifests/operators directory. Those files includes a basic configuration, so extra ops files might be needed for additional configuration.

Please review the op files before deploying them to check the requirements, dependencies and necessary variables.

File Description exporter dashboards alerts
alertmanager-group-by-alertname.yml Groups alertmanager alerts by name
alertmanager-hipchat-receiver.yml Configures a HipChat receiver for alertmanager
alertmanager-opsgenie-receiver.yml Configures a OpsGenie receiver for alertmanager
alertmanager-pagerduty-receiver.yml Configures a PagerDuty receiver for alertmanager
alertmanager-pushover-receiver.yml Configures a Pushover receiver for alertmanager
alertmanager-slack-receiver.yml Configures a Slack receiver for alertmanager
alertmanager-victorops-receiver.yml Configures a VictorOps receiver for alertmanager
alertmanager-webhook-receiver.yml Configures a generic webhook receiver for alertmanager
alertmanager-web-external-url.yml Configures the URL under which alertmanager is externally reachable
configure-bosh-exporter-uaa-client-id.yml Configures a custom bosh_exporter UAA client_id for the enable-bosh-uaa.yml op-file
enable-bosh-uaa.yml Configures monitor-bosh.yml to use an UAA client (you must apply the add-bosh-exporter-uaa-clients.yml op file to your bosh-deployment)
enable-cf-route-registrar.yml Registers alertmanager, grafana, and prometheus as Cloud Foundry routes (under your system domain)
enable-grafana-uaa.yml Configures grafana user authentication to use Cloud Foundry UAA (you must apply the add-grafana-uaa-clients.yml op file to your cf-deployment)
enable-grafana-generic-oauth.yml Configures grafana user authentication to use a generic OAuth2 provider
enable-service-discovery.yml Enable service discovery files using BOSH links
enable-proxy-alertmanager.yml Enables http(s) proxy for alertmanager
enable-proxy-blackbox-exporter.yml Enables http(s) proxy for blackbox_exporter
enable-proxy-bosh-exporter.yml Enables http(s) proxy for bosh_exporter
enable-proxy-cf-exporter.yml Enables http(s) proxy for cf_exporter
enable-proxy-firehose-exporter.yml Enables http(s) proxy for firehose_exporter
enable-proxy-grafana.yml Enables http(s) proxy for grafana
enable-proxy-kubernetes.yml Enables http(s) proxy for kube_state_metrics_exporter
enable-proxy-prometheus.yml Enables http(s) proxy for prometheus
enable-proxy-shield-exporter.yml Enables http(s) proxy for shield_exporter
enable-proxy-stackdriver-exporter.yml Enables http(s) proxy for stackdriver_exporter
enable-root-url.yml Enables root_url for grafana
migrate_from_prometheus_1.yml Allows migrating an instance from Prometheus 1.x to Prometheus 2.x
monitor-bosh.yml Enables monitoring BOSH jobs and processes and enables Service Discovery x x x
monitor-cadvisor.yml Enables monitoring cAdvisor x
monitor-cf.yml Enables monitoring Cloud Foundry via the Cloud Foundry and Cloud Foundry Firehose exporters (you must apply the add-prometheus-uaa-clients.yml op file to your cf-deployment) x x x
monitor-collectd.yml Enables monitoring Collectd x
monitor-concourse.yml Enables monitoring Concourse CI >= v3.8.0 (you must apply the enable-prometheus-metrics.yml op file to your concourse-deployment) x x
monitor-concourse-influxdb.yml Enables monitoring Concourse CI < v3.8.0. Requires node exporter on Concourse VMs (probably as a BOSH add-on) and InfluxDB to be deployed independently and configured as a data source in Grafana as well as Concourse configured to send events to InfluxDB x
monitor-consul.yml Enables monitoring Consul x x x
monitor-credhub.yml Enables monitoring Credhub x x
monitor-elasticsearch.yml Enables monitoring Elasticsearch x x x
monitor-graphite.yml Enables monitoring Graphite x
monitor-haproxy.yml Enables monitoring HAProxy x x x
monitor-http-probe.yml Enables monitoring HTTP(s) endpoints via the Blackbox exporter x x x
monitor-influxdb.yml Enables monitoring InfluxDB x
monitor-kubernetes.yml Enables monitoring Kubernetes x x x
monitor-memcached.yml Enables monitoring Memcached x
monitor-mongodb.yml Enables monitoring MongoDB x
monitor-mysql.yml Enables monitoring MySQL x x x
monitor-nats.yml Enables monitoring NATS x
monitor-node.yml Enables monitoring system metrics via the node exporter x
monitor-p-rabbitmq.yml Enables monitoring RabbitMQ for PCF (requires the monitor-cf.yml op file) x x
monitor-p-redis.yml Enables monitoring Redis for PCF (requires the monitor-cf.yml op file) x x
monitor-postgres.yml Enables monitoring PostgreSQL x x x
monitor-pushgateway.yml Deploys a PushGateway x
monitor-rabbitmq.yml Enables monitoring RabbitMQ x x x
monitor-redis.yml Enables monitoring Redis x x x
monitor-shield.yml Enables monitoring Shield x x x
monitor-stackdriver.yml Enables monitoring Stackdriver x
monitor-statsd.yml Enables monitoring Statsd x
monitor-vault.yml Enables monitoring Vault x x
nginx-vm-extension.yml Adds a VM Extension block to the nginx instance, useful to attach a Load Balancer
prometheus-web-external-url.yml Configures the URL under which prometheus is externally reachable
use-sqlite3.yml Use sqlite3 instead of postgres

In addition, some deprecated ops-files allows to switch back to legacy behaviours

File Description exporter dashboards alerts
deprecated/monitor-cf-attic.yml Use legacy implementation of monitor-cf.yml x x x
deprecated/cf/add-prometheus-uaa-clients-attic.yml Adds UAA client in cloud foundry deployment when using monitor-cf-attic.yml
deprecated/enable-cf-loggregator-v2.yml Enables Cloud Foundry Loggregator V2 API calls in the legacy firehose_exporter

Deployment variables and the var-store

Some operators files requires additional information to provide environment-specific or sensitive configuration such as various credentials. To do this in the default configuration, we use the --vars-store. This flag takes the name of a yml file that it will read and write to. Where necessary credential values are not present, it will generate new values based on the type information stored at the different deployment files. Necessary variables that BOSH can't generate need to be supplied as well. See each particular op files you're using for any additional necessary variables.

See also the BOSH CLI documentation for more information about ways to supply such additional variables.

Contributing

Refer to CONTRIBUTING.md.

License

Apache License 2.0, see LICENSE.