Monitoring Indicator Protocol

This is an observability as code project which allows developers to define and expose performance, scaling, and service level indicators for monitoring, alerting, and documentation. The indicator definition ideally lives in the same repository as the code and is automatically registered when the code is deployed.

There are 3 main uses cases for this project: Generating documentation, validating against an actual deployment's data, and keeping a registry of indicators for use in monitoring tools such as prometheus alert manager and grafana.

This repository contains the OSS distributions of Indicator Protocol components:

• BOSH
• Binaries
• Kubernetes

See the wiki for more detailed information and documentation.

Developing Locally

This project currently requires only the Go programming language to develop. Install go using brew install go, then proceed to the next steps. If you would prefer to use a docker image instead, skip down to the developing locally with docker section

Goland Setup:

1. Preferences -> Go -> Build Tags & Vendoring -> Custom tags: -mod=vendor
2. Preferences -> Go -> Go Modules (vgo) -> Enable Go Modules (vgo) integration

• If the vendored import statements are still red: option + enter -> Sync packages

Running tests

Use the provided script to run tests: ./scripts/test.sh. By default, this runs the local tests (unit and integration tests that can be run without external dependencies). Run with --help to see the full list of test suites you can run.

Starting a registry and registry agent

1. A script is provided to start both components: ./scripts/run_registry_and_agent.sh

2. To verify that this worked, open another terminal window, and use ./scripts/curl_indicators.sh (or ./scripts/curl_indicators.sh | jq if you have access to jq)

   • Your result should look something like this:

   [
     {
       "apiVersion": "indicatorprotocol.io/v1",
       "product": {
         "name": "my-component",
         "version": "1.2.3"
       },
       "metadata": {
         "deployment": "my-service-deployment",
         "source_id": "my-metric-source"
       },
       "indicators": [
         {
           "name": "only_in_example_yml",
           "promql": "test_query"
         },
         {
           "name": "doc_performance_indicator",
           "promql": "avg_over_time(demo_latency{source_id=\"my-metric-source\",deployment=\"my-service-deployment\"}[5m])",
           "thresholds": [
             {
               "level": "warning",
               "operator": "gte",
               "value": 50
             },
             {
               "level": "critical",
               "operator": "gt",
               "value": 100
             }
           ],
           "documentation": {
             "description": "This is a valid markdown description.\n\n**Use**: This indicates nothing. It is placeholder text.\n\n**Type**: Gauge\n**Frequency**: 60 s\n",
             "measurement": "Average latency over last 5 minutes per instance",
             "recommendedResponse": "Panic! Run around in circles flailing your arms.",
             "thresholdNote": "These are environment specific",
             "title": "Doc Performance Indicator"
           },
           "presentation": {
             "chartType": "line",
             "currentValue": false,
             "interval": "1m0s"
           }
         },
         {
           "name": "success_percentage",
           "promql": "success_percentage_promql{source_id=\"origin\"}",
           "documentation": {
             "title": "Success Percentage"
           }
         }
       ],
       "layout": {
         "title": "Monitoring Document Product",
         "description": "Document description",
         "sections": [
           {
             "title": "Indicators",
             "description": "This section includes indicators",
             "indicators": [
               "doc_performance_indicator"
             ]
           }
         ],
         "owner": "Example Team"
       }
     }
   ]
   

   Specifically, you should get a single product called my-component with 3 indicators, some metadata, and a layout section.

Developing with Docker

We publish Docker images for all available jobs:

• indicatorprotocol/bosh-indicator-protocol-registry
• indicatorprotocol/bosh-indicator-protocol-registry-proxy
• indicatorprotocol/bosh-indicator-protocol-registry-agent
• indicatorprotocol/bosh-indicator-protocol-cf-auth-proxy
• indicatorprotocol/bosh-indicator-protocol-status-controller
• indicatorprotocol/bosh-indicator-protocol-prometheus-controller
• indicatorprotocol/bosh-indicator-protocol-grafana-controller

Additionally, there is a docker-compose.yml file to orchestrate all of the jobs. There are a number of environment variables required, see ./scripts/start_docker_compose.sh for reference.

The registry proxy runs on port 10567 by default. To curl indicator documents:

curl https://localhost:10567/v1/indicator-documents -k \
--key test_fixtures/client.key \
--cert test_fixtures/client.pem \
--cacert test_fixtures/ca.key

Grafana and Prometheus controller jobs generate dashboard and alerting rules files, respectively, based on indicator documents availble in the registry. Both controllers write the generated files internally to the container in /alerts and /dashboards.