Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add NewRelic system monitor DaemonSet example #16014

Merged
merged 1 commit into from
Nov 3, 2015
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
179 changes: 179 additions & 0 deletions examples/newrelic/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,179 @@
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->

<!-- BEGIN STRIP_FOR_RELEASE -->

<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
width="25" height="25">
<img src="http://kubernetes.io/img/warning.png" alt="WARNING"
width="25" height="25">

<h2>PLEASE NOTE: This document applies to the HEAD of the source tree</h2>

If you are using a released version of Kubernetes, you should
refer to the docs that go with that version.

<strong>
The latest 1.0.x release of this document can be found
[here](http://releases.k8s.io/release-1.0/examples/newrelic/README.md).

Documentation for other releases can be found at
[releases.k8s.io](http://releases.k8s.io).
</strong>
--

<!-- END STRIP_FOR_RELEASE -->

<!-- END MUNGE: UNVERSIONED_WARNING -->

## New Relic Server Monitoring Agent Example

This example shows how to run a New Relic server monitoring agent as a pod in a DaemonSet on an existing Kubernetes cluster.

This example will create a DaemonSet which places the New Relic monitoring agent on every node in the cluster. It's also fairly trivial to exclude specific Kubernetes nodes from the DaemonSet to just monitor specific servers.

### Step 0: Prerequisites

This process will create priviliged containers which have full access to the host system for logging. Beware of the security implications of this.

If you are using a Salt based KUBERNETES\_PROVIDER (**gce**, **vagrant**, **aws**), you should make sure the creation of privileged containers via the API is enabled. Check `cluster/saltbase/pillar/privilege.sls`.

DaemonSets must be enabled on your cluster. Instructions for enabling DaemonSet can be found [here](../../docs/api.md#enabling-the-extensions-group).

### Step 1: Configure New Relic Agent

The New Relic agent is configured via environment variables. We will configure these environment variables in a sourced bash script, encode the environment file data, and store it in a secret which will be loaded at container runtime.

The [New Relic Linux Server configuration page]
(https://docs.newrelic.com/docs/servers/new-relic-servers-linux/installation-configuration/configuring-servers-linux) lists all the other settings for nrsysmond.

To create an environment variable for a setting, prepend NRSYSMOND_ to its name. For example,

```console
loglevel=debug
```

translates to

```console
NRSYSMOND_loglevel=debug
```

Edit examples/newrelic/nrconfig.env and set up the environment variables for your NewRelic agent. Be sure to edit the license key field and fill in your own New Relic license key.

Now, let's vendor the config into a secret.

```console
$ cd examples/newrelic/
$ ./config-to-secret.sh
```

<!-- BEGIN MUNGE: EXAMPLE newrelic-config-template.yaml -->

```yaml
apiVersion: v1
kind: Secret
metadata:
name: newrelic-config
type: Opaque
data:
config: {{config_data}}
```

[Download example](newrelic-config-template.yaml?raw=true)
<!-- END MUNGE: EXAMPLE newrelic-config-template.yaml -->

The script will encode the config file and write it to `newrelic-config.yaml`.

Finally, submit the config to the cluster:

```console
$ kubectl create -f examples/newrelic/newrelic-config.yaml
```

### Step 2: Create the DaemonSet definition.

The DaemonSet definition instructs Kubernetes to place a newrelic sysmond agent on each Kubernetes node.

<!-- BEGIN MUNGE: EXAMPLE newrelic-daemonset.yaml -->

```yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: newrelic-agent
labels:
tier: monitoring
app: newrelic-agent
version: v1
spec:
template:
metadata:
labels:
name: newrelic
spec:
# Filter to specific nodes:
# nodeSelector:
# app: newrelic
securityContext:
privileged: true
hostPID: true
hostIPC: true
hostNetwork: true
containers:
- resources:
requests:
cpu: 0.15
env:
- name: NRSYSMOND_logfile
value: "/var/log/nrsysmond.log"
image: newrelic/nrsysmond
name: newrelic
command: [ "bash", "-c", "source /etc/kube-newrelic/config && /usr/sbin/nrsysmond -E -F" ]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a valid yaml array? Have not seen this syntax before (although I am not a yaml expert by any means).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep it's valid. For whatever reason I couldn't get it working using the other syntax.

volumeMounts:
- name: newrelic-config
mountPath: /etc/kube-newrelic
readOnly: true
- name: dev
mountPath: /dev
- name: run
mountPath: /var/run/docker.sock
- name: sys
mountPath: /sys
- name: log
mountPath: /var/log
volumes:
- name: newrelic-config
secret:
secretName: newrelic-config
- name: dev
hostPath:
path: /dev
- name: run
hostPath:
path: /var/run/docker.sock
- name: sys
hostPath:
path: /sys
- name: log
hostPath:
path: /var/log
```

[Download example](newrelic-daemonset.yaml?raw=true)
<!-- END MUNGE: EXAMPLE newrelic-daemonset.yaml -->

The daemonset instructs Kubernetes to spawn pods on each node, mapping /dev/, /run/, /sys/, and /var/log to the container. It also maps the secrets we set up earlier to /etc/kube-newrelic/config, and sources them in the startup script, configuring the agent properly.

### Known issues

It's a bit cludgy to define the environment variables like we do here in these config files. There is [another issue](https://github.com/kubernetes/kubernetes/issues/4710) to discuss adding mapping secrets to environment variables in Kubernetes.

<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/examples/newrelic/README.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
20 changes: 20 additions & 0 deletions examples/newrelic/config-to-secret.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/bin/bash

# Copyright 2014 The Kubernetes Authors All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Encodes the environment variables into a Kubernetes secret.

BASE64_ENC=$(cat nrconfig.env | base64 --wrap=0)
sed -e "s#{{config_data}}#${BASE64_ENC}#g" ./newrelic-config-template.yaml > newrelic-config.yaml
7 changes: 7 additions & 0 deletions examples/newrelic/newrelic-config-template.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
apiVersion: v1
kind: Secret
metadata:
name: newrelic-config
type: Opaque
data:
config: {{config_data}}
10 changes: 10 additions & 0 deletions examples/newrelic/newrelic-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# This file should be overwritten by ./config-to-secret.sh
# This file is in place to statisfy the kubernetes documentation tests.

# apiVersion: v1
# kind: Secret
# metadata:
# name: newrelic-config
# type: Opaque
# data:
# config: base64 encoded
60 changes: 60 additions & 0 deletions examples/newrelic/newrelic-daemonset.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
name: newrelic-agent
labels:
tier: monitoring
app: newrelic-agent
version: v1
spec:
template:
metadata:
labels:
name: newrelic
spec:
# Filter to specific nodes:
# nodeSelector:
# app: newrelic
securityContext:
privileged: true
hostPID: true
hostIPC: true
hostNetwork: true
containers:
- resources:
requests:
cpu: 0.15
env:
- name: NRSYSMOND_logfile
value: "/var/log/nrsysmond.log"
image: newrelic/nrsysmond
name: newrelic
command: [ "bash", "-c", "source /etc/kube-newrelic/config && /usr/sbin/nrsysmond -E -F" ]
volumeMounts:
- name: newrelic-config
mountPath: /etc/kube-newrelic
readOnly: true
- name: dev
mountPath: /dev
- name: run
mountPath: /var/run/docker.sock
- name: sys
mountPath: /sys
- name: log
mountPath: /var/log
volumes:
- name: newrelic-config
secret:
secretName: newrelic-config
- name: dev
hostPath:
path: /dev
- name: run
hostPath:
path: /var/run/docker.sock
- name: sys
hostPath:
path: /sys
- name: log
hostPath:
path: /var/log
2 changes: 2 additions & 0 deletions examples/newrelic/nrconfig.env
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
export NRSYSMOND_loglevel=debug
export NRSYSMOND_license_key=REPLACE_LICENSE_KEY_HERE