Forwarding the SNMP traps from snmp_notifier running on kubernetes to alerting system

[snimje]

What did you do?
ran the snmp-notifier as a deployment inside a kubernetes cluster. the YAML file is as below:
`---
apiVersion: apps/v1
kind: Deployment
metadata:
name: snmp-notifier-deployment
namespace: pf9-monitoring
labels:
component: snmp-notifier
spec:
replicas: 1
selector:
matchLabels:
app: snmp-notifier
template:
metadata:
labels:
app: snmp-notifier
spec:
containers:
- image: maxwo/snmp-notifier:v1.2.1
name: snmp-notifier
ports:
- containerPort: 9464
name: web-endpoint
protocol: TCP
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 500m
memory: 256Mi

apiVersion: v1
kind: Service
metadata:
name: snmp-notifier-service
namespace: pf9-monitoring
spec:
type: ClusterIP
ports:
- name: web-endpoint
port: 9464
targetPort: 9464
protocol: TCP
selector:
app: snmp-notifier
...Webhook: My webhook config:global:
resolve_timeout: 5m
route:
group_by: ['...']
group_wait: 3s
group_interval: 1m
repeat_interval: 15m
receiver: 'snmp_notifier'
receivers:

• name: 'snmp_notifier'
  webhook_configs:
  • send_resolved: true
    url: http://10.140.15.246:9464/alerts`
    10.140.15.246 is IP of snmp_notifier service.
    I can see the alerts are posted from alert-manager to snmp_notifier in the snmp_notifier pod's log
    10.139.200.2 - - [21/Jan/2022:16:30:09 +0000] "POST /alerts HTTP/1.1" 200 0

I would like to know to things
[1] Is this configuration correct ?
[2] Additional configuration procedure to forward these alerts from snmp_notifier pod to the external legacy alerting system say netcool. This I could not find anywhere. Is there a secret that one need to create ? It will be helpful if I can get some help on how to set this up with an example. Thank you!

What did you expect to see?
procedure to forward the SNMP traps received in the snmp_notifier pod to the alerting system.

What did you see instead? Under which circumstances?

Environment
Kubernetes 1.21

• System information:

Linux 3.10.0-957.1.3.el7.x86_64 x86_64

• SNMP notifier version:

snmp_notifier, version 1.2.1 (branch: HEAD, revision: 73773b7)
build user: root@4c095c7fad1b
build date: 20210820-08:20:04
go version: go1.16
platform: linux/amd64

• Alertmanager version:

alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d)
build user: root@dee35927357f
build date: 20200617-08:54:02
go version: go1.14.4

• Prometheus version:

prometheus, version 2.32.1 (branch: HEAD, revision: 41f1a8125e664985dd30674e5bdf6b683eff5d32)
build user: root@54b6dbd48b97
build date: 20211217-22:08:06
go version: go1.17.5
platform: linux/amd64

• Alertmanager command line:

    1 nobody    1:54 /bin/alertmanager --config.file=/etc/alertmanager/config/alertmanager.yaml --storage.path=/alertmanager --data.retention=120h --cluster.listen-address= --web.listen-ad

* SNMP notifier command line:

/bin/snmp_notifier --snmp.trap-description-template=/etc/snmp_notifier/description-template.tpl

[snimje]

sharing the alertmanagert YAML and the snmp_notifier deployment manifest files again as they might not be clear in the above issue opening page

global:
resolve_timeout: 5m
route:
group_by: ['...']
group_wait: 3s
group_interval: 1m
repeat_interval: 15m
receiver: 'snmp_notifier'
receivers:

• name: 'snmp_notifier'
  webhook_configs:
  • send_resolved: true
    url: http://10.140.15.246:9464/alerts

---

apiVersion: apps/v1
kind: Deployment
metadata:
name: snmp-notifier-deployment
namespace: pf9-monitoring
labels:
component: snmp-notifier
spec:
replicas: 1
selector:
matchLabels:
app: snmp-notifier
template:
metadata:
labels:
app: snmp-notifier
spec:
containers:
- image: maxwo/snmp-notifier:v1.2.1
name: snmp-notifier
ports:
- containerPort: 9464
name: web-endpoint
protocol: TCP
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 500m
memory: 256Mi

apiVersion: v1
kind: Service
metadata:
name: snmp-notifier-service
namespace: pf9-monitoring
spec:
type: ClusterIP
ports:
- name: web-endpoint
port: 9464
targetPort: 9464
protocol: TCP
selector:
app: snmp-notifier

[albertocrj]

Hi @snimje,

I have a similar case of yours and got it working, so I have some tips for you:

• In alertmanager, use service domain address instead service IP address:
  Usually in a cluster your local domain is cluster.local, so you can refer to your to service like this:
  name-of-your-service.namescapece-of-your-service.svc.cluster.local, so In my case the name of the service is "snmp-notifier". In alertmanager I have this config:

receivers:
- name: snmp_notifier
webhook_configs:
- send_resolved: true
url: http://snmp-notifier.my-namespace.svc.cluster.local:9464/alerts
`

• In your deployment you need to change the snmp server destination (--snmp.destination), by default is "localhost(127.0.0.1)", so in my case here's what I changed:

spec:
containers:
- image: maxwo/snmp-notifier
imagePullPolicy: Always
command: ["/bin/snmp_notifier"]
args: ["--snmp.trap-description-template=/etc/snmp_notifier/description-template.tpl", "--snmp.community=$(SNMP_NOTIFIER_COMMUNITY)", "--snmp.destination=$(SNMP_NOTIFIER_SERVER):$(SNMP_NOTIFIER_SERVER_PORT)"]
env:
- name: SNMP_NOTIFIER_COMMUNITY
value: "public"
- name: SNMP_NOTIFIER_SERVER
value: "IP_OF_YOUR_DEST_SERVER"
- name: SNMP_NOTIFIER_SERVER_PORT
value: "162"

You may change as many paremeters as you want. You have other ways to inject data/variables in kubernetes, I just gave you
a simple example, please refer to: https://kubernetes.io/docs/tasks/inject-data-application/define-environment-variable-container/

@maxwo,
May be it's a good idea to provide an example file to deploy snmp_notifier on kubernetes, I can help to provide it if you want.

Hope that's helps.

Cheers,

Alberto Viana

[maxwo]

I am considering providing a Helm chart to help deploy on Kubernetes clusters.

There is already an issue on this: #22

[maxwo]

@albertocrj Here is a Helm chart to deploy on Kubernetes: https://github.com/maxwo/snmp-notifier-helm-chart

I will create a PR to add it the the Prometheus community charts, so that deployment next to an already up Prometheus instance will be even easier.

[albertocrj]

Hi @maxwo,
Great news man, I will give a try on it.

Thanks