Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Split metrics to multiple http sources #13

Merged
merged 3 commits into from
Apr 18, 2019
Merged

Split metrics to multiple http sources #13

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
39891fa
Split metrics to multiple http sources
samjsong Apr 16, 2019
ae6dd95
split metrics before carbonv2 output plugin
samjsong Apr 17, 2019
f9657d0
Merge branch 'master' of github.com:SumoLogic/sumologic-kubernetes-co…
samjsong Apr 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions deploy/docker/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ RUN apk add --no-cache --update --virtual .build-deps sudo build-base ruby-dev \
&& gem install snappy

RUN gem install fluent-plugin-sumologic_output \
&& gem install fluent-plugin-rewrite-tag-filter \
&& gem install fluent-plugin-carbon-v2 \
&& gem install fluent-plugin-datapoint \
&& gem install fluent-plugin-protobuf
Expand Down
109 changes: 107 additions & 2 deletions deploy/kubernetes/fluentd-sumologic.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,14 @@ metadata:
type: Opaque
stringData:
endpoint-metrics: XXXX
endpoint-metrics-apiserver: XXXX
endpoint-metrics-cadvisor: XXXX
endpoint-metrics-kubelet: XXXX
endpoint-metrics-kube-controller-manager: XXXX
endpoint-metrics-kube-scheduler: XXXX
endpoint-metrics-kube-state: XXXX
endpoint-metrics-node-exporter: XXXX
endpoint-metrics-prometheus-operator: XXXX
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
Expand Down Expand Up @@ -76,10 +84,67 @@ data:
@type datapoint
tag prometheus.datapoint
</match>
<filter prometheus.datapoint>
<match prometheus.datapoint>
@type rewrite_tag_filter
samjsong marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this part do?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The rewrite_tag_filter output plugin allows you to rewrite the tag for rerouting. In this case the plugin looks for the job and appends it to the already existing tag, which becomes prometheus.datapoint.<JOB>

<rule>
key job
pattern ^(.*)$
tag ${tag}.$1
</rule>
</match>
<filter prometheus.datapoint.**>
@type carbon_v2
</filter>
<match prometheus.datapoint>

<match prometheus.datapoint.apiserver**>
@type sumologic
endpoint "#{ENV['SUMO_ENDPOINT_METRICS_APISERVER']}"
data_type metrics
metric_data_format carbon2
</match>
<match prometheus.datapoint.cadvisor**>
@type sumologic
endpoint "#{ENV['SUMO_ENDPOINT_METRICS_CADVISOR']}"
data_type metrics
metric_data_format carbon2
</match>
<match prometheus.datapoint.kubelet**>
@type sumologic
endpoint "#{ENV['SUMO_ENDPOINT_METRICS_KUBELET']}"
data_type metrics
metric_data_format carbon2
</match>
<match prometheus.datapoint.kube-controller-manager**>
@type sumologic
endpoint "#{ENV['SUMO_ENDPOINT_METRICS_KUBE_CONTROLLER_MANAGER']}"
data_type metrics
metric_data_format carbon2
</match>
<match prometheus.datapoint.kube-scheduler**>
@type sumologic
endpoint "#{ENV['SUMO_ENDPOINT_METRICS_KUBE_SCHEDULER']}"
data_type metrics
metric_data_format carbon2
</match>
<match prometheus.datapoint.kube-state**>
@type sumologic
endpoint "#{ENV['SUMO_ENDPOINT_METRICS_KUBE_STATE']}"
data_type metrics
metric_data_format carbon2
</match>
<match prometheus.datapoint.node-exporter**>
@type sumologic
endpoint "#{ENV['SUMO_ENDPOINT_METRICS_NODE_EXPORTER']}"
data_type metrics
metric_data_format carbon2
</match>
<match prometheus.datapoint.prometheus-operator**>
@type sumologic
endpoint "#{ENV['SUMO_ENDPOINT_METRICS_PROMETHEUS_OPERATOR']}"
data_type metrics
metric_data_format carbon2
</match>
<match prometheus.datapoint.**>
@type sumologic
endpoint "#{ENV['SUMO_ENDPOINT_METRICS']}"
data_type metrics
Expand Down Expand Up @@ -172,6 +237,46 @@ spec:
secretKeyRef:
name: sumologic
key: endpoint-metrics
- name: SUMO_ENDPOINT_METRICS_APISERVER
valueFrom:
secretKeyRef:
name: sumologic
key: endpoint-metrics-apiserver
- name: SUMO_ENDPOINT_METRICS_CADVISOR
valueFrom:
secretKeyRef:
name: sumologic
key: endpoint-metrics-cadvisor
- name: SUMO_ENDPOINT_METRICS_KUBELET
valueFrom:
secretKeyRef:
name: sumologic
key: endpoint-metrics-kubelet
- name: SUMO_ENDPOINT_METRICS_KUBE_CONTROLLER_MANAGER
valueFrom:
secretKeyRef:
name: sumologic
key: endpoint-metrics-kube-controller-manager
- name: SUMO_ENDPOINT_METRICS_KUBE_SCHEDULER
valueFrom:
secretKeyRef:
name: sumologic
key: endpoint-metrics-kube-scheduler
- name: SUMO_ENDPOINT_METRICS_KUBE_STATE
valueFrom:
secretKeyRef:
name: sumologic
key: endpoint-metrics-kube-state
- name: SUMO_ENDPOINT_METRICS_NODE_EXPORTER
valueFrom:
secretKeyRef:
name: sumologic
key: endpoint-metrics-node-exporter
- name: SUMO_ENDPOINT_METRICS_PROMETHEUS_OPERATOR
valueFrom:
secretKeyRef:
name: sumologic
key: endpoint-metrics-prometheus-operator
---
apiVersion: v1
kind: Service
Expand Down