Sumologic exporter and metric pipelines into OT Collector & agent #1647

mat-rumian · 2021-06-10T22:09:32Z

Description

Addition of the Sumologic exporter to the Tracing OT Collector pipeline. This will add possibility to send received metrics on the OTC receiver (OTLP protocol) side to Sumo.

Testing performed

Redeploy fluentd and fluentd-events pods
Confirm events, logs, and metrics are coming in

mat-rumian · 2021-06-10T22:11:11Z

@pmm-sumo I'm not sure from where should I take a value for source_host tag in the Sumologic exporter configuration - it can be set here

mat-rumian · 2021-06-10T22:18:36Z

@perk-sumo maybe you will be able to help me to get a single environment variable related to the metrics endpoint url.

{{ include "kubernetes.sources.envs" (dict "Context" $ctx "Type" "metrics") }} - this returns all environment variables related to the metrics. I would like to add only one env var which is related to "default-metrics" source.

sumo-drosiek · 2021-06-11T05:30:40Z

@mat-rumian Regarding hostname. I think we should use hostname from exporters (attribute net.peer.name), otherwise we should use local hostname, but it is not implemented in exporter righrt now

Regarding metrics:
You can add helper for single source. For now there is no possibility to extract single source (related code)

deploy/helm/sumologic/values.yaml

sumo-drosiek · 2021-06-14T06:05:55Z

@mat-rumian LGTM

pmalek-sumo · 2021-10-01T12:06:58Z

@sumo-drosiek @mat-rumian Do we want to merge this in? And include in 2.2?

mat-rumian · 2021-10-01T12:12:31Z

@pmalek-sumo yes, it is needed in case of customer call for metrics from OpenTelemetry instrumentation

pmalek-sumo · 2021-10-01T12:14:43Z

tests/tracing/static/simple.output.yaml

+          queue_size: 5000
+        source_category: 
+        source_host: '%{host.name}'
+        source_name: 


Should this be empty?

pmalek-sumo · 2021-10-01T12:14:47Z

tests/tracing/static/simple.output.yaml

+          enabled: false
+          num_consumers: 10
+          queue_size: 5000
+        source_category: 


Should this be empty?

pmalek-sumo · 2021-10-01T12:14:52Z

tests/tracing/static/collection-monitoring-false.output.yaml

+          enabled: false
+          num_consumers: 10
+          queue_size: 5000
+        source_category: 


Should this be empty?

pmalek-sumo · 2021-10-01T12:14:59Z

tests/tracing/static/collection-monitoring-false.output.yaml

+          queue_size: 5000
+        source_category: 
+        source_host: '%{host.name}'
+        source_name: 


Should this be empty?

pmalek-sumo · 2021-10-01T12:15:45Z

deploy/helm/sumologic/templates/traces/otelcol/deployment.yaml

+{{ include "kubernetes.sources.envs" (dict "Context" $ctx "Type" "traces") | nindent 8 }}
+{{- include "kubernetes.sources.env" (dict "Context" $ctx "Type" "metrics" "Name" "default" ) | nindent 8 }}


{{ and {{, can we stick to one or the other?

sumo-drosiek · 2021-10-01T12:55:50Z

Isn't that kind of duplication of replacement fluentd by otc?

perk-sumo · 2021-10-25T09:58:39Z

Hey, I think we'd like to go with otelcol-sumo binary (the one we use for metadata) in this case. Let's revisit that for the v2.3.

perk-sumo · 2021-11-24T10:53:26Z

Hm, now that I think of it I think this should be enabled in both tracing otelcol and the metadata one.

Co-authored-by: Patryk Małek <pmalek@sumologic.com>

perk-sumo

LGTM

pmalek-sumo · 2021-12-01T10:24:05Z

@pmm-sumo Can we add a meaningful description to this PR 🙏 ? And after that let's merge it

mat-rumian requested review from pmm-sumo and sumo-drosiek June 10, 2021 22:09

mat-rumian requested a review from a team as a code owner June 10, 2021 22:09

mat-rumian requested review from perk-sumo and pmalek-sumo June 10, 2021 22:16

mat-rumian requested a review from kkujawa-sumo June 10, 2021 22:20

mat-rumian changed the title ~~[WIP] Sumologic exporter into OT Agent & Collector~~ Sumologic exporter into OT Agent & Collector Jun 11, 2021

mat-rumian changed the title ~~Sumologic exporter into OT Agent & Collector~~ Sumologic exporter and metric pipelines into OT Collector & agent Jun 11, 2021

pmalek-sumo reviewed Jun 11, 2021

View reviewed changes

deploy/helm/sumologic/values.yaml Outdated Show resolved Hide resolved

deploy/helm/sumologic/values.yaml Outdated Show resolved Hide resolved

deploy/helm/sumologic/values.yaml Outdated Show resolved Hide resolved

perk-sumo added this to the v2.2 milestone Jun 29, 2021

pmalek-sumo reviewed Oct 1, 2021

View reviewed changes

perk-sumo modified the milestones: v2.2, v2.3 Oct 25, 2021

perk-sumo modified the milestones: v2.4, v2.3 Nov 24, 2021

mat-rumian added 6 commits December 1, 2021 10:05

Add metrics environment variable

7df06b6

Add Sumologic metrics exporter into OT agent&collector

039d8ce

Replace source tags

dcd5fe9

Revert memory limit value

419765d

Update tests

12585ed

Add new function to get specific key

75d5766

mat-rumian and others added 9 commits December 1, 2021 10:05

Add indents

1cfe06a

Add resourcedetection, setup source_host

5fb1293

Fix tests

a025d00

Setup detector, fix host.name

440d51d

Add space

f16e990

Add space, fix tests

86d3fac

Comments update

b4a1e63

Co-authored-by: Patryk Małek <pmalek@sumologic.com>

Comment update

a3ea758

Co-authored-by: Patryk Małek <pmalek@sumologic.com>

Comment update

5db08eb

Co-authored-by: Patryk Małek <pmalek@sumologic.com>

pmm-sumo force-pushed the mat-rumian-sumologic-exporter-into-otc branch from 887cf6f to 5db08eb Compare December 1, 2021 09:05

github-actions bot approved these changes Dec 1, 2021

View reviewed changes

pmm-sumo approved these changes Dec 1, 2021

View reviewed changes

perk-sumo approved these changes Dec 1, 2021

View reviewed changes

pmalek-sumo merged commit fb09256 into main Dec 1, 2021

pmalek-sumo deleted the mat-rumian-sumologic-exporter-into-otc branch December 1, 2021 11:46

This was referenced Mar 14, 2022

fix(helm): always create default metrics source if traces are enabled #2182

Merged

Having traces enabled and metrics disabled still allows to send metrics via (traces) otelcol's instances #2183

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sumologic exporter and metric pipelines into OT Collector & agent #1647

Sumologic exporter and metric pipelines into OT Collector & agent #1647

mat-rumian commented Jun 10, 2021 •

edited

mat-rumian commented Jun 10, 2021

mat-rumian commented Jun 10, 2021

sumo-drosiek commented Jun 11, 2021

sumo-drosiek commented Jun 14, 2021

pmalek-sumo commented Oct 1, 2021

mat-rumian commented Oct 1, 2021

pmalek-sumo Oct 1, 2021

pmalek-sumo Oct 1, 2021

pmalek-sumo Oct 1, 2021

pmalek-sumo Oct 1, 2021

pmalek-sumo Oct 1, 2021

sumo-drosiek commented Oct 1, 2021

perk-sumo commented Oct 25, 2021

perk-sumo commented Nov 24, 2021

perk-sumo left a comment

pmalek-sumo commented Dec 1, 2021

		{{ include "kubernetes.sources.envs" (dict "Context" $ctx "Type" "traces") \| nindent 8 }}
		{{- include "kubernetes.sources.env" (dict "Context" $ctx "Type" "metrics" "Name" "default" ) \| nindent 8 }}

Sumologic exporter and metric pipelines into OT Collector & agent #1647

Sumologic exporter and metric pipelines into OT Collector & agent #1647

Conversation

mat-rumian commented Jun 10, 2021 • edited

Description

Testing performed

mat-rumian commented Jun 10, 2021

mat-rumian commented Jun 10, 2021

sumo-drosiek commented Jun 11, 2021

sumo-drosiek commented Jun 14, 2021

pmalek-sumo commented Oct 1, 2021

mat-rumian commented Oct 1, 2021

pmalek-sumo Oct 1, 2021

Choose a reason for hiding this comment

pmalek-sumo Oct 1, 2021

Choose a reason for hiding this comment

pmalek-sumo Oct 1, 2021

Choose a reason for hiding this comment

pmalek-sumo Oct 1, 2021

Choose a reason for hiding this comment

pmalek-sumo Oct 1, 2021

Choose a reason for hiding this comment

sumo-drosiek commented Oct 1, 2021

perk-sumo commented Oct 25, 2021

perk-sumo commented Nov 24, 2021

perk-sumo left a comment

Choose a reason for hiding this comment

pmalek-sumo commented Dec 1, 2021

mat-rumian commented Jun 10, 2021 •

edited