-
Notifications
You must be signed in to change notification settings - Fork 185
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add OtelCol as alternative logs metadata provider #1697
Merged
Merged
Changes from all commits
Commits
Show all changes
37 commits
Select commit
Hold shift + click to select a range
e29edca
introduce sumologic.logs.provider
sumo-drosiek a9336d3
feat(helm): add otelcol logs provider (statefulset)
sumo-drosiek d3f3b91
feat(helm,logs): use fluentds hpa, services, pdb by otelcol
sumo-drosiek 410f1ce
feat(helm,logs): add logs.otelcol.enabled and logs.fluentd.enabled he…
sumo-drosiek c0fbb58
feat(helm,logs,otelcol): add k8s metadata
sumo-drosiek 501b33c
feat(helm,logs): add common objects for logs pipeline
sumo-drosiek fa55e27
feat(helm,logs): add source headers to otelcol logs
sumo-drosiek 4d9fd4f
refactor(yamllint): update values.yaml
sumo-drosiek 1cb8e52
refactor(helm,otelcol): converts fluentd-specific values to common fo…
sumo-drosiek 1dff831
refactor(helm,logs,otelcol): add persistance todo
sumo-drosiek 2fe73f4
refactor(helm,logs,otelcol): remove traves envs from logs sts
sumo-drosiek 30cdd3c
refactor(helm,otelcol,logs): remove agent attribute
sumo-drosiek be26e00
refactor(helm,otelcol,logs): clean up config
sumo-drosiek 691ca80
refactor,docs: update otelcol related comments
sumo-drosiek a747cb2
feat(helm,logs): add check for provider values
sumo-drosiek 0ca797c
feat(helm,logs,otelcol): add source processor
sumo-drosiek 63d0d2e
feat(helm): add experimental note to provider
sumo-drosiek 25dac5f
fix(helm): revert autoscalling option for fluent hpa
sumo-drosiek ac2127d
fix(helm): use autoscalling.enabled option for common hpa
sumo-drosiek 6f958bc
fix(helm): fix extracting k8s metadata from fluent.tag
pmalek 96c1ddd
fix: remove `sumologic.open_endpoint` property from values.yaml
andrzej-stencel 20c366a
cleanup: remove commented-out sumologic extension
andrzej-stencel 14a8943
cleanup: remove unused comments
andrzej-stencel 352c3db
fix: use sanitized pod name in _sourceCategory for logs
andrzej-stencel cb4814f
fix: correct comment indentation
andrzej-stencel 279f46a
cleanup: remove incorrect comment
andrzej-stencel 75bd207
fix: Move `otelcol.logs` to `otelcol.metadata.logs`
andrzej-stencel 60300b5
fix: move `sumologic.logs.provider` to `sumologic.logs.metadata.provi…
andrzej-stencel fa22248
fix: fix property name in error message
andrzej-stencel fd3c93b
refactor: simplify service labels logic for OtelCol
andrzej-stencel 353d480
fix: move `logs.*` to `logs.metadata.*`
andrzej-stencel eda1093
docs: Add otelcol logs configuration properties to README
andrzej-stencel cd77253
docs: fix markdown formatting
andrzej-stencel ab0758d
docs: add link to OT distro configuration
andrzej-stencel 0daa72f
docs: remove redundant word
andrzej-stencel e935ec5
feat(configuration): add logLevel option for logs otelcol
andrzej-stencel 471f869
fix: use pod/namespace name in logs enrichment otelcol
andrzej-stencel File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
{{ tpl (toYaml .Values.otelcol.metadata.logs.config | replace ": '{{" ": {{" | replace "}}'" "}}") . | nindent 2 }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
{{- if and (eq (include "logs.otelcol.enabled" .) "true") (.Values.logs.metadata.autoscaling.enabled) }} | ||
apiVersion: autoscaling/v2beta2 | ||
kind: HorizontalPodAutoscaler | ||
metadata: | ||
name: {{ template "sumologic.metadata.name.logs.hpa" . }} | ||
labels: | ||
app: {{ template "sumologic.labels.app.logs.hpa" . }} | ||
{{- include "sumologic.labels.common" . | nindent 4 }} | ||
spec: | ||
scaleTargetRef: | ||
apiVersion: apps/v1 | ||
kind: StatefulSet | ||
name: {{ template "sumologic.metadata.name.logs.statefulset" . }} | ||
minReplicas: {{ .Values.logs.metadata.autoscaling.minReplicas }} | ||
maxReplicas: {{ .Values.logs.metadata.autoscaling.maxReplicas }} | ||
metrics: | ||
- type: Resource | ||
resource: | ||
name: cpu | ||
target: | ||
type: Utilization | ||
averageUtilization: {{ .Values.logs.metadata.autoscaling.targetCPUUtilizationPercentage }} | ||
{{- if .Values.logs.metadata.autoscaling.targetMemoryUtilizationPercentage }} | ||
- type: Resource | ||
resource: | ||
name: memory | ||
target: | ||
type: Utilization | ||
averageUtilization: {{ .Values.logs.metadata.autoscaling.targetMemoryUtilizationPercentage }} | ||
{{- end -}} | ||
{{- end -}} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
{{- if eq (include "logs.otelcol.enabled" .) "true" }} | ||
{{- if .Values.logs.metadata.podDisruptionBudget -}} | ||
apiVersion: policy/v1beta1 | ||
kind: PodDisruptionBudget | ||
metadata: | ||
name: {{ template "sumologic.metadata.name.logs.pdb" . }} | ||
spec: | ||
selector: | ||
matchLabels: | ||
app: {{ template "sumologic.labels.app.logs.statefulset" . }} | ||
{{ toYaml .Values.logs.metadata.podDisruptionBudget | indent 2 }} | ||
{{- end -}} | ||
{{- end -}} |
31 changes: 31 additions & 0 deletions
31
deploy/helm/sumologic/templates/logs/common/service-headless.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
{{- if eq (include "logs.otelcol.enabled" .) "true" }} | ||
apiVersion: v1 | ||
kind: Service | ||
metadata: | ||
name: {{ template "sumologic.metadata.name.logs.service-headless" . }} | ||
labels: | ||
app: {{ template "sumologic.labels.app.logs.service-headless" . }} | ||
{{- include "sumologic.labels.logs" . | nindent 4 }} | ||
{{- if .Values.otelcol.serviceLabels }} | ||
{{ toYaml .Values.otelcol.serviceLabels | indent 4 }} | ||
{{- end }} | ||
spec: | ||
selector: | ||
app: {{ template "sumologic.labels.app.logs.pod" . }} | ||
clusterIP: None | ||
ports: | ||
{{- if .Values.sumologic.traces.enabled }} | ||
- name: zipkin-write | ||
port: 9411 | ||
targetPort: 9411 | ||
protocol: TCP | ||
{{- end }} | ||
- name: fluent-bit | ||
port: 24321 | ||
targetPort: 24321 | ||
protocol: TCP | ||
- name: metrics | ||
port: 24231 | ||
targetPort: 24231 | ||
protocol: TCP | ||
{{- end }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
{{- if eq (include "logs.otelcol.enabled" .) "true" }} | ||
apiVersion: v1 | ||
kind: Service | ||
metadata: | ||
name: {{ template "sumologic.metadata.name.logs.service" . }} | ||
labels: | ||
app: {{ template "sumologic.labels.app.logs.service" . }} | ||
{{- include "sumologic.labels.scrape.logs" . | nindent 4 }} | ||
{{- include "sumologic.labels.common" . | nindent 4 }} | ||
{{- if .Values.otelcol.serviceLabels }} | ||
{{ toYaml .Values.otelcol.serviceLabels | indent 4 }} | ||
{{- end }} | ||
spec: | ||
selector: | ||
app: {{ template "sumologic.labels.app.logs.pod" . }} | ||
ports: | ||
{{- if .Values.sumologic.traces.enabled }} | ||
- name: zipkin-write | ||
port: 9411 | ||
targetPort: 9411 | ||
protocol: TCP | ||
{{- end }} | ||
- name: fluent-bit | ||
port: 24321 | ||
targetPort: 24321 | ||
protocol: TCP | ||
- name: metrics | ||
port: 24231 | ||
targetPort: 24231 | ||
protocol: TCP | ||
{{- end }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
2 changes: 1 addition & 1 deletion
2
deploy/helm/sumologic/templates/logs/fluentd/service-headless.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
11 changes: 11 additions & 0 deletions
11
deploy/helm/sumologic/templates/logs/otelcol/configmap.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,11 @@ | ||
{{- if eq (include "logs.otelcol.enabled" .) "true" }} | ||
apiVersion: v1 | ||
kind: ConfigMap | ||
metadata: | ||
name: {{ template "sumologic.metadata.name.logs.configmap" . }} | ||
labels: | ||
app: {{ template "sumologic.labels.app.logs.configmap" . }} | ||
{{- include "sumologic.labels.common" . | nindent 4 }} | ||
data: | ||
{{- (tpl (.Files.Glob "conf/logs/otelcol/config.yaml").AsConfig .) | nindent 2 }} | ||
{{- end }} |
147 changes: 147 additions & 0 deletions
147
deploy/helm/sumologic/templates/logs/otelcol/statefulset.yaml
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,147 @@ | ||
{{- if eq (include "logs.otelcol.enabled" .) "true" }} | ||
apiVersion: apps/v1 | ||
kind: StatefulSet | ||
metadata: | ||
name: {{ template "sumologic.metadata.name.logs.statefulset" . }} | ||
labels: | ||
app: {{ template "sumologic.labels.app.logs.statefulset" . }} | ||
{{- include "sumologic.labels.common" . | nindent 4 }} | ||
spec: | ||
selector: | ||
matchLabels: | ||
app: {{ template "sumologic.labels.app.logs.pod" . }} | ||
serviceName: {{ template "sumologic.metadata.name.logs.service-headless" . }} | ||
podManagementPolicy: "Parallel" | ||
{{- if not .Values.logs.metadata.autoscaling.enabled }} | ||
replicas: {{ .Values.logs.metadata.statefulset.replicaCount }} | ||
{{- end }} | ||
template: | ||
metadata: | ||
annotations: | ||
checksum/config: {{ include (print $.Template.BasePath "/logs/otelcol/configmap.yaml") . | sha256sum }} | ||
{{- if .Values.sumologic.podAnnotations }} | ||
{{ toYaml .Values.sumologic.podAnnotations | indent 8 }} | ||
{{- end }} | ||
{{- if .Values.otelcol.podAnnotations }} | ||
{{ toYaml .Values.otelcol.podAnnotations | indent 8 }} | ||
{{- end }} | ||
{{- if .Values.logs.metadata.statefulset.podAnnotations }} | ||
{{ toYaml .Values.logs.metadata.statefulset.podAnnotations | indent 8 }} | ||
{{- end }} | ||
labels: | ||
app: {{ template "sumologic.labels.app.logs.pod" . }} | ||
{{- include "sumologic.labels.common" . | nindent 8 }} | ||
{{- if .Values.sumologic.podLabels }} | ||
{{ toYaml .Values.sumologic.podLabels | indent 8 }} | ||
{{- end }} | ||
{{- if .Values.otelcol.podLabels }} | ||
{{ toYaml .Values.otelcol.podLabels | indent 8 }} | ||
{{- end }} | ||
{{- if .Values.logs.metadata.statefulset.podLabels }} | ||
{{ toYaml .Values.logs.metadata.statefulset.podLabels | indent 8 }} | ||
{{- end }} | ||
spec: | ||
serviceAccountName: {{ template "sumologic.metadata.name.roles.serviceaccount" . }} | ||
{{- if .Values.logs.metadata.statefulset.nodeSelector }} | ||
nodeSelector: | ||
{{ toYaml .Values.logs.metadata.statefulset.nodeSelector | indent 8 }} | ||
{{- end }} | ||
{{- if .Values.logs.metadata.statefulset.tolerations }} | ||
tolerations: | ||
{{ toYaml .Values.logs.metadata.statefulset.tolerations | indent 8 }} | ||
{{- end }} | ||
{{- if or .Values.logs.metadata.statefulset.affinity .Values.fluentd.logs.statefulset.podAntiAffinity }} | ||
affinity: | ||
{{- if .Values.logs.metadata.statefulset.affinity }} | ||
{{ toYaml .Values.logs.metadata.statefulset.affinity | indent 8 }} | ||
{{- end }} | ||
{{- if eq .Values.logs.metadata.statefulset.podAntiAffinity "hard"}} | ||
podAntiAffinity: | ||
requiredDuringSchedulingIgnoredDuringExecution: | ||
- labelSelector: | ||
matchExpressions: | ||
- key: app | ||
operator: In | ||
values: | ||
- {{ template "sumologic.labels.app.logs" . }} | ||
- {{ template "sumologic.labels.app.metrics" . }} | ||
- key: app | ||
operator: In | ||
values: | ||
- prometheus-operator-prometheus | ||
topologyKey: "kubernetes.io/hostname" | ||
{{else if eq .Values.logs.metadata.statefulset.podAntiAffinity "soft"}} | ||
podAntiAffinity: | ||
preferredDuringSchedulingIgnoredDuringExecution: | ||
- weight: 100 | ||
podAffinityTerm: | ||
labelSelector: | ||
matchExpressions: | ||
- key: app | ||
operator: In | ||
values: | ||
- {{ template "sumologic.labels.app.logs.pod" . }} | ||
- {{ template "sumologic.labels.app.metrics.pod" . }} | ||
- key: app | ||
operator: In | ||
values: | ||
- prometheus-operator-prometheus | ||
topologyKey: "kubernetes.io/hostname" | ||
{{- end }} | ||
{{- end }} | ||
volumes: | ||
- name: config-volume | ||
configMap: | ||
name: {{ template "sumologic.metadata.name.logs.configmap" . }} | ||
{{- if .Values.logs.extraVolumes }} | ||
{{ toYaml .Values.logs.extraVolumes | indent 6 }} | ||
{{- end }} | ||
securityContext: | ||
{{- toYaml .Values.otelcol.securityContext | nindent 8 }} | ||
{{- if .Values.logs.metadata.statefulset.priorityClassName }} | ||
priorityClassName: {{ .Values.logs.metadata.statefulset.priorityClassName | quote }} | ||
{{- end }} | ||
containers: | ||
- name: otelcol | ||
image: {{ .Values.otelcol.statefulset.image.repository }}:{{ .Values.otelcol.statefulset.image.tag }} | ||
imagePullPolicy: {{ .Values.otelcol.statefulset.image.pullPolicy }} | ||
args: | ||
- --log-level={{ .Values.otelcol.metadata.logs.logLevel }} | ||
- --config=/etc/otel/config.yaml | ||
resources: | ||
{{- toYaml .Values.logs.metadata.statefulset.resources | nindent 10 }} | ||
{{- if .Values.logs.metadata.statefulset.containers.otelcol.securityContext }} | ||
securityContext: | ||
{{- toYaml .Values.logs.metadata.statefulset.containers.otelcol.securityContext | nindent 10 }} | ||
{{- end }} | ||
ports: | ||
{{- if .Values.sumologic.traces.enabled }} | ||
- name: zipkin-write | ||
containerPort: 9411 | ||
protocol: TCP | ||
{{- end }} | ||
- name: fluent-bit | ||
containerPort: 24321 | ||
protocol: TCP | ||
livenessProbe: | ||
httpGet: | ||
path: / | ||
port: 13133 # Health Check extension default port. | ||
readinessProbe: | ||
httpGet: | ||
path: / | ||
port: 13133 # Health Check extension default port. | ||
volumeMounts: | ||
- name: config-volume | ||
mountPath: /etc/otel/config.yaml | ||
subPath: config.yaml | ||
{{- if .Values.logs.extraVolumeMounts }} | ||
{{ toYaml .Values.logs.extraVolumeMounts | indent 8 }} | ||
{{- end }} | ||
env: | ||
{{- $ctx := .Values -}} | ||
{{ include "kubernetes.sources.envs" (dict "Context" $ctx "Type" "logs")}} | ||
{{- if .Values.logs.extraEnvVars }} | ||
{{ toYaml .Values.logs.extraEnvVars | nindent 8 }} | ||
{{- end }} | ||
{{- end }} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This also needs to be changed to take into account the provider. As of now I'm seeing