Skip to content

Commit

Permalink
feat(otellogs)!: add config.merge option
Browse files Browse the repository at this point in the history
  • Loading branch information
Mikołaj Świątek committed Nov 28, 2022
1 parent 8786fba commit 587cad7
Show file tree
Hide file tree
Showing 9 changed files with 123 additions and 14 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- feat(logs)!: simplify metadata configuration [#2626]
- feat(metrics)!: simplify metadata configuration [#2622]
- feat(events)!: add config.merge option [#2643]
- feat(otellogs)!: add config.merge option [#2652]

### Changed

Expand Down Expand Up @@ -88,6 +89,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
[#2622]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2622
[#2643]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2643
[#2654]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2654
[#2652]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2652
[Unreleased]: https://github.com/SumoLogic/sumologic-kubernetes-collection/compare/v2.17.0...main
[telegraf_operator_comapare_1.3.5_and_1.3.10]: https://github.com/influxdata/helm-charts/compare/telegraf-operator-1.3.5...telegraf-operator-1.3.10
[cert-manager-1.4]: https://github.com/cert-manager/cert-manager/releases/tag/v1.4.0
Expand Down
9 changes: 7 additions & 2 deletions deploy/helm/sumologic/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -331,8 +331,13 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `otelcol.config.processors.batch.send_batch_max_size` | Sets the maximum allowed size of a batch (in number of spans). Use with caution, setting too large value might cause 413 Payload Too Large errors. | `512` |
| `otelcol.logLevelFilter` | Do not send otelcol logs if `true`. | `false` |
| `otelgateway.deployment.nodeSelector` | Node selector for otelgateway deployment. [See docs/best-practices.md for more information.](/docs/best-practices.md) | `{}` |
| `otellogs.daemonset.nodeSelector` | Node selector for otellogs daemonset. [See docs/best-practices.md for more information.](/docs/best-practices.md) | `{}` |
| `otellogs.daemonset.priorityClassName` | Priority class name for OpenTelemetry Agent log pods. | If not provided then set to `RELEASE-NAME-sumologic-priorityclass`. |
| `otellogs.image.repository` | Image repository for otelcol docker container. | `public.ecr.aws/sumologic/sumologic-otel-collector` |
| `otellogs.image.tag` | Image tag for otelcol docker container. | `0.54.0-sumo-0` |
| `otellogs.image.pullPolicy` | Image pullPolicy for otelcol docker container. | `IfNotPresent` |
| `otellogs.logLevel` | Log level for the OpenTelemtry Collector. Can be `debug`, `info`, `warn`, `error`, `dpanic`, `panic`, `fatal`. | `info` | | `{}` |
| `otellogs.config.merge` | Configuration for log collector otelcol, merged with defaults. See also https://github.com/SumoLogic/sumologic-otel-collector/blob/main/docs/configuration.md. | {} |
| `otellogs.config.override` | Configuration for log collector otelcol, replaces defaults. See also https://github.com/SumoLogic/sumologic-otel-collector/blob/main/docs/configuration.md. | {} |
| `otellogs.daemonset` | OpenTelemetry Collector Daemonset customization options. See values.yaml for more details. | See [values.yaml] |
| `metadata.image.repository` | Image repository for otelcol docker container. | `public.ecr.aws/sumologic/sumologic-otel-collector` |
| `metadata.image.tag` | Image tag for otelcol docker container. | `0.0.18` |
| `metadata.image.pullPolicy` | Image pullPolicy for otelcol docker container. | `IfNotPresent` |
Expand Down
14 changes: 10 additions & 4 deletions deploy/helm/sumologic/templates/logs/collector/otelcol/configmap.yaml
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,13 @@
{{- if eq (include "logs.collector.otelcol.enabled" .) "true" }}
{{ $baseConfig := (tpl (.Files.Get "conf/logs/collector/otelcol/config.yaml") .) | fromYaml }}
{{ $mergeConfig := .Values.otellogs.config.merge }}
{{ $overrideConfig := .Values.otellogs.config.override }}
{{ $finalConfig := "" }}
{{ if $overrideConfig }}
{{ $finalConfig = $overrideConfig }}
{{ else }}
{{ $finalConfig = mergeOverwrite $baseConfig $mergeConfig }}
{{ end }}
apiVersion: v1
kind: ConfigMap
metadata:
Expand All @@ -7,9 +16,6 @@ metadata:
app: {{ template "sumologic.labels.app.logs.collector.configmap" . }}
{{- include "sumologic.labels.common" . | nindent 4 }}
data:
{{- $baseConfig := (tpl (.Files.Get "conf/logs/collector/otelcol/config.yaml") .) | fromYaml -}}
{{- $overrideConfig := .Values.otellogs.config.override -}}
{{- $finalConfig := mergeOverwrite $baseConfig $overrideConfig }}
config.yaml: |
{{- $finalConfig | toYaml | nindent 4 }}
{{- $finalConfig | toYaml | nindent 4 }}
{{- end }}
21 changes: 16 additions & 5 deletions deploy/helm/sumologic/values.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4151,11 +4151,6 @@ otelevents:
# readOnly: true

## Configure log collection with Otelcol
## This is currently highly experimental and not recommended unless you really know what you're doing.
## In particular, this configuration currently only covers container logs, and only supports docker-shim.
## Unlike the fluent-bit configuration, it also reads logs from /var/log/pods instead of /var/log/containers
## which is consistent with CRI, but may possibly cause issues on older K8s versions.
## This is an alpha feature, and may change in the near future.
otellogs:

## Metrics from Collector
Expand All @@ -4174,6 +4169,22 @@ otellogs:
logLevel: info

config:
## Directly alter the OT configuration. The value of this key should be a dictionary, that will
## be directly merged with the generated configuration, overriding existing values.
## For example:
# override:
# processors:
# batch:
# send_batch_size: 512
## will change the batch size of the pipeline.
##
## WARNING: This field is not subject to backwards-compatibility guarantees offered by the rest
## of this chart. It involves implementation details that may change even in minor versions.
## Use with caution, and consider opening an issue, so your customization can be added in a safer way.
merge: {}
## Completely override existing config and replace it with the contents of this value.
## The value of this key should be a dictionary, that will replace the normal configuration.
## This is an advanced feature, use with caution, and review the generated configuration first.
override: {}

daemonset:
Expand Down
2 changes: 1 addition & 1 deletion docs/opentelemetry-collector.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ versions while preserving the same end-to-end behaviour. Use of this field is di
the necessary customizations should be able to be achieved without touching the otel configuration
directly. Please open an issue if your use case requires the use of this field.

The `metadata.metrics.config.override` and `otellogs.config.override` keys can be used to provide configuration that will be completely
The `metadata.logs.config.override` and `otellogs.config.override` keys can be used to provide configuration that will be completely
replace the default configuration. As above, care must be taken not to depend on implementation details
that may change between minor releases of this Chart.

Expand Down
2 changes: 1 addition & 1 deletion .../helm/logs_otc/static/override.input.yaml → tests/helm/logs_otc/static/merge.input.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ fluent-bit:

otellogs:
config:
override:
merge:
receivers:
journald:
units:
Expand Down
0 ...helm/logs_otc/static/override.output.yaml → tests/helm/logs_otc/static/merge.output.yaml
View file
File renamed without changes.
2 changes: 1 addition & 1 deletion tests/integration/values/values_helm_default_otelcol_logs.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ fluent-bit:

otellogs:
config:
override:
merge:
receivers:
journald:
directory: /run/log/journal
Expand Down
85 changes: 85 additions & 0 deletions vagrant/opentelemetry-collector.yaml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: prometheus
namespace: sumologic
spec:
mode: statefulset
targetAllocator:
enabled: true
allocationStrategy: consistent-hashing
prometheusCR:
enabled: true
image: ghcr.io/open-telemetry/opentelemetry-operator/target-allocator:0.64.1
serviceAccount: prometheus-collector
replicas: 3
config: |
receivers:
prometheus:
config:
target_allocator:
endpoint: http://prometheus-targetallocator
interval: 30s
collector_id: ${POD_NAME}
scrape_configs:
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
processors:
memory_limiter:
check_interval: 1s
limit_percentage: 75
spike_limit_percentage: 15
batch:
send_batch_size: 10000
timeout: 10s
exporters:
logging:
service:
pipelines:
metrics:
receivers: [prometheus]
processors: []
exporters: [logging]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus-collector
namespace: sumologic
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: prometheus-collector
namespace: sumologic
rules:
- apiGroups:
- "*"
resources:
- "*"
verbs:
- "*"
- apiGroups:
- "monitoring.coreos.com"
resources:
- "*"
verbs:
- "*"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus-collector
namespace: sumologic
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: prometheus-collector
namespace: sumologic

0 comments on commit 587cad7

Please sign in to comment.