Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: adjust queueing and batching configurarion for logs, OTC accord… #2479

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- chore: upgrade sumologic terraform provider to 2.18 [#2399]
- chore: update kubernetes-tools to 2.12.0 [#2472]
- feat(otelcol/metrics): adjust metric otelcol configuration [#2474]
- feat(otelcol/metrics/logs): adjust metric otelcol configuration [#2474], [#2479]

[#2466]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2466
[#2399]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2399
[#2472]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2472
[#2474]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2474
[#2479]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2479
[Unreleased]: https://github.com/SumoLogic/sumologic-kubernetes-collection/compare/v2.14.1...main

## [v2.14.1]
Expand Down
10 changes: 7 additions & 3 deletions deploy/helm/sumologic/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3979,7 +3979,7 @@ metadata:
## Number of spans after which a batch will be sent regardless of time
send_batch_size: 1_024
## Time duration after which a batch will be sent regardless of size
timeout: 30s
timeout: 1s
## Configuration for Kubernetes Processor
## ref: https://github.com/SumoLogic/sumologic-otel-collector/tree/main/pkg/processor/k8sprocessor
k8s_tagger:
Expand Down Expand Up @@ -4151,6 +4151,8 @@ metadata:
sending_queue:
enabled: true
persistent_storage_enabled: '{{ .Values.metadata.persistence.enabled }}'
num_consumers: 10
queue_size: 10_000
sumologic/systemd:
log_format: json
json_logs:
Expand All @@ -4169,6 +4171,8 @@ metadata:
sending_queue:
enabled: true
persistent_storage_enabled: '{{ .Values.metadata.persistence.enabled }}'
num_consumers: 10
queue_size: 10_000

processors:
## Common processors
Expand All @@ -4192,9 +4196,9 @@ metadata:
## The batch processor accepts spans and places them into batches grouped by node and resource
batch:
## Number of spans after which a batch will be sent regardless of time
send_batch_size: 256
send_batch_size: 1_024
## Time duration after which a batch will be sent regardless of size
timeout: 5s
timeout: 1s

## Containers related processors
filter/include_fluent_tag_containers:
Expand Down
8 changes: 6 additions & 2 deletions tests/helm/metadata_logs_otc/static/basic.output.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ data:
log_format: json
sending_queue:
enabled: true
num_consumers: 10
persistent_storage_enabled: true
queue_size: 10000
source_category: '%{_sourceCategory}'
source_host: '%{_sourceHost}'
source_name: '%{_sourceName}'
Expand All @@ -34,7 +36,9 @@ data:
log_format: json
sending_queue:
enabled: true
num_consumers: 10
persistent_storage_enabled: true
queue_size: 10000
source_category: '%{_sourceCategory}'
source_host: '%{_sourceHost}'
source_name: '%{_sourceName}'
Expand Down Expand Up @@ -83,8 +87,8 @@ data:
- action: delete
key: fluent.tag
batch:
send_batch_size: 256
timeout: 5s
send_batch_size: 1024
timeout: 1s
filter/exclude_kubelet:
logs:
exclude:
Expand Down
8 changes: 6 additions & 2 deletions tests/helm/metadata_logs_otc/static/templates.output.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@ data:
log_format: json
sending_queue:
enabled: true
num_consumers: 10
persistent_storage_enabled: true
queue_size: 10000
source_category: '%{_sourceCategory}'
source_host: '%{_sourceHost}'
source_name: '%{_sourceName}'
Expand All @@ -34,7 +36,9 @@ data:
log_format: json
sending_queue:
enabled: true
num_consumers: 10
persistent_storage_enabled: true
queue_size: 10000
source_category: '%{_sourceCategory}'
source_host: '%{_sourceHost}'
source_name: '%{_sourceName}'
Expand Down Expand Up @@ -83,8 +87,8 @@ data:
- action: delete
key: fluent.tag
batch:
send_batch_size: 256
timeout: 5s
send_batch_size: 1024
timeout: 1s
filter/exclude_kubelet:
logs:
exclude:
Expand Down