Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add fluentd.apiServerUrl property #2077

Merged
merged 2 commits into from
Feb 3, 2022
Merged

feat: add fluentd.apiServerUrl property #2077

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
2b307dc
test(helm): add tests for events config
andrzej-stencel Feb 3, 2022
6413f7b
feat: add `fluentd.apiServerUrl` property
andrzej-stencel Feb 3, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

- feat: add CRI support to experimental otelcol log collector[#2017][#2017]
- docs(readme): add support for AKS 1.22 [#2075][#2075]
- feat: add `fluentd.apiServerUrl` property [#2077]

### Changed

Expand All @@ -30,6 +31,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
[#2063]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2063
[#2075]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2075
[#2017]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2017
[#2077]: https://github.com/SumoLogic/sumologic-kubernetes-collection/pull/2077
[Unreleased]: https://github.com/SumoLogic/sumologic-kubernetes-collection/compare/v2.4.1...main

## [v2.4.1][v2_4_1]
Expand Down
1 change: 1 addition & 0 deletions deploy/helm/sumologic/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ The following table lists the configurable parameters of the Sumo Logic chart an
| `fluentd.metadata.addService` | Option to control the enrichment of logs and metrics with `service` metadata. | `true` |
| `fluentd.metadata.annotation_match` | Option to control capturing of annotations by metadata filter plugin. | `['sumologic\.com.*']` |
| `fluentd.metadata.apiGroups` | List of supported kubernetes API groups. | `['apps/v1']` |
| `fluentd.metadata.apiServerUrl` | Option to specify custom API server URL instead of the default, that is taken from KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables. Example: `"https://kubernetes.default.svc:443"`. | `""` |
| `fluentd.metadata.coreApiVersions` | List of supported kubernetes API versions. | `['v1']` |
| `fluentd.metadata.cacheSize` | Option to control the enabling of metadata filter plugin cache_size. | `10000` |
| `fluentd.metadata.cacheTtl` | Option to control the enabling of metadata filter plugin cache_ttl (in seconds). | `7200` |
Expand Down
6 changes: 6 additions & 0 deletions deploy/helm/sumologic/conf/events/events.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,18 @@
deploy_namespace {{ $.Release.Namespace }}
resource_name {{ $name }}
api_version {{ $version }}
{{- if $.Values.fluentd.apiServerUrl }}
kubernetes_url {{ $.Values.fluentd.apiServerUrl | quote }}
{{- end }}
Comment on lines +8 to +10
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could convert it to template evnetually

</source>
{{- end }}
{{- else }}
<source>
@type events
deploy_namespace {{ .Release.Namespace }}
{{- if .Values.fluentd.apiServerUrl }}
kubernetes_url {{ .Values.fluentd.apiServerUrl | quote }}
{{- end }}
</source>
{{- end }}
# Prevent fluentd from handling records containing its own logs.
Expand Down
5 changes: 4 additions & 1 deletion deploy/helm/sumologic/conf/logs/fluentd/logs.enhance.k8s.metadata.filter.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,7 @@ api_groups {{ join "," .Values.fluentd.metadata.apiGroups }}
data_type logs
add_owners {{ .Values.fluentd.metadata.addOwners }}
add_service {{ .Values.fluentd.metadata.addService }}
cache_refresh_apiserver_request_delay {{ .Values.fluentd.metadata.cacheRefreshApiserverRequestDelay | quote }}
cache_refresh_apiserver_request_delay {{ .Values.fluentd.metadata.cacheRefreshApiserverRequestDelay | quote }}
{{- if .Values.fluentd.apiServerUrl }}
kubernetes_url {{ .Values.fluentd.apiServerUrl | quote }}
{{- end }}
5 changes: 4 additions & 1 deletion deploy/helm/sumologic/conf/logs/fluentd/logs.kubernetes.metadata.filter.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,7 @@ client_key {{ .Values.fluentd.logs.containers.k8sMetadataFilter.clientKey | quot
bearer_token_file {{ .Values.fluentd.logs.containers.k8sMetadataFilter.bearerTokenFile | quote }}
cache_size {{ .Values.fluentd.metadata.cacheSize | quote }}
cache_ttl {{ .Values.fluentd.metadata.cacheTtl | quote }}
tag_to_kubernetes_name_regexp '.+?\.containers\.(?<pod_name>[^_]+)_(?<namespace>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$'
tag_to_kubernetes_name_regexp '.+?\.containers\.(?<pod_name>[^_]+)_(?<namespace>[^_]+)_(?<container_name>.+)-(?<docker_id>[a-z0-9]{64})\.log$'
{{- if .Values.fluentd.apiServerUrl }}
kubernetes_url {{ .Values.fluentd.apiServerUrl | quote }}
{{- end }}
3 changes: 3 additions & 0 deletions deploy/helm/sumologic/conf/metrics/fluentd/metrics.conf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,9 @@
api_groups {{ join "," .Values.fluentd.metadata.apiGroups }}
add_owners {{ .Values.fluentd.metadata.addOwners }}
add_service {{ .Values.fluentd.metadata.addService }}
{{- if .Values.fluentd.apiServerUrl }}
kubernetes_url {{ .Values.fluentd.apiServerUrl | quote }}
{{- end }}
</filter>
{{- .Values.fluentd.metrics.extraFilterPluginConf | nindent 4 }}
<filter prometheus.metrics**>
Expand Down
4 changes: 4 additions & 0 deletions deploy/helm/sumologic/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -361,6 +361,10 @@ fluentd:
addOwners: true
## Option to control the enrichment of logs and metrics with `service` metadata.
addService: true
## Option to specify custom API server URL instead of the default,
## that is taken from KUBERNETES_SERVICE_HOST and KUBERNETES_SERVICE_PORT environment variables.
## Example: "https://kubernetes.default.svc:443".
apiServerUrl: ""

logs:
enabled: true
Expand Down
3 changes: 3 additions & 0 deletions tests/helm/events/config.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
#!/bin/bash

export TEST_TEMPLATE="templates/events/fluentd/configmap.yaml"
2 changes: 2 additions & 0 deletions tests/helm/events/static/apiServerUrl.input.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
fluentd:
apiServerUrl: "https://kubernetes.default.svc:443"
71 changes: 71 additions & 0 deletions tests/helm/events/static/apiServerUrl.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
---
# Source: sumologic/templates/events/fluentd/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: RELEASE-NAME-sumologic-fluentd-events
labels:
app: RELEASE-NAME-sumologic-fluentd-events
chart: "sumologic-%CURRENT_CHART_VERSION%"
release: "RELEASE-NAME"
heritage: "Helm"
data:
fluent.conf: |-
@include events.conf
events.conf: |
<source>
@type events
deploy_namespace sumologic
kubernetes_url "https://kubernetes.default.svc:443"
</source>
# Prevent fluentd from handling records containing its own logs.
<match fluentd.**>
@type null
</match>
# expose the Fluentd metrics to Prometheus
<source>
@type prometheus
metrics_path /metrics
port 24231
</source>
<source>
@type prometheus_output_monitor
</source>
<source>
@type http
port 9880
bind 0.0.0.0
</source>
<match kubernetes.**>
@type copy
<store>
@type sumologic
@id sumologic.endpoint.events
sumo_client "k8s_2.4.0"
endpoint "#{ENV['SUMO_ENDPOINT_DEFAULT_EVENTS_SOURCE']}"
data_type logs
disable_cookies true
verify_ssl "true"
proxy_uri ""
compress "true"
compress_encoding "gzip"
<buffer>
@type file
path /fluentd/buffer/events
@include buffer.output.conf
</buffer>
</store>
</match>
<system>
log_level info
</system>
buffer.output.conf: |
compress "gzip"
flush_interval "5s"
flush_thread_count "8"
chunk_limit_size "1m"
total_limit_size "128m"
queued_chunks_limit_size "128"
overflow_action drop_oldest_chunk
retry_max_interval "10m"
retry_forever "true"
2 changes: 2 additions & 0 deletions tests/helm/events/static/apiServerUrl_null.input.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
fluentd:
apiServerUrl: null
70 changes: 70 additions & 0 deletions tests/helm/events/static/apiServerUrl_null.output.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
---
# Source: sumologic/templates/events/fluentd/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: RELEASE-NAME-sumologic-fluentd-events
labels:
app: RELEASE-NAME-sumologic-fluentd-events
chart: "sumologic-%CURRENT_CHART_VERSION%"
release: "RELEASE-NAME"
heritage: "Helm"
data:
fluent.conf: |-
@include events.conf
events.conf: |
<source>
@type events
deploy_namespace sumologic
</source>
# Prevent fluentd from handling records containing its own logs.
<match fluentd.**>
@type null
</match>
# expose the Fluentd metrics to Prometheus
<source>
@type prometheus
metrics_path /metrics
port 24231
</source>
<source>
@type prometheus_output_monitor
</source>
<source>
@type http
port 9880
bind 0.0.0.0
</source>
<match kubernetes.**>
@type copy
<store>
@type sumologic
@id sumologic.endpoint.events
sumo_client "k8s_2.4.0"
endpoint "#{ENV['SUMO_ENDPOINT_DEFAULT_EVENTS_SOURCE']}"
data_type logs
disable_cookies true
verify_ssl "true"
proxy_uri ""
compress "true"
compress_encoding "gzip"
<buffer>
@type file
path /fluentd/buffer/events
@include buffer.output.conf
</buffer>
</store>
</match>
<system>
log_level info
</system>
buffer.output.conf: |
compress "gzip"
flush_interval "5s"
flush_thread_count "8"
chunk_limit_size "1m"
total_limit_size "128m"
queued_chunks_limit_size "128"
overflow_action drop_oldest_chunk
retry_max_interval "10m"
retry_forever "true"
3 changes: 3 additions & 0 deletions tests/helm/events/static/default.input.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
fluentd:
events:
enabled: true
Loading