feat: add OtelCol as alternative logs metadata provider #1697
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
sumo-drosiek
commented
Jul 19, 2021
sumo-drosiek
force-pushed
the
drosiek-otc
branch
from
e8bc01a
to
57433c1
Compare
pmalek-sumo
reviewed
Jul 19, 2021
sumo-drosiek
force-pushed
the
drosiek-otc
branch
2 times, most recently
from
5a151fe
to
afad9f3
Compare
sumo-drosiek
changed the title
sumo-drosiek
commented
Aug 6, 2021
sumo-drosiek
commented
Aug 6, 2021
kkujawa-sumo
approved these changes
Sep 1, 2021
There was a problem hiding this comment.
Logs are coming to Sumo, hpa works.
I think that only description of new parameters in values.yaml needs to be added to documentation.
Test with following values.yaml:
sumologic:
accessId: *******
accessKey: ****************
endpoint: ""
clusterName: "kubernetesvagrant"
logs:
provider: otelcol
logs:
## Option to turn autoscaling on for logs and specify params for HPA.
## Autoscaling needs metrics-server to access cpu metrics.
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 50
# targetMemoryUtilizationPercentage: 50
metrics-server:
## Set the enabled flag to true for enabling metrics-server.
## This is required before enabling fluentd autoscaling unless you have an existing metrics-server in the cluster.
enabled: true
kkujawa-sumo
reviewed
Sep 1, 2021
|
Add change analogic to 55793c6 |
sumo-drosiek
commented
Sep 7, 2021
deploy/helm/sumologic/values.yaml
Outdated
| actions: | ||
| - action: extract | ||
| key: fluent.tag | ||
| pattern: ^containers\.var\.log\.pods\.(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]{36})\.(?P<container_name>[^\._]+)\.(?P<run_id>\d+)\.log$ |
There was a problem hiding this comment.
Update to support /var/log/containers to avoid changing fluent-bit config
There was a problem hiding this comment.
and update metadata enrichment to enrich using pod name and namespace name
sumo-drosiek
commented
Sep 7, 2021
deploy/helm/sumologic/values.yaml
Outdated
Comment on lines
2505
to
2751
| pod_association: | ||
| - from: resource_attribute | ||
| name: k8s.pod.uid ## Source processor adds Sumo Logic related metadata |
There was a problem hiding this comment.
change to
sumologic-kubernetes-collection/deploy/helm/sumologic/values.yaml
Lines 2583 to 2584 in 48492ff
k8s.pod.name and k8s.namespace.name resource attributes are required
|
Update |
pmalek-sumo
reviewed
Sep 7, 2021
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
…lpers Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
Signed-off-by: Dominik Rosiek <drosiek@sumologic.com>
andrzej-stencel
force-pushed
the
drosiek-otc
branch
from
0823d19
to
ac2127d
Compare
It is not going to be used, as the OtelCol metadata provider will be using the same `sumologic.endpoint` property as Fluentd.
It is not going to be used. OtelCol will be using the same endpoint as Fluentd is using now, as opposed to the new open endpoint, so the extension is not needed.
pmalek-sumo
reviewed
Sep 8, 2021
| {{ toYaml .Values.logs.statefulset.podAnnotations | indent 8 }} | ||
| {{- end }} | ||
| labels: | ||
| app: {{ template "sumologic.labels.app.logs.pod" . }} |
There was a problem hiding this comment.
This also needs to be changed to take into account the provider. As of now I'm seeing
Name: collection-sumologic-otelcol-logs-0
Namespace: sumologic
Priority: 0
Node: sumologic-kubernetes-collection/10.0.2.15
Start Time: Wed, 08 Sep 2021 12:08:51 +0200
Labels: app=collection-sumologic-fluentd-logs
chart=sumologic-2.2.0-dev.0
controller-revision-hash=collection-sumologic-otelcol-logs-56cf654b8
heritage=Helm
release=collection
statefulset.kubernetes.io/pod-name=collection-sumologic-otelcol-logs-0
...
0089ada
to
56bc8f9
Compare
andrzej-stencel
force-pushed
the
drosiek-otc
branch
from
56bc8f9
to
75bd207
Compare
These templates are currently only used by OtelCol and not by Fluentd, so the additional logic is not needed at the moment.
andrzej-stencel
force-pushed
the
drosiek-otc
branch
from
b0da9e5
to
eda1093
Compare
andrzej-stencel
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add otelcol support for logs pipeline
Testing performed