Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(kubernetes_logs source): Add configurable delay to deletion of k8s metadata #10031

Merged
merged 8 commits into from
Nov 16, 2021
Merged

feat(kubernetes_logs source): Add configurable delay to deletion of k8s metadata #10031

merged 8 commits into from
Nov 16, 2021

Conversation

spencergilbert
Copy link
Contributor

  • feat(kubernetes_logs source): Add configurable delay to deletion of k8s metadata
  • docs: Document delay_deletion_ms option

Closes #8345

I found this difficult to reliably test in an automated way, if anyone has suggestions please let me know.

spencergilbert and others added 2 commits November 12, 2021 13:00
@spencergilbert
feat(kubernetes_logs source): Add configurable delay to deletion of k…
cf14086 
…8s metadata

Signed-off-by: Spencer Gilbert <spencer.gilbert@gmail.com>
@spencergilbert
docs: Document delay_deletion_ms option
91e0c3c 
Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
@spencergilbert spencergilbert added ci-condition: k8s e2e all targets Run Kubernetes E2E test suite for all targets (instead of just the essential subset) ci-condition: k8s e2e tests enable Run Kubernetes E2E test suite for this PR source: kubernetes_logs Anything `kubernetes_logs` source related labels Nov 15, 2021
@spencergilbert spencergilbert self-assigned this Nov 15, 2021
@netlify
Copy link

netlify bot commented Nov 15, 2021 •
edited
Loading

✔️ Deploy Preview for vector-project ready!

🔨 Explore the source changes: 08aa1ef

🔍 Inspect the deploy log: https://app.netlify.com/sites/vector-project/deploys/6193f9e46ef7510008440373

😎 Browse the preview: https://deploy-preview-10031--vector-project.netlify.app/reports/lighthouse

@github-actions github-actions bot added domain: external docs Anything related to Vector's external, public documentation domain: sources Anything related to the Vector's sources labels Nov 15, 2021
@spencergilbert
Trigger k8s e2e checks
6341479 
Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
StephenWakely
StephenWakely approved these changes Nov 16, 2021
View reviewed changes
Copy link
Contributor

@StephenWakely StephenWakely left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. Extra points for remembering the docs!

@spencergilbert
Copy link
Contributor Author

Nice. Extra points for remembering the docs!

I mostly always sometimes remember docs!

jszwedko
jszwedko reviewed Nov 16, 2021
View reviewed changes
src/sources/kubernetes_logs/mod.rs Show resolved Hide resolved
website/cue/reference/components/sources/kubernetes_logs.cue Outdated
@@ -263,6 +263,15 @@ components: sources: kubernetes_logs: {
unit: "milliseconds"
}
}
delay_deletion_ms: {
common: false
description: "Delay between receiving a `DELETE` event and removing any related metadata Vector has stored. This controls how quickly Vector will remove metadata for resources that have been removed from Kubernetes, a longer delay will allow Vector to continue processing and enriching logs after the source Pod has been deleted."
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might word wrap this 😄

Just to clarify, what happens if the metadata is deleted before Vector has finished processing the log files for those pods?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

😆 I expected cue fmt to do so but.... I figured it would either wrap it or be angry about my manual wrapping - I can break it up. I'll update the description to mention what happens if things have been deleted.

@github-actions
Copy link

Soak Test Results

Baseline: b53f2bb
Comparison: f41684c
Total Vector CPUs: 4

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 10.27Mi 10.47Mi 10.49Mi 10.49Mi -0.30 -1.61

datadog_agent_remap_datadog_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 19.20Mi 19.25Mi 19.25Mi 19.25Mi -0.14 -1.12

fluent_elasticsearch

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

fluent_remap_aws_firehose

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 36.75Mi 36.90Mi 36.97Mi 36.98Mi 0.67 -0.26

splunk_hec_route_s3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.54Mi 5.98Mi 6.05Mi 6.09Mi -0.10 -0.57

splunk_transforms_splunk3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 2.54Mi 2.75Mi 2.77Mi 2.82Mi -0.15 -1.06
comparison 2.61Mi 2.75Mi 2.81Mi 2.81Mi 0.01 0.08

syslog_humio_logs

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

syslog_log2metric_humio_metrics

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

syslog_log2metric_splunk_hec_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 4.97Mi 5.01Mi 5.01Mi 5.01Mi -1.00 -0.64

syslog_loki

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 3.70Mi 4.15Mi 4.26Mi 4.26Mi 0.75 -0.31

syslog_regex_logs2metric_ddmetrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 3.50Mi 3.73Mi 3.81Mi 3.82Mi 1.35 0.57
comparison 3.77Mi 3.80Mi 3.81Mi 3.81Mi -0.08 -1.49

syslog_splunk_hec_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 7.28Mi 7.30Mi 7.30Mi 7.30Mi 1.02 0.09

@spencergilbert
docs: Clarify delay_deletion_ms option
4674935 
Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
jszwedko
jszwedko approved these changes Nov 16, 2021
View reviewed changes
Copy link
Member

@jszwedko jszwedko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good! Thanks for adding the note to the documentation about what happens if the metadata is not present.

website/cue/reference/components/sources/kubernetes_logs.cue Outdated Show resolved Hide resolved
@spencergilbert
Fix typo
6151286 
Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
@spencergilbert spencergilbert enabled auto-merge (squash) November 16, 2021 15:58
@github-actions
Copy link

Soak Test Results

Baseline: b53f2bb
Comparison: 6151286
Total Vector CPUs: 4

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 10.48Mi 10.51Mi 10.52Mi 10.52Mi 0.41 0.17

datadog_agent_remap_datadog_logs

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

fluent_elasticsearch

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

fluent_remap_aws_firehose

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 36.01Mi 37.52Mi 37.56Mi 37.57Mi -6.85 59.46

splunk_hec_route_s3

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

splunk_transforms_splunk3

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

syslog_humio_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 7.02Mi 7.20Mi 7.20Mi 7.20Mi -0.34 -1.27

syslog_log2metric_humio_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 4.94Mi 5.01Mi 5.01Mi 5.01Mi -0.56 -1.19

syslog_log2metric_splunk_hec_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 5.25Mi 5.27Mi 5.27Mi 5.27Mi 0.12 -1.43
comparison 5.06Mi 5.07Mi 5.07Mi 5.07Mi -0.29 -0.76

syslog_loki

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 4.05Mi 4.23Mi 4.34Mi 4.35Mi 0.53 0.72

syslog_regex_logs2metric_ddmetrics

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

syslog_splunk_hec_logs

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

@spencergilbert
cue fmt passes locally but not in CI, again
dbaec99 
Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
@github-actions
Copy link

Soak Test Results

Baseline: cf51ae0
Comparison: dbaec99
Total Vector CPUs: 4

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 10.24Mi 10.60Mi 10.60Mi 10.61Mi -1.17 0.91

datadog_agent_remap_datadog_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 19.16Mi 19.21Mi 19.21Mi 19.22Mi 0.06 -0.67

fluent_elasticsearch

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

fluent_remap_aws_firehose

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

splunk_hec_route_s3

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

splunk_transforms_splunk3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 2.40Mi 2.66Mi 2.68Mi 2.68Mi -0.77 -0.38
comparison 2.54Mi 2.70Mi 2.75Mi 2.76Mi 0.06 -1.13

syslog_humio_logs

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

syslog_log2metric_humio_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 5.09Mi 5.11Mi 5.11Mi 5.11Mi -0.04 -0.38

syslog_log2metric_splunk_hec_metrics

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

syslog_loki

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

syslog_regex_logs2metric_ddmetrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 3.84Mi 3.86Mi 3.86Mi 3.87Mi 0.39 -0.61

syslog_splunk_hec_logs

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

@spencergilbert
fix style
08aa1ef 
Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
@github-actions
Copy link

Soak Test Results

Baseline: cf51ae0
Comparison: 08aa1ef
Total Vector CPUs: 4

What follows is a statistical summary of the soak captures between the SHAs given above. Units are bytes/second/CPU, except for 'skewness' and 'kurtosis'. Higher numbers in 'comparison' is generally better. Higher skewness or kurtosis numbers indicate a lack of consistency in behavior, making predictions of fitness in the field challenging.

datadog_agent_remap_blackhole

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 10.82Mi 10.84Mi 10.84Mi 10.85Mi -0.13 -0.41
comparison 10.57Mi 10.59Mi 10.59Mi 10.59Mi -0.19 -0.29

datadog_agent_remap_datadog_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 19.00Mi 19.03Mi 19.03Mi 19.04Mi 0.03 -0.75

fluent_elasticsearch

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

fluent_remap_aws_firehose

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 40.79Mi 40.97Mi 41.02Mi 41.03Mi 0.60 -0.68
comparison 38.73Mi 38.87Mi 38.91Mi 38.91Mi -0.04 -0.97

splunk_hec_route_s3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 5.68Mi 6.00Mi 6.07Mi 6.11Mi 0.06 -0.72

splunk_transforms_splunk3

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 2.43Mi 2.71Mi 2.74Mi 2.76Mi 0.12 -1.13
comparison 2.53Mi 2.70Mi 2.73Mi 2.73Mi 0.05 -0.92

syslog_humio_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 7.16Mi 7.24Mi 7.24Mi 7.24Mi -0.14 -1.65

syslog_log2metric_humio_metrics

| |
| --- | --- | --- | --- | --- | --- | --- |
| |

syslog_log2metric_splunk_hec_metrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 4.90Mi 4.92Mi 4.93Mi 4.93Mi -0.76 0.84

syslog_loki

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 4.06Mi 4.47Mi 4.53Mi 4.53Mi -0.10 -1.47

syslog_regex_logs2metric_ddmetrics

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
comparison 3.86Mi 3.87Mi 3.87Mi 3.87Mi -0.40 -0.63

syslog_splunk_hec_logs

EXPERIMENT VALUE_min VALUE_p90 VALUE_p99 VALUE_max VALUE_skewness VALUE_kurtosis
baseline 7.15Mi 7.20Mi 7.21Mi 7.21Mi -0.69 -1.25

@spencergilbert spencergilbert merged commit 24ea6de into master Nov 16, 2021
@spencergilbert spencergilbert deleted the spencer/k8s-logs-delay-deletion branch November 16, 2021 21:21
jdrouet pushed a commit that referenced this pull request Nov 18, 2021
@spencergilbert @jdrouet
feat(kubernetes_logs source): Add configurable delay to deletion of k…
53b78f6 
…8s metadata (#10031)

* feat(kubernetes_logs source): Add configurable delay to deletion of k8s metadata

Signed-off-by: Spencer Gilbert <spencer.gilbert@gmail.com>

* docs: Document delay_deletion_ms option

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

* Trigger k8s e2e checks

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>

* docs: Clarify delay_deletion_ms option

Signed-off-by: Spencer Gilbert <spencer.gilbert@datadoghq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@StephenWakely StephenWakely StephenWakely approved these changes

@jszwedko jszwedko jszwedko approved these changes

@JeanMertz JeanMertz Awaiting requested review from JeanMertz

Assignees

@spencergilbert spencergilbert

Labels
ci-condition: k8s e2e all targets Run Kubernetes E2E test suite for all targets (instead of just the essential subset) ci-condition: k8s e2e tests enable Run Kubernetes E2E test suite for this PR domain: external docs Anything related to Vector's external, public documentation domain: sources Anything related to the Vector's sources source: kubernetes_logs Anything `kubernetes_logs` source related
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

"annotate event with pod metadata" on removed pod
3 participants
@spencergilbert @StephenWakely @jszwedko