Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes #4368 #4421

Closed
Closed
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 57 additions & 0 deletions content/en/docs/collector/deployment/single-writer-principle.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
---
title: The Single Writer Principle
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
title: The Single Writer Principle
title: The Single Writer Principle

How can we better contextualize what this doc is about when it comes to Collector deployments? Think of a nav title that would fit the progression in the current section. Why should someone know about this principle?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, could this be a section in an existing doc rather than its own doc?

description: How to handle the single writer principle in Collector deployments
weight: 1
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alter the weight of the page so that it shows below "Gateway".

Suggested change
weight: 1
weight: 4

---

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As the doc is likely not self-contained, consider adding links from other topics.

The Single Writer Principle is a design principle that refers to maintaining a single logical writer specific data types
of events or data. The intention is to prevent concurrent write attempts from multiple writers
to the same data source, potentially leading to data corruption or data loss

## Context in OTLP
Collector deployments can involve multiple instance across different nodes in a distributed systems.
It is possible in this case for these instances to receive and process data from the same resources.
When multiple collectors write data to the same backend (such as Prometheus), the issues that arise are
a result of violating the single writer principle.

Duplicate samples for the same metric can be submitted.


## Potential Problems caused by multiple writers

An example scenario might be where multiple collectors are submitting the same samples, as
they are receiving traces from same resources. If each collector is scraped, it's possible to
observe inconsistent values for the same series due to data being collected by different collectors

Any inconsistency in the metric could be explained by a single series referencing each individaul collector, which

Check warning on line 26 in content/en/docs/collector/deployment/single-writer-principle.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (individaul)
would have varying values due to inconsistent time intervals.

For this reason uniqueness is a crucial concept in most metric systems.

## Detection

One way to identify this is by inspecting the series visually.
A series with significant gaps or jumps may be a clue that multiple collectors are submitting the same samples.
Finding a pattern of plateaus that seem to oscillate may indicate that the data is coming from two sources.

## Prevention
Implement a globally unique identifier for all metric data streams.

A potential approach is to use a unique identifier, such as a dynamic hostname in your configuration.

```yaml
connectors:
spanmetrics:

Check warning on line 44 in content/en/docs/collector/deployment/single-writer-principle.md

View workflow job for this annotation

GitHub Actions / SPELLING check

Unknown word (spanmetrics)
dimensions:
- name: http.method
- name: http.status_code
- name: k8s.namespace.name
- name: gateway_host: "{{ .hostname }}"

``






Loading