Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Prometheus remote write receiver compatible with sfx gateway #3064

Merged
merged 56 commits into from
May 8, 2023

Conversation

hughesjj
Copy link
Contributor

@hughesjj hughesjj commented May 4, 2023

  • Will set status stabilitylevel to alpha once approved + read in version and name from metadata.yaml generated status.go
  • See README.md for nuances in configuration

@hughesjj hughesjj requested review from a team as code owners May 4, 2023 20:47
Co-authored-by: Antoine Toulme <antoine@lunar-ocean.com>
@hughesjj hughesjj marked this pull request as draft May 8, 2023 00:07
@hughesjj hughesjj marked this pull request as ready for review May 8, 2023 14:44
hughesjj and others added 2 commits May 8, 2023 11:22
…/internal/prometheus_spec_utils.go

Co-authored-by: Antoine Toulme <antoine@lunar-ocean.com>
@hughesjj hughesjj merged commit 43ce40f into main May 8, 2023
119 checks passed
@delete-merged-branch delete-merged-branch bot deleted the sfxcompat branch May 8, 2023 19:47
@hughesjj hughesjj restored the sfxcompat branch May 8, 2023 19:53
@@ -0,0 +1,24 @@
# SignalFx Gateway Prometheus Remote write Receiver Example
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A bit long(ish). Can we shorten to, say, Prometheus remote write example?

Comment on lines +3 to +5
This example provides a `docker-compose` environment that continually sends some fake prometheus remote writes to an otel receiver replacement for the deprecated SignalFx Gateway for Prometheus Remote Writes.
To run this, ensure you have `docker-compose` installed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
This example provides a `docker-compose` environment that continually sends some fake prometheus remote writes to an otel receiver replacement for the deprecated SignalFx Gateway for Prometheus Remote Writes.
To run this, ensure you have `docker-compose` installed.
This example provides a `docker-compose` environment that continually sends fake prometheus remote writes to an OTel receiver replacement for the deprecated SignalFx Gateway for Prometheus Remote Writes. To run the example, make sure you have `docker-compose` installed.

To run this, ensure you have `docker-compose` installed.

## Configuration
You can change the exporters to your liking by modifying `otel-collector-config.yaml`.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can change the exporters to your liking by modifying `otel-collector-config.yaml`.
To change the exporters, edit the `otel-collector-config.yaml` configuration file.

Comment on lines +9 to +15
Ensure the following environment variables are properly set, should you wish to send data to splunk observability cloud:
1. `SPLUNK_ACCESS_TOKEN`
2. `SPLUNK_REALM`

Alternatively, you can remove the `signalfx` array item from the `exporters` configuration map in `otel-collector-config.yaml`

Feel free to modify the sample client to your liking, or even disable it and write your own!
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Ensure the following environment variables are properly set, should you wish to send data to splunk observability cloud:
1. `SPLUNK_ACCESS_TOKEN`
2. `SPLUNK_REALM`
Alternatively, you can remove the `signalfx` array item from the `exporters` configuration map in `otel-collector-config.yaml`
Feel free to modify the sample client to your liking, or even disable it and write your own!
If you want to send data to Splunk Observability Cloud, set the following environment variables:
-`SPLUNK_ACCESS_TOKEN`
-`SPLUNK_REALM`
You can also remove the `signalfx` array item from the `exporters` configuration map in the `otel-collector-config.yaml` configuration file.
> **Tip:** Experiment and modify the sample client, or even disable it and write your own.

Comment on lines +17 to +24
## Running
Once you've verified your environment, you can run the example by

```bash
$> docker-compose up
```

If everything is configured properly, logs with sample writes should start appearing in stdout shortly.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Running
Once you've verified your environment, you can run the example by
```bash
$> docker-compose up
```
If everything is configured properly, logs with sample writes should start appearing in stdout shortly.
## Run the example
After you've verified your environment, enter the following command to run the example:
```bash
$> docker-compose up

If everything is configured properly, logs with sample writes start appearing in stdout.

@@ -0,0 +1,32 @@
# SignalFx Gateway Prometheus remote write receiver

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hughesjj Could you add here what the receiver is for?

Comment on lines +3 to +18
## Limitations and Nuances in translation
This receiver specifically obsoletes the near-exact behavior of the [SignalFx prometheus remote write gateway](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go).
The behavior of the prometheus remote write gateway predates the formalization of the PRW v1 specification, and thus differs in the following ways.

- We do not [remove suffixes](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/6658646e7705b74f13031c777fcd8dd1cd64c850/receiver/prometheusreceiver/internal/metricfamily.go#L316) as is done in the otel-contrib `prometheusreceiver`
- It will transform histograms [into counters](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#L98).
- It will transform quantiles (summaries) into gauges.
- If the representation of a float could be expressed as an integer without loss, we will set it as an integer
- If the representation of a sample is NAN, we will report an additional counter with the metric name [`"prometheus.total_NAN_samples"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL190C24-L190C53)
- If the representation of a sample is missing a metric name, we will report an additional counter with the metric name [`"prometheus.total_bad_datapoints"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL191C24-L191C24)
- Any errors in parsing the request will report an additional counter [`"prometheus.invalid_requests"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL189C80-L189C91)
- Metadata from the `prompb.WriteRequest` is **ignored**

The following behavior from sfx gateway is not supported
- `"request_time.ns"` is no longer reported. `obsreport` handles similar functionality.
- `"drain_size"` is no longer reported. `obsreport` handles similar functionality.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Limitations and Nuances in translation
This receiver specifically obsoletes the near-exact behavior of the [SignalFx prometheus remote write gateway](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go).
The behavior of the prometheus remote write gateway predates the formalization of the PRW v1 specification, and thus differs in the following ways.
- We do not [remove suffixes](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/6658646e7705b74f13031c777fcd8dd1cd64c850/receiver/prometheusreceiver/internal/metricfamily.go#L316) as is done in the otel-contrib `prometheusreceiver`
- It will transform histograms [into counters](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#L98).
- It will transform quantiles (summaries) into gauges.
- If the representation of a float could be expressed as an integer without loss, we will set it as an integer
- If the representation of a sample is NAN, we will report an additional counter with the metric name [`"prometheus.total_NAN_samples"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL190C24-L190C53)
- If the representation of a sample is missing a metric name, we will report an additional counter with the metric name [`"prometheus.total_bad_datapoints"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL191C24-L191C24)
- Any errors in parsing the request will report an additional counter [`"prometheus.invalid_requests"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL189C80-L189C91)
- Metadata from the `prompb.WriteRequest` is **ignored**
The following behavior from sfx gateway is not supported
- `"request_time.ns"` is no longer reported. `obsreport` handles similar functionality.
- `"drain_size"` is no longer reported. `obsreport` handles similar functionality.
## Known limitations
This receiver obsoletes the near-exact behavior of the [SignalFx Prometheus Remote-Writegateway](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go). The behavior of the Prometheus Remote-Write gateway predates the formalization of the Prometheus Remote-Write specification version 1, and differs in the following ways:
- The receiver doesn't [remove suffixes](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/6658646e7705b74f13031c777fcd8dd1cd64c850/receiver/prometheusreceiver/internal/metricfamily.go#L316) as this is done in the otel-contrib `prometheusreceiver`.
- The receiver transforms histograms [into counters](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#L98).
- The receiver transforms quantiles (summaries) into gauges.
- If the representation of a float can be expressed as an integer without loss, the receiver sets the representation of a float as an integer.
- If the representation of a sample is NaN, the receiver reports an additional counter with the metric name [`"prometheus.total_NAN_samples"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL190C24-L190C53).
- If the representation of a sample is missing a metric name, the receiver reports an additional counter with the metric name [`"prometheus.total_bad_datapoints"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL191C24-L191C24).
- Any errors in parsing the request report an additional counter, [`"prometheus.invalid_requests"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL189C80-L189C91).
- Metadata from the `prompb.WriteRequest` is ignored.
The following behavior from sfx gateway is not supported:
- `"request_time.ns"` is no longer reported. `obsreport` handles similar functionality.
- `"drain_size"` is no longer reported. `obsreport` handles similar functionality.

Comment on lines +20 to +31
## Receiver Configuration
This receiver is configured via standard OpenTelemetry mechanisms. See [`config.go`](./config.go) for specific details.

* `path` is the path in which the receiver should respond to prometheus remote write requests.
* Defaults to `/metrics`
* `buffer_size` is the degree to which metric translations may be buffered without blocking further write requests.
* Defaults to `100`

This receiver uses `opentelemetry-collector`'s [`confighttp`](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/confighttp.go#L206) options if you would like to set up tls or similar. (See linked documentation for the most up-to-date details).
However, we make the following changes to their default options:
* `endpoint` is the default interface + port to listen on
* Defaults to `localhost:19291`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
## Receiver Configuration
This receiver is configured via standard OpenTelemetry mechanisms. See [`config.go`](./config.go) for specific details.
* `path` is the path in which the receiver should respond to prometheus remote write requests.
* Defaults to `/metrics`
* `buffer_size` is the degree to which metric translations may be buffered without blocking further write requests.
* Defaults to `100`
This receiver uses `opentelemetry-collector`'s [`confighttp`](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/confighttp.go#L206) options if you would like to set up tls or similar. (See linked documentation for the most up-to-date details).
However, we make the following changes to their default options:
* `endpoint` is the default interface + port to listen on
* Defaults to `localhost:19291`
## Receiver configuration
This receiver is configured through standard OpenTelemetry mechanisms. See [`config.go`](./config.go) for details.
* `path` is the path in which the receiver responds to prometheus remote-write requests. The default values is `/metrics`.
* `buffer_size` is the degree to which metric translations can be buffered without blocking further write requests. The default value is `100`.
This receiver uses `opentelemetry-collector`'s [`confighttp`](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/confighttp.go#L206) options if you want to set up TLS and other features. However, the receiver makes the following changes to upstream default options:
* `endpoint` is the default interface and port to listen on. The default value is `localhost:19291`.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants