Support for file-logging OTLP Metrics/Traces/LogsData data for experimental and research purposes

[jmacd]

The exporter/fileexporter code could easily be extended to support capturing OTLP data for research purposes, as follows.

• Add an option to exporter/fileexporter to encode data using a protobuf stream in OTLP batch format described in Add proto messages for signals data independent of OTLP protocol opentelemetry-proto#332
• Add an option to self-rotate log files. I have found the gopkg.in/natefinch/lumberjack.v2 API is easy to adopt for this purpose (e.g., in this forked copy of the fileexporter).
• Add a processor to apply format-preserving encryption to obfuscate the data. I have found https://github.com/cyrildever/feistel is easy to adopt for this purpose.
• Add a receiver/filereceiver to replay telemetry recorded to the file exporter. (Although this could be done with the existing JSON support, it would be more efficient to use protobuf.)
• Add a Zstd compression option.

With these steps accomplished, it should be possible to setup an OTel collector that records large volumes of data with relatively low overhead. It should be possible to obfuscate telemetry data on the write path, or it should be possible to re-execute an OpenTelemetry collector that replays those logs to apply obfuscation after the fact. It should be possible to replay those logs to test OTel processors and exporters. Lastly, it is important that any code used for obfuscating telemetry data for research be widely reviewed by the community--these tools need to be well reviewed and held in community.

[yurishkuro]

apply format-preserving encryption to obfuscate the data

not sure why format-preserving would be a requirement

I think the biggest challenge with exporting data for research purposes that I encountered at two large companies is privacy concerns. The guidelines I got were: obfuscated or not, any privacy-sensitive data must not be included in public data sets.

[jmacd]

Format-preserving helps make compression results comparable.

obfuscated or not, any privacy-sensitive data must not be included in public data sets.

This is likely true in most places. Even so, it would be useful to have the plugins I described so that a customer could capture data and run diagnostic tools on them while protecting their data appropriately. For example, the benchmarks here (https://github.com/lquerel/otel-arrow-adapter/tree/main/tools) could be adapted to use data files generated in this way, where the obfuscation is just an extra layer of security.

[atingchen]

Can I take on this issue?

[atingchen]

@jmacd Could you help review?Thank you.

[jmacd]

Thank you @atingchen!

[atingchen]

Hi @jmacd.
I find that feistel may not obfuscate the data with format-preserving encryption.
I try to use it to encrypt 12 digit number '123123123123'. According to the format-preserving encryption what I should get is another 12 digit number. But what I actually get is `ZW±ªX^²[[®R' which is a string of length 12.
I'm also communicating with its developers about this issue, but haven't heard back yet.

[jmacd]

Hi @atingchen. Thank you and sorry for the delay in answering. I agree with your analysis that feistel cannot be applied in a format-preserving way to non-utf-8-formatted data, so numbers won't work. In my prototype work on this topic, I simply did not obfuscate number data, since I was reasonably sure in this case that numeric data would not leak sensitive information. Since feistel's obfuscation would change numeric field values, it would also impact studies meant to evaluate compression performance of numeric data, so it would not be "format preserving" in the sense we need it to for evaluating protocol decisions.

So, I think it would be reasonable for the tools in question to obfuscate only string-value fields in the protocol.

[github-actions]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

• exporter/file: @atingchen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[github-actions]

This issue has been closed as inactive because it has been stale for 120 days with no activity.