Reduce network bandwidth with log variables

[nathanrstacey]

Description

Issue to resolve:
Cloud ingestion is expensive! I think we can reduce our ESS ingest costs significantly with a little work. My first test reduced log ingest by 80%!
This will also help reduce traffic in buildings/datacenters with many logs being created and a single pipe out of the building

How:
Most logs consist of the same data, only a few variables are changed between logs. Timestamp, CPU, packets sent, alert name, etc. All other data in the log is always the same

What if we only send those variables across the network. This can be done in many ways, below is a method. Not that it is the best method but simply to show what I am thinking

1. Let Elasticsearch auto-define templates of logs that it sees over and over again
2. When Elasticsearch sees a new template, have it define the template and give the template an ID
3. Have Elasticsearch send this template to the Agent/Beat sending the logs
4. When a new log is generated that matches a template, only send over the variables with the template ID
5. When Elasticsearch sees this new variable-log, it enters the variables into the template to create a true raw log. It will look exactly like it should for Elasticsearch to process it, the only difference is that only the necessary data was sent across the network

Attached is an example:
The "Raw JSON" is 5000B and the "Variable JSON" is 612 Bytes or 87% smaller
The pdf is the "Raw JSON" file but with variable lines highlighted to show what was and what was not part of the variable JSON
The Variable JSON starts with a line defining the template it is utilizing. Everything else in the JSON are variables. I imagine that I missed a few lines required to really make this work. I am sure that my above process may be faulty but the general idea is here. The general idea is to send over the changing variables in logs, not the whole log

rawauditbeat.pdf

variablesjson.txt
rawauditbeat.txt

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[DaveCTurner]

In principle I think this could work but it sounds awfully complex to me. Moreover I would expect that Content-Encoding: gzip would achieve roughly the same effect, and that works today.

Relates #94319.

(edit: I said Transfer-Encoding but I meant Content-Encoding)

[DaveCTurner]

Closing this as there was no further feedback.