Support Splunk "cooked" data #3848
Comments
binarylogic
added
needs: approval
|
In the world of Splunk, the question of cooked vs uncooked is to choose between Splunk proprietary protocol (S2S AKA cooked) or just sending raw data over a TCP connection (uncooked). The benefit of using cooked data is that you get to include metadata. Suppose I have a Universal Forwarder installed on a Linux machine, configured to 2 types of log: Journal and Apache Access logs. If I ship those 2 sources as uncooked data, the receiver would simply receive the contents of the files and it is up to the receiver to guess whether a piece of data is from Journal or from an Apache Access log. To get around this, we could have the forwarder ship the Journal contents over one connection and the Apache access logs over another differentiated by destination ports - However, I hope you agree that this wouldn't scale very well. Now, if we ship the data as cooked data, we'll ship the source type (and some other data) alongside the raw data to the receiver. In general, a Splunk user is never forced by Splunk to use uncooked data and is generally advised to use S2S/cooked data. One usecase for uncooked data that I've encountered was when building a new log collection pipeline. We had Splunk as the data repo and we had Splunk agents installed on all endpoints. But for Reasons(TM), we wanted the log pipeline to not be Splunk software. In this case, since Vector didn't support S2S (#2537) we ended up with tons of Vector instances receiving raw TCP from Splunk (as uncooked data). It wasn't pretty and didn't scale well... but it worked. |
|
It was discovered in #11292 that the Splunk UF's |
|
👍 |
I've seen users request the ability to accept Splunk "cooked" data in various issue trackers and forums. This style of data can be received by other upstream Splunk forwarders. Decoding this data would make Vector unique for use cases where this data must be decoded and forwarded to non-Splunk destinations. It looks like this:
There are a few things we need to do before spending time on this:
The text was updated successfully, but these errors were encountered: