add option to skip log tokenization

[haimrubinstein]

Description: Performance improvement for UDP receiver.
Currently, UDP created on each log line a scanner object with split functions, even if there was no need for log tokenizing.
this PR adds a new config option 'tokenize_log' (default value set to true) which lets you decide if this is the behavior you want.
if set as false the scanner object is not created and the performance is increased significantly (up to 3X)

Link to tracking Issue: #23237

Testing:
Created a local build of the UDP receiver and tested that the logs arrive and exported as expected.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: haim6678 / name: Haim Rubinstein (dddfa6f, 12645b1, 11ec359, b7cfbcd, b86140a, c4c6788, eeae0f9, c5ead69, 0c5bc28, ca2ca7c)
• ✅ login: djaglowski / name: Daniel Jaglowski (7826a82, 65299b9, 9ea78f0)

[djaglowski]

I like how simple this solution is, but I wonder if we're applying it too specifically to the udp input. Should this also be a concern of tcp?

[djaglowski]

@haim6678, please sign the CLA

[haimrubinstein]

@djaglowski I added a change log file, can you re-approve, please?

[haimrubinstein]

I like how simple this solution is, but I wonder if we're applying it too specifically to the UDP input. Should this also be a concern of tcp?

I agree, I thought of this too. TCP input has the same logic so I assume it's suffering from the same issue. I can do it once this PR is merged if you want

[djaglowski]

I like how simple this solution is, but I wonder if we're applying it too specifically to the UDP input. Should this also be a concern of tcp?

I agree, I thought of this too. TCP input has the same logic so I assume it's suffering from the same issue. I can do it once this PR is merged if you want

I think the two receivers are related enough that it would be best to keep them in sync. Do you mind adding the tcp changes to this PR?

[haimrubinstein]

I like how simple this solution is, but I wonder if we're applying it too specifically to the UDP input. Should this also be a concern of tcp?

I agree, I thought of this too. TCP input has the same logic so I assume it's suffering from the same issue. I can do it once this PR is merged if you want

I think the two receivers are related enough that it would be best to keep them in sync. Do you mind adding the tcp changes to this PR?

done

[djaglowski]

Thanks for iterating @haim6678. This looks good but I have a few nits.

We also need to document the new parameter in the receiver README and operator README.

[haimrubinstein]

@djaglowski can you merge this PR? I'm not authorized.

[djaglowski]

We also need to document the new parameter in the receiver README and operator README.

@haim6678, please also update the operator readmes

[haimrubinstein]

We also need to document the new parameter in the receiver README and operator README.

@haim6678, please also update the operator readmes

done