Add async/concurrency to udp receiver (stanza udp input operator)

[hovavza]

Description: Adding a feature - Adding asynchronous & concurrency mode to the UDP receiver/stanza input operator - goal is to reduce UDP packet loss in high-scale scenarios.
Added 5 new flags to the config: AsyncConcurrentMode, FixedAsyncReaderRoutineCount, FixedAsyncProcessorRoutineCount, MaxAsyncQueueLength, MaxGracefulShutdownTimeInMS.

Goal:
Since the underlying computer's network buffer is limited and there's no handshake in UDP (that would cause the sender to reduce rate, as in TCP), even short term latency on the otel-collector's side (for example, the endpoint receiving logs from the otel-exporter has increased latency) causes UDP packet drop in high scale scenarios (and as a result, data loss).
This feature allows the consumer to load data into the memory (instead of the limited network buffer), so short term latency or issues downstream, will not quickly result in data loss.
Also, since receiver can be configured to perform split & add attributes operations on each packet, it improves performance since it separates the threads that read from UDP from the threads that process (before sending the packets downstream). It allows to raise concurrency level of either thread reader routines & the processor routines, depending on scenario.

Link to tracking Issue: #27613

Testing: Local stress tests ran and indicated data-loss issue was solved (during high scale scenarios), when MaxAsyncQueueLength was set sufficiently high (+AsyncConcurrentMode==true).
In repo, added single test to udp_test, config_test (in stanza udp operator), and udp_test (in udplogreceiver).

Documentation: Updated md file for both udplogreceiver & stanza udp_input operator with the new flags.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: hovavza (63f120d, fa459e0)

[djaglowski]

Can we take these changes incrementally?

I suggest first we define an async config block and within this define the number of concurrent readers.

As a separate PR, we can look at decoupling reading packets from processing.

Finally, we can look at the shutdown time.

[djaglowski]

Given the close relationship between these settings, can we group them into one struct?

I think async_concurrent_mode can be inferred from whether or not the struct is specified:

receivers:
  udplog:
    ...
  udplog/async_default:
    ...
    async: # with default async settings
  udplog/async_custom:
    ...
    async:
      ... # custom settings
      
      

[hovavza]

Can we take these changes incrementally?

I suggest first we define an async config block and within this define the number of concurrent readers.

As a separate PR, we can look at decoupling reading packets from processing.

Finally, we can look at the shutdown time.

Thanks. Actually, I think decoupling of reader and processor (in other words, making it asynchronous) is more important than making it concurrent (multiple readers/processors) due to the channel that can be defined to be pretty big. Stress tests indicated that as well. In other words, if there's short term latency downstream (in the exporter, for example), having multiple readers won't help too much.
But if you agree we can add all these 3 steps, we just need to do it one after the other, it doesn't really matter what comes first.

[djaglowski]

But if you agree we can add all these 3 steps, we just need to do it one after the other, it doesn't really matter what comes first.

I'm sold on the first two steps. I think the order I mentioned will help me review the changes.

The shutdown delay is likely reasonable but it's not something the component can guarantee. It's probably reasonable to add a best effort attempt up to a certain threshold but I'd like to consider this separately.