Pass Headers KV[string]string from source to sink

[vigith]

Summary

ISB.Header should support an additional field which a map of key, value of String type. Since headers are part of the data, the Datum, we should expose Datum.GetHeaders() method that returns KV[string]string.

Use Cases

• Sources like HTTP, Kafka, etc., have headers and UDFs want to access those
• Distributed Tracing
• Ability to pass transaction-id(tx_id), auth_headers, etc.

Notes

It is easy to propagate for Map vertices, but for Reduce, we cannot do a merge of many to one, hence it is okay to create an empty header after the Reduce operation.

---

Numaflow

Beta Give feedback

1. add KV[string]string to isb.Headers
2. update WALs to take in the headers
3. expose GetHeaders() in SDKs
4. SetHeader in User-Defined Source
5. Reset Header after Reduce (M:1 merge)

SDK Tracking

Beta Give feedback

1. Expose Headers for Golang SDK #1586
   area/sdk +1
   
2. Expose Headers for Python SDK #1587
   area/sdk +1
   
3. Expose Headers for Java SDK #1588
   area/sdk +1