Sources have metadata associated with payload and we need to propagate those along with the payload

[vigith]

Summary

Numaflow source should have a way to propagate headers (e.g., Kafka Record Headers, HTTP Headers, etc.) along with the payload. The only way it is possible today is by passing the headers along with the payload after encoding it in JSON.

---

We could perhaps pass it in ISB.Header and expose it via the Metadata interface. This is not a clean approach because only sources have metadata and could be dropped after the first hop (transformer, Map UDF, etc.).

Another approach is not to include and let users write custom User-defined sources if they need to access that data. The most significant side effect is the need for sharability of commonly used sources, and hence, it is not worthwhile to reimplement them repeatedly.

Use Cases

Needs to access Header in processing vertex because they contain valuable metadata.

---

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[vigith]

depends on #1572