feat(python): add user header and origin timestamp support

[jiengup]

Add user headers to SendMessage and ReceiveMessage, expose origin timestamp, and fix Docker test infrastructure.

Which issue does this PR address?

Closes #3601 #3612

Rationale

The Python SDK could send and receive message payloads, but it did not expose the typed user headers already carried by the underlying Rust IggyMessage. This left Python behind the Rust, Node, Go, Java, and C# SDKs for message metadata.

What changed?

Python messages can attach user headers and read them back from received messages. SendMessage accepts user_headers and an optional custom id, while ReceiveMessage exposes user_headers() and origin_timestamp().

According to the discussion below, unlike the original proposal in #3601 (which exposed only a plain dict[str, str | bytes | bool | int | float]), the binding now supports the full Rust header type surface through two exposed classes, plus a plain-scalar convenience layer:

• HeaderKey / HeaderValue — typed complex enums covering every HeaderKind: Raw, String, Bool, Int8/16/32/64/128, UnsignedInt8/16/32/64/128, Float32, Float64.
• UserHeaders — the mapping returned by ReceiveMessage.user_headers(). It is a real dict subclass (dict[HeaderKey, HeaderValue]), so all mapping operations work, and it adds a chainable to_scalar_dict() for the convenient scalar form: message.user_headers().to_scalar_dict().

Sending (plain → Rust)

Each key/value pair is converted independently, so typed and plain forms can be freely mixed in one dict (e.g. {HeaderKey.String("a"): 7, "b": HeaderValue.Bool(True), "c": "plain"}):

• Plain scalars map by best-effort, lossless conversion for both keys and values (keys are no longer restricted to str):
  • str → String, bytes → Raw, bool → Bool
  • int → the smallest header kind that holds it exactly: non-negative → Uint8/16/32/64/128, negative → Int8/16/32/64/128; values beyond the 128-bit range raise ValueError.
  • float → Float32 when the value is exactly representable as f32, otherwise Float64 (always lossless).
• Explicit typed values are range-checked: integer variants are enforced by the binding at construction (OverflowError for out-of-range, e.g. HeaderValue.Int8(300)), and an explicit Float32 whose value overflows f32 (→ inf) is rejected with ValueError.

Receiving (Rust → plain)

Every HeaderKind maps losslessly onto a Python scalar (128-bit ints fit Python's arbitrary-precision int; f32 → f64 is exact), so:

• user_headers() returns typed UserHeaders (dict[HeaderKey, HeaderValue]), or None when the message carries no user headers.
• UserHeaders.to_scalar_dict() returns dict[str | bytes | bool | int | float, str | bytes | bool | int | float] — a direct, lossless conversion (no logging); it only raises ValueError on a genuine decode error of a stored field.

Header decode errors from known-but-invalid or unknown semantic kinds surface as ValueError. SendMessage(id=...) (mapped to u128) is part of the same binding update.

Minor Fix

The Python test compose setup starts the server with fresh default root credentials, binds HTTP/TCP/QUIC addresses explicitly, and uses iggy ping for the healthcheck instead of the HTTP stats endpoint.

TODO: make pre-commit and other scripts compatible with other shells (e.g. zsh); commands like mapfile don't work in some shells.

Discussion Notes

Choices from the issue discussion, updated to the final implementation:

• Python supports both a plain dict API for the common case and explicit HeaderKey/HeaderValue wrapper classes; the two can be mixed per entry.
• Integers are stored in the narrowest lossless kind (unsigned for non-negative, signed for negative); only values beyond ±128-bit raise ValueError.
• Floats select Float32/Float64 by exact representability instead of always Float64.
• Non-string keys are now supported (any plain scalar or HeaderKey), instead of being rejected.
• ReceiveMessage.user_headers() returns None for no headers.
• origin_timestamp() is exposed on received messages.

API Usage

Sending — plain scalars (common case)

from apache_iggy import SendMessage as Message

# Keys and values are plain Python scalars; each is converted losslessly.
Message(
    "order-created",
    user_headers={
        "content-type": "application/json",  # -> String
        "trace-blob": b"\x00\x01",           # -> Raw
        "retryable": False,                  # -> Bool
        "attempt": 3,                        # -> UnsignedInt8 (smallest fit)
        "created-at-ms": 1_700_000_000_000,  # -> UnsignedInt64
        "score": 1.25,                       # -> Float32 (exactly representable)
    },
    id=42,  # optional custom message id (u128)
)

Sending — explicit typed kinds

from apache_iggy import HeaderKey, HeaderValue
from apache_iggy import SendMessage as Message

Message(
    "order-created",
    user_headers={
        HeaderKey.String("schema-version"): HeaderValue.UnsignedInt16(1),
        HeaderKey.UnsignedInt32(7): HeaderValue.String("order-id"),  # non-string key
    },
)

Sending — mixed typed + plain in one dict

Message(
    "order-created",
    user_headers={
        HeaderKey.String("typed"): HeaderValue.Int32(-5),
        "plain-key": HeaderValue.Bool(True),
        HeaderKey.String("plain-val"): "hello",
        "both-plain": 9,
    },
)

Receiving

message = polled_messages[0]

message.origin_timestamp()          # int

headers = message.user_headers()    # UserHeaders | None
if headers is not None:
    # Typed access (full wire-type fidelity):
    value = headers.get(HeaderKey.String("schema-version"))
    if isinstance(value, HeaderValue.UnsignedInt16):
        print(value.value)          # 1

    # Convenient plain form (lossless):
    plain = headers.to_scalar_dict()      # dict[str|bytes|bool|int|float, str|bytes|bool|int|float]
    print(plain["schema-version"])  # 1

Validation / errors

Message("p", user_headers={"k": 2**128})            # ValueError: 128-bit range
Message("p", user_headers={"k": HeaderValue.Int8(300)})   # OverflowError
Message("p", user_headers={"k": HeaderValue.Float32(1e40)})  # ValueError: 32-bit float
Message("p", user_headers={object(): "v"})          # ValueError: keys must be str/bytes/bool/int/float/HeaderKey

Local Execution

• Passed: cargo fmt --check --manifest-path foreign/python/Cargo.toml
• Passed: cargo check --manifest-path foreign/python/Cargo.toml
• Passed: cargo test --manifest-path foreign/python/Cargo.toml
• Passed: uv run --extra dev ruff check tests/test_message_operations.py tests/test_consumer_group.py
• Passed: .venv/bin/python -m pytest tests/test_message_operations.py::TestMessageOperations::test_invalid_user_headers_are_rejected -q
• Successfully run: example/message-headers/typed-headers/consumer.py, example/message-headers/typed-headers/producer.py, example/message-headers/plain-headers/consumer.py, example/message-headers/plain-headers/producer.py

AI Usage

Codex was used to inspect the existing Python, Rust, Node, and Go SDK behavior, implement the Python binding changes, add tests. All the modification was reviewed carefully by the human.

[hubcio]

one additional request from my side: please update python examples to include a code that would show how to use iggy message user headers. there is one for rust already.

[codecov]

Codecov Report

❌ Patch coverage is 80.65134% with 101 lines in your changes missing coverage. Please review.
✅ Project coverage is 75.81%. Comparing base (e2bbaea) to head (25aae9f).

Files with missing lines	Patch %	Lines
foreign/python/src/user_headers.rs	79.17%	101 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #3613      +/-   ##
============================================
- Coverage     75.86%   75.81%   -0.06%     
  Complexity      969      969              
============================================
  Files          1323     1324       +1     
  Lines        160111   160623     +512     
  Branches     133497   133576      +79     
============================================
+ Hits         121468   121770     +302     
- Misses        35001    35122     +121     
- Partials       3642     3731      +89     

Components	Coverage Δ
Rust Core	75.74% <ø> (-0.05%)	⬇️
Java SDK	62.71% <ø> (ø)
C# SDK	71.16% <ø> (-1.11%)	⬇️
Python SDK	89.45% <80.65%> (-3.65%)	⬇️
PHP SDK	84.52% <ø> (ø)
Node SDK	96.30% <ø> (+0.15%)	⬆️
Go SDK	43.08% <ø> (ø)

Files with missing lines	Coverage Δ
foreign/python/src/lib.rs	100.00% <100.00%> (ø)
foreign/python/src/receive_message.rs	88.37% <100.00%> (+3.99%)	⬆️
foreign/python/src/send_message.rs	97.72% <100.00%> (+0.95%)	⬆️
foreign/python/src/user_headers.rs	79.17% <79.17%> (ø)

... and 53 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jiengup]

one additional request from my side: please update python examples to include a code that would show how to use iggy message user headers. there is one for rust already.

That will be included when the PR is ready.

[slbotbm]

@hubcio @jiengup

I was thinking that maybe we should change the signature of the headers from dict[str, str | bytes | bool | int | float] | None to dict[str, str | bytes | bool | int | float] | dict[HeaderKey, HeaderValue] | None, where HeaderKey and HeaderValue are python classes mirroring rust and expose subclasses like HeaderKey.String(value) and HeaderValue.UnsignedInt128(value), etc..

This would allow us to have two advantages: users that do not have much experience/do not require explicit control can use dict[str, str | bytes | bool | int | float], while users that require precise control on the types of headers can use dict[HeaderKey, HeaderValue]. Also, by providing dict[HeaderKey, HeaderValue] as an option, we allow AI Agents to make less mistakes as compared to dict[str, str | bytes | bool | int | float], which is a little vague from the agents' point of view.

What do you guys think?

[jiengup]

change the signature of the headers from dict[str, str | bytes | bool | int | float] | None to dict[str, str | bytes | bool | int | float] | dict[HeaderKey, HeaderValue] | None

I think it's a good idea and well aligned with the first point worth discussing in my PR. I'll push a new patch soon.

[hubcio]

@slbotbm for me this sounds fine, however one thing comes to my mind: will it have any impact on performance?

[jiengup]

@slbotbm for me this sounds fine, however one thing comes to my mind: will it have any impact on performance?

Actually, I think additional typed HeaderKey support will not introduce more overhead compared to the current implementation.

The only overhead we introduce is the iteration of user_header and the transformation of Rust and Python types, which need to be considered if we have a large number of user header pairs in some user cases.

[hubcio]

alright, then don't worry about it now.

[jiengup]

@hubcio @slbotbm

Hi, I just reflected the change and merged all the commits into a final one. The PR message is also updated.

Instead of a plain-only dict[str, ...] with int → Int64 / float → Float64 and string-only keys, the binding now exposes the full typed HeaderKey/HeaderValue surface with smallest-fit kind selection, supports non-string keys, and allows typed and plain pairs to be freely mixed in one dict.

Please have a look at it when you are free.

[slbotbm]

first round of review

[slbotbm]

/author

[jiengup]

/ready

[slbotbm]

Some more comments:

• You have set the type of user headers in some places to [Any, Any], which makes any type checker incapable of catching mistakes. Would be better to explicitly declare the types.

[jiengup]

/ready

[jiengup]

/ready

[slbotbm]

@jiengup Could you resolve the conflicts? Other than that LGTM

[slbotbm]

/author

[jiengup]

/ready