Add ByteChannel API for raw byte streaming#35
Merged
Conversation
ByteChannel provides raw byte streaming between Erlang and Python without term serialization overhead, suitable for HTTP bodies, file transfers, and binary protocols. Erlang API: - py_byte_channel:new/0,1 - Create channel with optional backpressure - py_byte_channel:send/2 - Send raw bytes - py_byte_channel:recv/1,2 - Blocking receive with optional timeout - py_byte_channel:try_receive/1 - Non-blocking receive - py_byte_channel:close/1 - Close channel Python API: - ByteChannel class with send_bytes, receive_bytes, try_receive_bytes - async_receive_bytes for asyncio compatibility - Sync and async iteration support Implementation reuses the existing py_channel_t infrastructure with new NIF functions that skip term_to_binary/binary_to_term conversion.
Use simple polling (asyncio.sleep) like the existing Channel.async_receive() instead of the more complex event loop dispatch integration. Both can be upgraded to proper event-driven async in a future change. Added async e2e test for ByteChannel.
Replaces polling with proper event loop integration: - Register with channel via direct C method (no Erlang callback overhead) - Wait for EVENT_TYPE_TIMER dispatch when data arrives - Falls back to polling for non-Erlang event loops New direct Python methods (bypass erlang.call): - erlang._channel_wait(ch, callback_id, loop_capsule) - erlang._channel_cancel_wait(ch, callback_id) - erlang._byte_channel_wait(ch, callback_id, loop_capsule) - erlang._byte_channel_cancel_wait(ch, callback_id) When ErlangEventLoop is used: 1. async_receive registers handle in loop._timers 2. Direct C call registers waiter with channel 3. channel_send dispatches via event_loop_add_pending 4. Event loop _dispatch fires callback, resolves Future 5. No polling overhead
- Add behavior notes for async_receive() explaining event-driven vs polling - Add event-driven async for ByteChannel documentation - Add architecture diagram showing async receive flow with ErlangEventLoop
Benchmark measures: - Erlang send throughput (no Python) - Erlang roundtrip (send + receive) - Python receive performance - Streaming throughput (1MB transfer) Run with: escript examples/bench_byte_channel.erl
Only use _timers dict for channel waiter registration. The _handle_to_callback_id dict is for timer cancellation and shouldn't be modified by channel waiters - this could cause race conditions in free-threaded Python.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ByteChannel API
Erlang:
{ok, Ch} = py_byte_channel:new(), ok = py_byte_channel:send(Ch, <<"raw bytes">>), {ok, Data} = py_byte_channel:recv(Ch), py_byte_channel:close(Ch).Python:
Event-Driven Async
When using
ErlangEventLoop,async_receive()andasync_receive_bytes()now use event-driven notification instead of polling:EVENT_TYPE_TIMERdispatch when data arrivesBenchmark Results