Migrate worker server behaviour to libp2p-stream NET-924#211
Merged
Conversation
kalabukdima
changed the title
Replace the worker's request_response server with a new stream_server behaviour built on raw libp2p-stream, mirroring the already-migrated stream_client. Responses are now written directly from the per-stream handler task instead of round-tripping through the swarm loop, so the response queues, ResponseChannel plumbing, and per-result select arms are gone. The wire protocol (EOF-delimited protobuf, one request per stream) is unchanged, so old portals interoperate with the new worker. - Add stream_server behaviour (bytes-level Request/ResponseSender API) - Decode requests in the worker actor; drop the response queues and make send_* async, writing straight to the stream - Bump the libp2p fork rev for Control::accept_with_capacity, which buffers a burst of inbound streams instead of dropping them - Patch crates-io libp2p-identity to the fork: the fork now builds against its in-tree identity crate, so consumers must use the same source or two incompatible libp2p_identity copies get linked - Migrate only the worker; the portal logs collector keeps request_server
kalabukdima
force-pushed
the
net-stream-server-worker
branch
from
July 6, 2026 12:36
9426fb8 to
541684c
Compare
Contributor
|
I've opened a follow-up draft #212 (based on this branch): it makes
The cost is one type parameter on WDYT? |
Decode ran in on_query/on_sql_query/on_logs_request, which Wrapped calls from the swarm poll thread — the single task that also drives Kademlia, ping, and identify — and before the lossy enqueue, so every request was decoded even when it was about to be dropped. WorkerEvent now carries raw request bytes and the consumer decodes them off the swarm loop. Responses go straight through ResponseSender, so the send_* handle methods are gone and WorkerTransportHandle is a pure lifetime guard. on_query/on_sql_query keep a byte-level is_empty drop; empty logs requests are valid and forwarded as-is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
kalabukdima
marked this pull request as draft
July 8, 2026 07:38
Collaborator
Author
|
Sorry for the confusion. This PR shouldn't have been marked as ready for review yet |
kalabukdima
marked this pull request as ready for review
July 15, 2026 09:05
Collaborator
Author
|
This one now has a bit different approach but solves the same problems. Protobuf decoding is left for the caller site. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace the worker's
request_responseserver with a newstream_serverbehaviour built on the forkedlibp2p-stream.Responses can now be written directly from the per-stream handler task instead of syncing on the swarm.
stream_serverbehaviour.send_*async, writing straight to the stream.Control::accept_with_capacity, which buffers a burst of inbound streams instead of dropping them.request_serverfor now.