Skip to content

Fix Nonblocking writes#14

Merged
TheoHollender merged 3 commits into
releasefrom
fix/nonblocking-writes
Jun 9, 2026
Merged

Fix Nonblocking writes#14
TheoHollender merged 3 commits into
releasefrom
fix/nonblocking-writes

Conversation

@Zeleph25

@Zeleph25 Zeleph25 commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator

Add non-blocking write buffer to NetQueue

Problem: write_n_bytes spun with usleep(200) on EAGAIN. Since this runs inside dispatch handlers inside wait_and_process, send buffer pressure on any single connection stalled the entire event loop. Dropping on EAGAIN wasn't an option — losing CONNECT_TO_SWITCH or AGENT_CONNECTION_TOKEN breaks the handshake.

Fix: Each NetworkElement gets an outbound write buffer. On EAGAIN mid-write, remaining bytes are saved into the buffer and EPOLLOUT is registered on the fd. When the kernel send buffer drains, epoll fires and flush_write_buffer drains the queue. Once empty, EPOLLOUT is deregistered. Connections exceeding 64KB buffered are marked SCK_ERROR and closed on the next loop iteration — a connection that full isn't reading.

All in-loop sends in conductor, switch, and agent go through queue->send and queue->send_heartbeat. Blocking MIP phase sends stay on send_protobuf_packet since those fds aren't in the queue yet and are still in blocking mode.

Comment thread src/netqueue/netqueue.cpp
Comment on lines +75 to +82
// Buffer fully drained — clear it and remove EPOLLOUT
buf.clear();
offset = 0;

struct epoll_event ev;
ev.data.ptr = element;
ev.events = EPOLLIN | EPOLLET | EPOLLONESHOT;
epoll_ctl(epoll_fd, EPOLL_CTL_MOD, element->fd, &ev);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May be interesting to do some form of garbage collection at some point if the buffer is too big, but for now should be fine.

@TheoHollender TheoHollender merged commit 52f475c into release Jun 9, 2026
2 checks passed
@Zeleph25 Zeleph25 deleted the fix/nonblocking-writes branch June 9, 2026 16:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants