Skip to content

server : add custom socket options to disable SO_REUSEPORT#21056

Merged
pwilkin merged 4 commits intoggml-org:masterfrom
angt:server-add-custom-socket-options-to-disable-so_reuseport
Mar 28, 2026
Merged

server : add custom socket options to disable SO_REUSEPORT#21056
pwilkin merged 4 commits intoggml-org:masterfrom
angt:server-add-custom-socket-options-to-disable-so_reuseport

Conversation

@angt
Copy link
Copy Markdown
Member

@angt angt commented Mar 27, 2026

Overview

cpp-httplib uses SO_REUSEPORT by default, see #20963

Additional information

yhirose/cpp-httplib#2409
yhirose/cpp-httplib#2411

Requirements

@angt
server : add custom socket options to disable SO_REUSEPORT
1140ea2 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
@angt angt requested a review from a team as a code owner March 27, 2026 06:55
@angt angt linked an issue Mar 27, 2026 that may be closed by this pull request
Eval bug: SO_REUSEPORT in cpp-httplib causes silent cross-contamination on multi-tenant systems #20963
Closed
@angt
Add --reuse-port
104b3fb 
    $ strace -e trace=setsockopt,bind build/bin/llama-server -lv 2 --reuse-port
    setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
    setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
    setsockopt(3, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0
    bind(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = 0

    $ strace -e trace=setsockopt,bind build/bin/llama-server -lv 2
    setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
    setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
    bind(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = 0

Signed-off-by: Adrien Gallouët <angt@huggingface.co>
@angt angt requested a review from a team as a code owner March 27, 2026 09:00
@angt
Copy link
Copy Markdown
Member Author

angt commented Mar 27, 2026

with --reuse-port:

$ strace -e trace=setsockopt,bind build/bin/llama-server -lv 2 --reuse-port
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = 0

without:

$ strace -e trace=setsockopt,bind build/bin/llama-server -lv 2
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
bind(3, {sa_family=AF_INET, sin_port=htons(8080), sin_addr=inet_addr("127.0.0.1")}, 16) = 0

@ServeurpersoCom
Copy link
Copy Markdown
Contributor

ServeurpersoCom commented Mar 27, 2026

I'm going to test this to a custom C++ script too, because I keep getting silently screwed by an old process still running. Super useful for dev!

@ServeurpersoCom
Copy link
Copy Markdown
Contributor

ServeurpersoCom commented Mar 27, 2026

Remember to launch llama-gen-docs to update doc :)

@angt
Update tools/server/README.md (llama-gen-docs)
18d0a4d 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
@angt
Copy link
Copy Markdown
Member Author

angt commented Mar 27, 2026

Damn windows 😅

@angt
Fix windows
6425e77 
Signed-off-by: Adrien Gallouët <angt@huggingface.co>
Green-Sky
Green-Sky approved these changes Mar 27, 2026
View reviewed changes
Copy link
Copy Markdown
Collaborator

@Green-Sky Green-Sky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this a thing again.

@angt
Copy link
Copy Markdown
Member Author

angt commented Mar 27, 2026

Last error is not related:

2026-03-27T12:39:12.3604906Z 0.00.025.560 I no remote preset found, skipping
2026-03-27T12:39:12.3605503Z 0.00.036.338 E get_repo_commit: error: GET failed (403): <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
2026-03-27T12:39:12.3605717Z <HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
2026-03-27T12:39:12.3605845Z <TITLE>ERROR: The request could not be satisfied</TITLE>

@pwilkin pwilkin merged commit 5c1a7b8 into ggml-org:master Mar 28, 2026
43 of 45 checks passed
@CISC
Copy link
Copy Markdown
Member

CISC commented Mar 28, 2026

Seems to consistently fail on this test now:
https://github.com/ggml-org/llama.cpp/actions/runs/23672869421/job/68969883927
https://github.com/ggml-org/llama.cpp/actions/runs/23680848194/job/68992335223

@ServeurpersoCom
Copy link
Copy Markdown
Contributor

ServeurpersoCom commented Mar 28, 2026

Oops, either we add the option to the test script (backward compatibility) or we find the real cause: (a process that's remain?)

@angt
Copy link
Copy Markdown
Member Author

angt commented Mar 28, 2026

Yes adding --reuse-port i guess, working on it

@angt angt mentioned this pull request Mar 28, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@taronaeo taronaeo taronaeo approved these changes

@ggerganov ggerganov Awaiting requested review from ggerganov

@ServeurpersoCom ServeurpersoCom Awaiting requested review from ServeurpersoCom

@ngxson ngxson Awaiting requested review from ngxson

+1 more reviewer

@Green-Sky Green-Sky Green-Sky approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

examples server

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Eval bug: SO_REUSEPORT in cpp-httplib causes silent cross-contamination on multi-tenant systems

8 participants

@angt @ServeurpersoCom @CISC @ggerganov @Green-Sky @ngxson @taronaeo @pwilkin