-
Notifications
You must be signed in to change notification settings - Fork 6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pybind/ceph_daemon: use small chunk for recv #13804
Conversation
src/pybind/ceph_daemon.py
Outdated
@@ -43,9 +43,12 @@ def do_sockio(path, cmd_bytes): | |||
l, = struct.unpack(">I", len_str) | |||
sock_ret = b'' | |||
|
|||
got = 0 | |||
got = long(0) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this breaks the python3 support.
src/pybind/ceph_daemon.py
Outdated
bit = sock.recv(l - got) | ||
# recv only accept signed int, i.e max 2GB | ||
# walk around by limit each recv to 128M | ||
want = min(l - got, 128*1024*1024) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i'd suggest define a "relatively small power of 2, for example, 4096." [1], i.e.
READ_CHUNK_SIZE = 4096
and use it instead,
a9164e3
to
367a035
Compare
@tchaikov addressed |
src/pybind/ceph_daemon.py
Outdated
@@ -45,7 +46,10 @@ def do_sockio(path, cmd_bytes): | |||
|
|||
got = 0 | |||
while got < l: | |||
bit = sock.recv(l - got) | |||
# recv only accept signed int, i.e max 2GB |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
recv() receives ...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated
367a035
to
f223309
Compare
src/pybind/ceph_daemon.py
Outdated
@@ -45,7 +46,10 @@ def do_sockio(path, cmd_bytes): | |||
|
|||
got = 0 | |||
while got < l: | |||
bit = sock.recv(l - got) | |||
# recv() receives signed int, i.e max 2GB | |||
# walk around by request max READ_CHUNK_SIZE per call. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
workaround by capping, might want to update the commit message also.
f223309
to
c1b98c6
Compare
src/pybind/ceph_daemon.py
Outdated
@@ -45,7 +46,10 @@ def do_sockio(path, cmd_bytes): | |||
|
|||
got = 0 | |||
while got < l: | |||
bit = sock.recv(l - got) | |||
# recv() receives signed int, i.e max 2GB | |||
# walkaround by capping READ_CHUNK_SIZE per call. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/walkaround/workaround/
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done, thanks for the english lesson :)
socket.recv(bufsize) accept signed int in python, so if we want to load huge data (mds -> cache_dump is an instance) from admin socket , an EOVERFLOW exception will be throw. Workaround by capping READ_CHUNK_SIZE(4096) bytes each call. Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>
c1b98c6
to
4e8a19b
Compare
socket.recv(bufsize) accept signed int in python, so if
we want to load huge data (mds -> cache_dump is an instance)
from admin socket , an EOVERFLOW exception will be throw.
Walk around this by only get 4096 bytes for each call
Signed-off-by: Xiaoxi Chen xiaoxchen@ebay.com