New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pybind/ceph_daemon: use small chunk for recv #13804

Merged
merged 1 commit into from Mar 20, 2017

Conversation

Projects
None yet
2 participants
@xiaoxichen
Contributor

xiaoxichen commented Mar 6, 2017

socket.recv(bufsize) accept signed int in python, so if
we want to load huge data (mds -> cache_dump is an instance)
from admin socket , an EOVERFLOW exception will be throw.

Walk around this by only get 4096 bytes for each call
Signed-off-by: Xiaoxi Chen xiaoxchen@ebay.com

@xiaoxichen xiaoxichen requested a review from tchaikov Mar 7, 2017

@@ -43,9 +43,12 @@ def do_sockio(path, cmd_bytes):
l, = struct.unpack(">I", len_str)
sock_ret = b''
got = 0
got = long(0)

This comment has been minimized.

@tchaikov

tchaikov Mar 7, 2017

Contributor

this breaks the python3 support.

bit = sock.recv(l - got)
# recv only accept signed int, i.e max 2GB
# walk around by limit each recv to 128M
want = min(l - got, 128*1024*1024)

This comment has been minimized.

@tchaikov

tchaikov Mar 7, 2017

Contributor

i'd suggest define a "relatively small power of 2, for example, 4096." [1], i.e.

READ_CHUNK_SIZE = 4096

and use it instead,


[1] https://docs.python.org/2/library/socket.html

@xiaoxichen xiaoxichen changed the title from pybind/ceph_daemon: limit each socket.recv to 128M to pybind/ceph_daemon: use small chunk for recv Mar 8, 2017

@xiaoxichen

This comment has been minimized.

Contributor

xiaoxichen commented Mar 8, 2017

@tchaikov addressed

@@ -45,7 +46,10 @@ def do_sockio(path, cmd_bytes):
got = 0
while got < l:
bit = sock.recv(l - got)
# recv only accept signed int, i.e max 2GB

This comment has been minimized.

@tchaikov

tchaikov Mar 8, 2017

Contributor

recv() receives ...

This comment has been minimized.

@xiaoxichen

xiaoxichen Mar 9, 2017

Contributor

Updated

better now

@@ -45,7 +46,10 @@ def do_sockio(path, cmd_bytes):
got = 0
while got < l:
bit = sock.recv(l - got)
# recv() receives signed int, i.e max 2GB
# walk around by request max READ_CHUNK_SIZE per call.

This comment has been minimized.

@tchaikov

tchaikov Mar 9, 2017

Contributor

workaround by capping, might want to update the commit message also.

@@ -45,7 +46,10 @@ def do_sockio(path, cmd_bytes):
got = 0
while got < l:
bit = sock.recv(l - got)
# recv() receives signed int, i.e max 2GB
# walkaround by capping READ_CHUNK_SIZE per call.

This comment has been minimized.

@tchaikov

tchaikov Mar 9, 2017

Contributor

s/walkaround/workaround/

This comment has been minimized.

@xiaoxichen

xiaoxichen Mar 10, 2017

Contributor

done, thanks for the english lesson :)

pybind/ceph_daemon: use small chunk for recv
socket.recv(bufsize) accept signed int in python, so if
we want to load huge data (mds -> cache_dump is an instance)
from admin socket , an EOVERFLOW exception will be throw.

Workaround by capping READ_CHUNK_SIZE(4096) bytes each call.

Signed-off-by: Xiaoxi Chen <xiaoxchen@ebay.com>

@tchaikov tchaikov added the needs-qa label Mar 10, 2017

@tchaikov tchaikov self-assigned this Mar 15, 2017

@tchaikov tchaikov merged commit cbd3f6c into ceph:master Mar 20, 2017

3 checks passed

Signed-off-by all commits in this PR are signed
Details
Unmodifed Submodules submodules for project are unmodified
Details
default Build finished.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment