Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bytes: iobuf_copy requests large contiguous ranges #3428

Closed
jcsp opened this issue Jan 10, 2022 · 1 comment · Fixed by #3435
Closed

bytes: iobuf_copy requests large contiguous ranges #3428

jcsp opened this issue Jan 10, 2022 · 1 comment · Fixed by #3435
Assignees
Labels

Comments

@jcsp
Copy link
Contributor

jcsp commented Jan 10, 2022

(Internal slack ref https://redpandadata.slack.com/archives/C01ND4SVB6Z/p1641835943108600)

  • iobuf_copy rounds required bytes down to next pow2 to choose allocation size. Means if the copy size is big, so is the allocation.
  • it's intentional that copy doesn't aggressively fragment the data, to avoid generating fragmentation that would hurt other subsystems that need contiguous ranges. However, this also means that copy is very sensitive to fragmentation, and will throw bad_alloc on systems with somewhat fragmented allocators.
  • Large calls to iobuf_copy originate in the kafka fetch path, where each partitions read_result structure has a buffer whose size is rougly the max_size the client requested (10MB default). Consequently iobuf_copy tries to do 8MB contiguous allocations.
@jcsp jcsp added kind/bug Something isn't working area/kafka area/redpanda labels Jan 10, 2022
@jcsp jcsp self-assigned this Jan 10, 2022
@jcsp
Copy link
Contributor Author

jcsp commented Jan 10, 2022

First cut of backoff on bad_alloc is in #3427, for the si-testing branch where the issue is cropping up in testing.

jcsp added a commit to jcsp/redpanda that referenced this issue Jan 11, 2022
This is the helper for iobuf_copy, that previously would
attempt to allocate huge contiguous extents when copying
large iobufs.

Make it respect the same max_chunk_size as iobuf
append/reserve operations, to avoid bad_allocs when
copying large buffers under memory pressure.

Fixes: redpanda-data#3428

Signed-off-by: John Spray <jcs@vectorized.io>
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 12, 2022
This is the helper for iobuf_copy, that previously would
attempt to allocate huge contiguous extents when copying
large iobufs.

Make it respect the same max_chunk_size as iobuf
append/reserve operations, to avoid bad_allocs when
copying large buffers under memory pressure.

Fixes: redpanda-data#3428

Signed-off-by: John Spray <jcs@vectorized.io>
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 13, 2022
This is the helper for iobuf_copy, that previously would
attempt to allocate huge contiguous extents when copying
large iobufs.

Make it respect the same max_chunk_size as iobuf
append/reserve operations, to avoid bad_allocs when
copying large buffers under memory pressure.

Fixes: redpanda-data#3428

Signed-off-by: John Spray <jcs@vectorized.io>
(cherry picked from commit a9a7a3e)
jcsp added a commit to jcsp/redpanda that referenced this issue Jan 13, 2022
This is the helper for iobuf_copy, that previously would
attempt to allocate huge contiguous extents when copying
large iobufs.

Make it respect the same max_chunk_size as iobuf
append/reserve operations, to avoid bad_allocs when
copying large buffers under memory pressure.

Fixes: redpanda-data#3428

Signed-off-by: John Spray <jcs@vectorized.io>
(cherry picked from commit a9a7a3e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
1 participant