Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

storage: integer overflow in compacted_index_chunk_reader.cc #7647

Closed
incident-io bot opened this issue Dec 7, 2022 · 0 comments · Fixed by #7687
Closed

storage: integer overflow in compacted_index_chunk_reader.cc #7647

incident-io bot opened this issue Dec 7, 2022 · 0 comments · Fixed by #7687
Assignees
@ztlpn
Labels
area/storage kind/bug Something isn't working

Comments

@incident-io
Copy link

incident-io bot commented Dec 7, 2022 •
edited by jason-da-redpanda

Error was seen when compacting logs

2022-12-01 11:50:29WARN 2022-12-01 11:50:29,204 [shard 2] storage - segment_utils.cc:303 - detected error while attempting recovery, std::runtime_error (Invalid file checksum. Expected: 3797636102, but got:0 - {type:compacted_index_chunk_reader, _max_chunk_memory:65536, _file_size:{2202393163}, _footer:{{size:2202393146, keys:13322801, flags:0, crc:3797636102, version: 1}}, active_cursor:no, end_of_stream:false, _byte_index:0}). marking as 'needs rebuild'. Common situation during crashes or hard shutdowns.

Issue identified as overflow of the int32 size value issue
https://github.com/redpanda-data/redpanda/blob/7a3a92474a95c457c7f5d0028e2e5d81b9750a4a/src/v/storage/compacted_index_chunk_reader.cc

This action was created from Incident 110 by jason barlow, using incident.io 🔥
@jason-da-redpanda jason-da-redpanda changed the title implement and release a fix for bug due to ooverflow of the int32 size value. ins compacted_index_chunk_reader.cc implement and release a fix for bug due to overflow of the int32 size value. incompacted_index_chunk_reader.cc Dec 7, 2022
@jason-da-redpanda jason-da-redpanda changed the title implement and release a fix for bug due to overflow of the int32 size value. incompacted_index_chunk_reader.cc implement and release a fix for bug due to overflow of the int32 size value. in compacted_index_chunk_reader.cc Dec 7, 2022
@jcsp jcsp added kind/bug Something isn't working area/storage labels Dec 12, 2022
@jcsp jcsp changed the title implement and release a fix for bug due to overflow of the int32 size value. in compacted_index_chunk_reader.cc storage: integer overflow in compacted_index_chunk_reader.cc Dec 12, 2022
ztlpn added a commit to ztlpn/redpanda that referenced this issue Dec 12, 2022
@ztlpn
s/compacted_index: use 64-bit ints for size and number of keys
1916f02 
As by default max compacted segment size is 5 GiB, compacted index size
for large segments can oveflow 32-bit ints. This commit substitutes size
and the number of keys in the footer for 64-bit ints. Old 32-bit fields are
left for backwards compatibility with the code that doesn't check
version number.

Fixes redpanda-data#7647
@ztlpn ztlpn mentioned this issue Dec 12, 2022
Merged
3 tasks
ztlpn added a commit to ztlpn/redpanda that referenced this issue Dec 16, 2022
@ztlpn
s/compacted_index: use 64-bit ints for size and number of keys
449208f 
As by default max compacted segment size is 5 GiB, compacted index size
for large segments can oveflow 32-bit ints. This commit substitutes size
and the number of keys in the footer for 64-bit ints. Old 32-bit fields are
left for backwards compatibility with the code that doesn't check
version number.

Fixes redpanda-data#7647
ztlpn added a commit to ztlpn/redpanda that referenced this issue Feb 8, 2023
@ztlpn
storage: clamp max_compacted_segment_size to 1.5 GiB
3bb9a3e 
Clamp max compacted segment size to 1.5GiB to avoid compaction index
size overflow.

Fixes redpanda-data#7647

This is a workaround for older versions, proper fix is
redpanda-data#7687
@ztlpn ztlpn mentioned this issue Feb 8, 2023
Merged
6 tasks
ztlpn added a commit to ztlpn/redpanda that referenced this issue Feb 8, 2023
@ztlpn
storage: clamp max_compacted_segment_size to 1.5 GiB
49e4952 
Clamp max compacted segment size to 1.5GiB to avoid compaction index
size overflow.

Fixes redpanda-data#7647

This is a workaround for older versions, proper fix is
redpanda-data#7687
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ztlpn ztlpn

Labels
area/storage kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix compaction index size overflow ztlpn/redpanda
2 participants
@jcsp @ztlpn