New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PXB-2854 - Quicklz decompression memory corruption issue fix #1366
Conversation
|
|
|
Related to this we have also submitted PierreLvx/qpress#6 |
|
Related blog post: https://lavaux.lv/2022/08/21/qpress-file-archiver-security-update.html |
7c41171
to
2aad9cd
Compare
|
Any possibility to get a review on this one? |
|
First of all, thanks for providing the patch for this issue. We have raised an internal bug to keep track of it https://jira.percona.com/browse/PXB-2854. This issue is currently a blocker for our next release. We are in the process of working on the issues that will be part of the release and this PR will get reviewed soon. Thanks |
|
@Chaloff I am working on reviewing this fix and merging it to our next release branch. Can you please sign the CLA agreement at #1366 (comment) |
|
AWS does not sign CLAs. We contribute this with the open source license of
the project.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will get back on the license once I hear back internally.
For now, I can see that the provided patch breaks the software functionality:
xtrabackup --backup --port=3306 --stream=xbstream --parallel=16 --compress --compress-threads=4 --encrypt=AES256 --encrypt-key='percona_xtrabackup_is_awesome___' --encrypt-threads=4 --encrypt-chunk-size=8K > backup.out
mkdir out
xbstream -xv --parallel=1 --decompress --decompress-threads=1 --decrypt=AES256 --encrypt-key='percona_xtrabackup_is_awesome___' --encrypt-threads=1 -C out < backup.out
This produces an error:
sys/sys_config.ibd.qp.xbcrypt
Error: compressed file was corrupted - header data size and actual data size mismatch - can’t decompress
decompress: error running decompression.
decrypt: write to destination failed.
xbstream: my_write() failed.
exit code: 1
2aad9cd
to
9154211
Compare
|
Hi @Chaloff . I am not sure if your last force push is intended to fix the encrypt issue. I tested it and I can still see the error: |
Checking... |
9154211
to
906fec9
Compare
|
Hi @Chaloff Using latest commit the same issue still happening: |
I probably need some assistance here if you don't mind. The fix in qpress are pretty simple and well tested - it just check boundaries of two arrays (source and target) before decompress. The problem seems to be in calling this qpress function - qlz_decompress(...) - we need to pass the allocated size of source and target arrays to be able to check against it. I do it like this: |
|
Hi @Chaloff . I am sharing the file compressed with qpress and compressed with qpress via xbstream (on xbstream format). When checking the same file with qpress under GDB I see that it produces the from_alloc_size as 403 (for reference, we are here): Adjusting xbstream code to use However, the second condition to the if is returning true: By disabling the error (return 0) on that if, I can see that the check fails multiple times for this file. However, it does produce the correct file as result and also the file has the same resulting size of 477 bytes as we computed when calling qlz_decompress_core ( Files I used: |
Thanks for the support - looking... |
There is a memory corruption issue inside the quicklz.c source file that ships
with Percona XtraBackup. Specifically the problem happens on copying
user-supplied binary data over heap allocated memory buffers of user-controlled
size. This allows corruption of heap data structures and potential arbitrary
code execution.
The code in question is inside the qlz_decompress function of quicklz.c file:
```
size_t qlz_decompress(const char *source, void *destination, qlz_state_decompress *state)
{
size_t dsiz = qlz_size_decompressed(source);
if (state->stream_counter + qlz_size_decompressed(source) - 1 >= QLZ_STREAMING_BUFFER)
{
if((*source & 1) == 1)
{
reset_table_decompress(state);
dsiz = qlz_decompress_core((const unsigned char *)source, (unsigned char *)destination, dsiz, state, (const unsigned char *)destination);
}
else
{
memcpy(destination, source + qlz_size_header(source), dsiz);
}
state->stream_counter = 0;
reset_table_decompress(state);
}
else
{
unsigned char *dst = state->stream_buffer + state->stream_counter;
if((*source & 1) == 1)
{
dsiz = qlz_decompress_core((const unsigned char *)source, dst, dsiz, state, (const unsigned char *)state->stream_buffer);
}
else
{
memcpy(dst, source + qlz_size_header(source), dsiz);
reset_table_decompress(state);
}
memcpy(destination, dst, dsiz);
state->stream_counter += dsiz;
}
return dsiz;
}
```
Note the first memcpy invocation: that does copy data from user-provided
compressed file into a heap-allocated buffer for which size is also controlled
by the user via the compressed file header. This allows heap corruption with
user-controlled data. Potentially this means arbitrary code execution for the
processes that utilize the vulnerable function - one example is xbstream with
—decompress flag.
Steps to reproduce:
- Create a compressed file, e.g. with qpress from some file larger than 65535
bytes.
- Edit compressed file so that the four bytes at offset 8 are changed to be
less than 0x10000, for example set to 0x1000 instead.
- Edit the file so that the byte at offset 50 is an even value to pass the
test: if((*source & 1) == 1)
- Replace the bytes of actual file with some recognizable pattern, e.g. 0x41
0x42 0x43 0x44
- Add the file to an xbstream file: xbstream -c Demo.qp > Demo.xbstream
- Now try to extract with decompression using xbstream under a debugger, e.g.
gdb and observe the corruption: xbstream —decompress -x < Demo.xbstream
```
head -c 100000 </dev/urandom > payload.bin
qpress payload.bin payload.qp
ls -l payload.qp -rw-r--r-- 1 me me 100107 Feb 17 18:08 payload.qp
printf '\x00\x01\x00' | dd of=payload.qp bs=1 seek=8 count=3 conv=notrunc
printf '\x10' | dd of=payload.qp bs=1 seek=49 count=1 conv=notrunc
python -c 'import sys; sys.stdout.write("A"*100040)' | dd of=payload.qp bs=1
seek=50 count=100040 conv=notrunc
xbstream-80 -c payload.qp > corrupted.xbstream
$ xbstream-80 --decompress -x < corrupted.xbstream Segmentation fault ```
Fix by prevent XtraBackup read/write outside array bounds ``` We noticed that
quicklz already has a mechanism to check if the data was corrupted and prevent
reading and writing outside array bounds. Programm should be compiled against
QLZ_MEMORY_SAFE flag. So we uncommented the flag and add the error message.
All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.
906fec9
to
dac3d82
Compare
|
@Chaloff, it appears that in the latest version of your code (dac3d82), it is no longer checking the data size in the header against the size of the actual data region of the file. That means it is no longer actually addressing the vulnerability it is intended to address (= Am I missing something? |
Yes, you are missing something but thanks anyway for the comment :) Please see the change on line 31 in quicklz.h |
|
Hi @Chaloff . I confirm that all tests are passing now including a new test case added to cover the memory corruption issue. Code has been incorporated into 8.0 trunk. Thanks for your contribution! |
There is a memory corruption issue inside the quicklz.c source file that ships
with Percona XtraBackup. Specifically the problem happens on copying
user-supplied binary data over heap allocated memory buffers of user-controlled
size. This allows corruption of heap data structures and potential arbitrary
code execution.
The code in question is inside the qlz_decompress function of quicklz.c file:
Note the first memcpy invocation: that does copy data from user-provided
compressed file into a heap-allocated buffer for which size is also controlled
by the user via the compressed file header. This allows heap corruption with
user-controlled data. Potentially this means arbitrary code execution for the
processes that utilize the vulnerable function - one example is xbstream with
—decompress flag.
Steps to reproduce:
bytes.
less than 0x10000, for example set to 0x1000 instead.
test: if((*source & 1) == 1)
0x42 0x43 0x44
gdb and observe the corruption: xbstream —decompress -x < Demo.xbstream
All new code of the whole pull request, including one or several
files that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.