New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KEYBIF: Add initial keybif tool #27

Closed
wants to merge 11 commits into
base: master
from

Conversation

Projects
None yet
2 participants
@Nostritius
Contributor

Nostritius commented Oct 14, 2018

This PR adds a new tool to create a key/bif archive structure, also with support for bzf files.

@Nostritius Nostritius force-pushed the Nostritius:tools_keybif branch from 247d644 to c51f612 Oct 20, 2018

@DrMcCoy

This comment has been minimized.

Member

DrMcCoy commented Nov 3, 2018

ASan throws in the BIFWriter and BZFWriter tests:

FAIL: tests/aurora/test_bifwriter
=================================

Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from BIFWriter
[ RUN      ] BIFWriter.writeFile
=================================================================
==7507==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6160000044ef at pc 0x7f7e82125933 bp 0x7ffe4b308a30 sp 0x7ffe4b3081d8
READ of size 624 at 0x6160000044ef thread T0
    #0 0x7f7e82125932 in __interceptor_strcmp /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:377
    #1 0x55db10abff08 in testing::internal::String::CStringEquals(char const*, char const*) tests/googletest/src/gtest.cc:899
    #2 0x55db10bbd942 in testing::internal::String::CStringEquals(char const*, char const*) tests/googletest/src/gtest.cc:897
    #3 0x55db10bbd942 in testing::internal::CmpHelperSTREQ(char const*, char const*, char const*, char const*) tests/googletest/src/gtest.cc:1491
    #4 0x55db10a9d9a4 in BIFWriter_writeFile_Test::TestBody() tests/aurora/bifwriter.cpp:430
    #5 0x55db10c4f037 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #6 0x55db10c4f037 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #7 0x55db10ba6e23 in testing::Test::Run() tests/googletest/src/gtest.cc:2474
    #8 0x55db10ba7e6b in testing::Test::Run() tests/googletest/src/gtest.cc:2466
    #9 0x55db10ba7e6b in testing::TestInfo::Run() tests/googletest/src/gtest.cc:2656
    #10 0x55db10ba93ad in testing::TestCase::Run() tests/googletest/src/gtest.cc:2774
    #11 0x55db10badbdc in testing::internal::UnitTestImpl::RunAllTests() tests/googletest/src/gtest.cc:4649
    #12 0x55db10c52097 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #13 0x55db10c52097 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #14 0x55db10bb05ea in testing::UnitTest::Run() tests/googletest/src/gtest.cc:4257
    #15 0x55db10a9b56f in RUN_ALL_TESTS() tests/googletest/include/gtest/gtest.h:2239
    #16 0x55db10a9b56f in main tests/googletest/src/gtest_main.cc:37
    #17 0x7f7e7f49dae6 in __libc_start_main ../csu/libc-start.c:308
    #18 0x55db10a9b959 in _start (/home/drmccoy/projects/xoreos/tools/tests/aurora/test_bifwriter+0x3d7959)

0x6160000044ef is located 0 bytes to the right of 623-byte region [0x616000004280,0x6160000044ef)
allocated by thread T0 here:
    #0 0x7f7e8215c390 in operator new[](unsigned long) /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/asan/asan_new_delete.cc:93
    #1 0x55db10a9d749 in BIFWriter_writeFile_Test::TestBody() tests/aurora/bifwriter.cpp:427
    #2 0x55db10c4f037 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #3 0x55db10c4f037 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #4 0x55db10ba6e23 in testing::Test::Run() tests/googletest/src/gtest.cc:2474
    #5 0x55db10ba7e6b in testing::Test::Run() tests/googletest/src/gtest.cc:2466
    #6 0x55db10ba7e6b in testing::TestInfo::Run() tests/googletest/src/gtest.cc:2656
    #7 0x55db10ba93ad in testing::TestCase::Run() tests/googletest/src/gtest.cc:2774
    #8 0x55db10badbdc in testing::internal::UnitTestImpl::RunAllTests() tests/googletest/src/gtest.cc:4649
    #9 0x55db10c52097 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #10 0x55db10c52097 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #11 0x55db10bb05ea in testing::UnitTest::Run() tests/googletest/src/gtest.cc:4257
    #12 0x55db10a9b56f in RUN_ALL_TESTS() tests/googletest/include/gtest/gtest.h:2239
    #13 0x55db10a9b56f in main tests/googletest/src/gtest_main.cc:37
    #14 0x7f7e7f49dae6 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: heap-buffer-overflow /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:377 in __interceptor_strcmp
Shadow bytes around the buggy address:
  0x0c2c7fff8840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff8850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff8860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff8870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff8880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c2c7fff8890: 00 00 00 00 00 00 00 00 00 00 00 00 00[07]fa fa
  0x0c2c7fff88a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff88b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff88c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff88d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff88e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==7507==ABORTING
FAIL tests/aurora/test_bifwriter (exit status: 1)
FAIL: tests/aurora/test_bzfwriter
=================================

Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from BZFWriter
[ RUN      ] BZFWriter.writeFile
=================================================================
==7514==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6160000047ef at pc 0x7f8c38cf8933 bp 0x7ffd22449ac0 sp 0x7ffd22449268
READ of size 624 at 0x6160000047ef thread T0
    #0 0x7f8c38cf8932 in __interceptor_strcmp /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:377
    #1 0x55ba38754248 in testing::internal::String::CStringEquals(char const*, char const*) tests/googletest/src/gtest.cc:899
    #2 0x55ba38851c82 in testing::internal::String::CStringEquals(char const*, char const*) tests/googletest/src/gtest.cc:897
    #3 0x55ba38851c82 in testing::internal::CmpHelperSTREQ(char const*, char const*, char const*, char const*) tests/googletest/src/gtest.cc:1491
    #4 0x55ba38732bb2 in BZFWriter_writeFile_Test::TestBody() tests/aurora/bzfwriter.cpp:428
    #5 0x55ba388e3377 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #6 0x55ba388e3377 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #7 0x55ba3883b163 in testing::Test::Run() tests/googletest/src/gtest.cc:2474
    #8 0x55ba3883c1ab in testing::Test::Run() tests/googletest/src/gtest.cc:2466
    #9 0x55ba3883c1ab in testing::TestInfo::Run() tests/googletest/src/gtest.cc:2656
    #10 0x55ba3883d6ed in testing::TestCase::Run() tests/googletest/src/gtest.cc:2774
    #11 0x55ba38841f1c in testing::internal::UnitTestImpl::RunAllTests() tests/googletest/src/gtest.cc:4649
    #12 0x55ba388e63d7 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #13 0x55ba388e63d7 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #14 0x55ba3884492a in testing::UnitTest::Run() tests/googletest/src/gtest.cc:4257
    #15 0x55ba38730bbf in RUN_ALL_TESTS() tests/googletest/include/gtest/gtest.h:2239
    #16 0x55ba38730bbf in main tests/googletest/src/gtest_main.cc:37
    #17 0x7f8c36070ae6 in __libc_start_main ../csu/libc-start.c:308
    #18 0x55ba38730fc9 in _start (/home/drmccoy/projects/xoreos/tools/tests/aurora/test_bzfwriter+0x3dbfc9)

0x6160000047ef is located 0 bytes to the right of 623-byte region [0x616000004580,0x6160000047ef)
allocated by thread T0 here:
    #0 0x7f8c38d2f390 in operator new[](unsigned long) /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/asan/asan_new_delete.cc:93
    #1 0x55ba38732957 in BZFWriter_writeFile_Test::TestBody() tests/aurora/bzfwriter.cpp:425
    #2 0x55ba388e3377 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #3 0x55ba388e3377 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #4 0x55ba3883b163 in testing::Test::Run() tests/googletest/src/gtest.cc:2474
    #5 0x55ba3883c1ab in testing::Test::Run() tests/googletest/src/gtest.cc:2466
    #6 0x55ba3883c1ab in testing::TestInfo::Run() tests/googletest/src/gtest.cc:2656
    #7 0x55ba3883d6ed in testing::TestCase::Run() tests/googletest/src/gtest.cc:2774
    #8 0x55ba38841f1c in testing::internal::UnitTestImpl::RunAllTests() tests/googletest/src/gtest.cc:4649
    #9 0x55ba388e63d7 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #10 0x55ba388e63d7 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #11 0x55ba3884492a in testing::UnitTest::Run() tests/googletest/src/gtest.cc:4257
    #12 0x55ba38730bbf in RUN_ALL_TESTS() tests/googletest/include/gtest/gtest.h:2239
    #13 0x55ba38730bbf in main tests/googletest/src/gtest_main.cc:37
    #14 0x7f8c36070ae6 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: heap-buffer-overflow /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:377 in __interceptor_strcmp
Shadow bytes around the buggy address:
  0x0c2c7fff88a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff88b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff88c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff88d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff88e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c2c7fff88f0: 00 00 00 00 00 00 00 00 00 00 00 00 00[07]fa fa
  0x0c2c7fff8900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff8910: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff8920: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff8930: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff8940: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==7514==ABORTING
FAIL tests/aurora/test_bzfwriter (exit status: 1)

@Nostritius Nostritius force-pushed the Nostritius:tools_keybif branch from c51f612 to b73cce7 Nov 5, 2018

@Nostritius Nostritius force-pushed the Nostritius:tools_keybif branch from b73cce7 to 3115288 Nov 5, 2018

@Nostritius

This comment has been minimized.

Contributor

Nostritius commented Nov 5, 2018

The Memory leaks should now be fixed

@@ -130,4 +131,67 @@ SeekableReadStream *decompressLZMA1(ReadStream &input, size_t inputSize, size_t
return new MemoryReadStream(outputData, outputSize, true);
}
SeekableReadStream * compressLZMA1(ReadStream &input, size_t inputSize, size_t &outputSize) {

This comment has been minimized.

@DrMcCoy

DrMcCoy Nov 11, 2018

Member

SeekableReadStream *compressLZMA1

And we don't really need that &outputSize parameter, do we? The returned stream already knows its size.

};
bool parseCommandLine(const std::vector<Common::UString> &argv, int &returnValue,
Common::UString &keyfile, std::set<Common::UString> &files);

This comment has been minimized.

@DrMcCoy

DrMcCoy Nov 11, 2018

Member

Broken alignment

}
bool parseCommandLine(const std::vector<Common::UString> &argv, int &returnValue,
Common::UString &keyfile, std::set<Common::UString> &files) {

This comment has been minimized.

@DrMcCoy

DrMcCoy Nov 11, 2018

Member

Broken alignment

Parser parser(argv[0], "BioWare KEY/BIF archive packer",
"",
returnValue,
makeEndArgs(&archiveOpt, &filesOpt));

This comment has been minimized.

@DrMcCoy

DrMcCoy Nov 11, 2018

Member

Broken alignment

.Sh DESCRIPTION
.Nm
packs files together into a BioWare KEY/BIF archive, found in many
BioWare Games as files with the extensions .key, .bif and .bzf where .bzf is

This comment has been minimized.

@DrMcCoy

DrMcCoy Nov 11, 2018

Member

Comma before the "where"

.It Ar keyfile
The .key file to create
.It Ar files
One or more files to pack together. When a .bif/.bzf file is occurs, the following

This comment has been minimized.

@DrMcCoy

DrMcCoy Nov 11, 2018

Member

"is occurs"?

@Nostritius Nostritius force-pushed the Nostritius:tools_keybif branch from 3115288 to 417650b Nov 12, 2018

@Nostritius

This comment has been minimized.

Contributor

Nostritius commented Nov 12, 2018

The issues should now be fixed

@DrMcCoy

This comment has been minimized.

Member

DrMcCoy commented Nov 12, 2018

@Nostritius

This comment has been minimized.

Contributor

Nostritius commented Nov 13, 2018

The warning should now be fixed

@Nostritius Nostritius force-pushed the Nostritius:tools_keybif branch from 417650b to 6a824bf Nov 13, 2018

@DrMcCoy

This comment has been minimized.

Member

DrMcCoy commented Nov 13, 2018

Merged as 2ba8961...1ff924f, thanks! :)

@DrMcCoy DrMcCoy closed this Nov 13, 2018

@Nostritius Nostritius deleted the Nostritius:tools_keybif branch Nov 28, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment