Skip to content

/ xoreos-tools

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

KEYBIF: Add initial keybif tool #27

Closed
wants to merge 11 commits into from
Closed

KEYBIF: Add initial keybif tool #27

wants to merge 11 commits into from

Conversation

@Nostritius
Copy link
Contributor

@Nostritius Nostritius commented Oct 14, 2018

This PR adds a new tool to create a key/bif archive structure, also with support for bzf files.
@Nostritius Nostritius force-pushed the Nostritius:tools_keybif branch from 247d644 to c51f612 Oct 20, 2018
@DrMcCoy
Copy link
Member

@DrMcCoy DrMcCoy commented Nov 3, 2018

ASan throws in the BIFWriter and BZFWriter tests:

FAIL: tests/aurora/test_bifwriter
=================================

Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from BIFWriter
[ RUN      ] BIFWriter.writeFile
=================================================================
==7507==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6160000044ef at pc 0x7f7e82125933 bp 0x7ffe4b308a30 sp 0x7ffe4b3081d8
READ of size 624 at 0x6160000044ef thread T0
    #0 0x7f7e82125932 in __interceptor_strcmp /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:377
    #1 0x55db10abff08 in testing::internal::String::CStringEquals(char const*, char const*) tests/googletest/src/gtest.cc:899
    #2 0x55db10bbd942 in testing::internal::String::CStringEquals(char const*, char const*) tests/googletest/src/gtest.cc:897
    #3 0x55db10bbd942 in testing::internal::CmpHelperSTREQ(char const*, char const*, char const*, char const*) tests/googletest/src/gtest.cc:1491
    #4 0x55db10a9d9a4 in BIFWriter_writeFile_Test::TestBody() tests/aurora/bifwriter.cpp:430
    #5 0x55db10c4f037 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #6 0x55db10c4f037 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #7 0x55db10ba6e23 in testing::Test::Run() tests/googletest/src/gtest.cc:2474
    #8 0x55db10ba7e6b in testing::Test::Run() tests/googletest/src/gtest.cc:2466
    #9 0x55db10ba7e6b in testing::TestInfo::Run() tests/googletest/src/gtest.cc:2656
    #10 0x55db10ba93ad in testing::TestCase::Run() tests/googletest/src/gtest.cc:2774
    #11 0x55db10badbdc in testing::internal::UnitTestImpl::RunAllTests() tests/googletest/src/gtest.cc:4649
    #12 0x55db10c52097 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #13 0x55db10c52097 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #14 0x55db10bb05ea in testing::UnitTest::Run() tests/googletest/src/gtest.cc:4257
    #15 0x55db10a9b56f in RUN_ALL_TESTS() tests/googletest/include/gtest/gtest.h:2239
    #16 0x55db10a9b56f in main tests/googletest/src/gtest_main.cc:37
    #17 0x7f7e7f49dae6 in __libc_start_main ../csu/libc-start.c:308
    #18 0x55db10a9b959 in _start (/home/drmccoy/projects/xoreos/tools/tests/aurora/test_bifwriter+0x3d7959)

0x6160000044ef is located 0 bytes to the right of 623-byte region [0x616000004280,0x6160000044ef)
allocated by thread T0 here:
    #0 0x7f7e8215c390 in operator new[](unsigned long) /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/asan/asan_new_delete.cc:93
    #1 0x55db10a9d749 in BIFWriter_writeFile_Test::TestBody() tests/aurora/bifwriter.cpp:427
    #2 0x55db10c4f037 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #3 0x55db10c4f037 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #4 0x55db10ba6e23 in testing::Test::Run() tests/googletest/src/gtest.cc:2474
    #5 0x55db10ba7e6b in testing::Test::Run() tests/googletest/src/gtest.cc:2466
    #6 0x55db10ba7e6b in testing::TestInfo::Run() tests/googletest/src/gtest.cc:2656
    #7 0x55db10ba93ad in testing::TestCase::Run() tests/googletest/src/gtest.cc:2774
    #8 0x55db10badbdc in testing::internal::UnitTestImpl::RunAllTests() tests/googletest/src/gtest.cc:4649
    #9 0x55db10c52097 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #10 0x55db10c52097 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #11 0x55db10bb05ea in testing::UnitTest::Run() tests/googletest/src/gtest.cc:4257
    #12 0x55db10a9b56f in RUN_ALL_TESTS() tests/googletest/include/gtest/gtest.h:2239
    #13 0x55db10a9b56f in main tests/googletest/src/gtest_main.cc:37
    #14 0x7f7e7f49dae6 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: heap-buffer-overflow /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:377 in __interceptor_strcmp
Shadow bytes around the buggy address:
  0x0c2c7fff8840: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff8850: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff8860: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff8870: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff8880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c2c7fff8890: 00 00 00 00 00 00 00 00 00 00 00 00 00[07]fa fa
  0x0c2c7fff88a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff88b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff88c0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff88d0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff88e0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==7507==ABORTING
FAIL tests/aurora/test_bifwriter (exit status: 1)

FAIL: tests/aurora/test_bzfwriter
=================================

Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from BZFWriter
[ RUN      ] BZFWriter.writeFile
=================================================================
==7514==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x6160000047ef at pc 0x7f8c38cf8933 bp 0x7ffd22449ac0 sp 0x7ffd22449268
READ of size 624 at 0x6160000047ef thread T0
    #0 0x7f8c38cf8932 in __interceptor_strcmp /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:377
    #1 0x55ba38754248 in testing::internal::String::CStringEquals(char const*, char const*) tests/googletest/src/gtest.cc:899
    #2 0x55ba38851c82 in testing::internal::String::CStringEquals(char const*, char const*) tests/googletest/src/gtest.cc:897
    #3 0x55ba38851c82 in testing::internal::CmpHelperSTREQ(char const*, char const*, char const*, char const*) tests/googletest/src/gtest.cc:1491
    #4 0x55ba38732bb2 in BZFWriter_writeFile_Test::TestBody() tests/aurora/bzfwriter.cpp:428
    #5 0x55ba388e3377 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #6 0x55ba388e3377 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #7 0x55ba3883b163 in testing::Test::Run() tests/googletest/src/gtest.cc:2474
    #8 0x55ba3883c1ab in testing::Test::Run() tests/googletest/src/gtest.cc:2466
    #9 0x55ba3883c1ab in testing::TestInfo::Run() tests/googletest/src/gtest.cc:2656
    #10 0x55ba3883d6ed in testing::TestCase::Run() tests/googletest/src/gtest.cc:2774
    #11 0x55ba38841f1c in testing::internal::UnitTestImpl::RunAllTests() tests/googletest/src/gtest.cc:4649
    #12 0x55ba388e63d7 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #13 0x55ba388e63d7 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #14 0x55ba3884492a in testing::UnitTest::Run() tests/googletest/src/gtest.cc:4257
    #15 0x55ba38730bbf in RUN_ALL_TESTS() tests/googletest/include/gtest/gtest.h:2239
    #16 0x55ba38730bbf in main tests/googletest/src/gtest_main.cc:37
    #17 0x7f8c36070ae6 in __libc_start_main ../csu/libc-start.c:308
    #18 0x55ba38730fc9 in _start (/home/drmccoy/projects/xoreos/tools/tests/aurora/test_bzfwriter+0x3dbfc9)

0x6160000047ef is located 0 bytes to the right of 623-byte region [0x616000004580,0x6160000047ef)
allocated by thread T0 here:
    #0 0x7f8c38d2f390 in operator new[](unsigned long) /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/asan/asan_new_delete.cc:93
    #1 0x55ba38732957 in BZFWriter_writeFile_Test::TestBody() tests/aurora/bzfwriter.cpp:425
    #2 0x55ba388e3377 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #3 0x55ba388e3377 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #4 0x55ba3883b163 in testing::Test::Run() tests/googletest/src/gtest.cc:2474
    #5 0x55ba3883c1ab in testing::Test::Run() tests/googletest/src/gtest.cc:2466
    #6 0x55ba3883c1ab in testing::TestInfo::Run() tests/googletest/src/gtest.cc:2656
    #7 0x55ba3883d6ed in testing::TestCase::Run() tests/googletest/src/gtest.cc:2774
    #8 0x55ba38841f1c in testing::internal::UnitTestImpl::RunAllTests() tests/googletest/src/gtest.cc:4649
    #9 0x55ba388e63d7 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2402
    #10 0x55ba388e63d7 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) tests/googletest/src/gtest.cc:2438
    #11 0x55ba3884492a in testing::UnitTest::Run() tests/googletest/src/gtest.cc:4257
    #12 0x55ba38730bbf in RUN_ALL_TESTS() tests/googletest/include/gtest/gtest.h:2239
    #13 0x55ba38730bbf in main tests/googletest/src/gtest_main.cc:37
    #14 0x7f8c36070ae6 in __libc_start_main ../csu/libc-start.c:308

SUMMARY: AddressSanitizer: heap-buffer-overflow /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:377 in __interceptor_strcmp
Shadow bytes around the buggy address:
  0x0c2c7fff88a0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff88b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff88c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff88d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0c2c7fff88e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x0c2c7fff88f0: 00 00 00 00 00 00 00 00 00 00 00 00 00[07]fa fa
  0x0c2c7fff8900: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff8910: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff8920: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff8930: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
  0x0c2c7fff8940: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==7514==ABORTING
FAIL tests/aurora/test_bzfwriter (exit status: 1)
@Nostritius Nostritius force-pushed the Nostritius:tools_keybif branch from c51f612 to b73cce7 Nov 5, 2018
Nostritius added 3 commits Oct 14, 2018
@Nostritius
AURORA: Add a writer for KEY files
7f4dd2e
@Nostritius
TESTS: Add unit test for KEY writer
0c3c467
@Nostritius
AURORA: Add abstract KEY data writer interface
d09efb4
@Nostritius Nostritius force-pushed the Nostritius:tools_keybif branch from b73cce7 to 3115288 Nov 5, 2018
@Nostritius
Copy link
Contributor Author

@Nostritius Nostritius commented Nov 5, 2018

The Memory leaks should now be fixed
@DrMcCoy
DrMcCoy reviewed Nov 11, 2018
View changes
src/common/lzma.cpp Outdated
@@ -130,4 +131,67 @@ SeekableReadStream *decompressLZMA1(ReadStream &input, size_t inputSize, size_t
return new MemoryReadStream(outputData, outputSize, true);
}

SeekableReadStream * compressLZMA1(ReadStream &input, size_t inputSize, size_t &outputSize) {

This comment has been minimized.

Sign in to view
@DrMcCoy

DrMcCoy Nov 11, 2018
Member

SeekableReadStream *compressLZMA1

And we don't really need that &outputSize parameter, do we? The returned stream already knows its size.
@DrMcCoy
DrMcCoy reviewed Nov 11, 2018
View changes
src/keybif.cpp Outdated
};

bool parseCommandLine(const std::vector<Common::UString> &argv, int &returnValue,
Common::UString &keyfile, std::set<Common::UString> &files);

This comment has been minimized.

Sign in to view
@DrMcCoy

DrMcCoy Nov 11, 2018
Member

Broken alignment
@DrMcCoy
DrMcCoy reviewed Nov 11, 2018
View changes
src/keybif.cpp Outdated
}

bool parseCommandLine(const std::vector<Common::UString> &argv, int &returnValue,
Common::UString &keyfile, std::set<Common::UString> &files) {

This comment has been minimized.

Sign in to view
@DrMcCoy

DrMcCoy Nov 11, 2018
Member

Broken alignment
@DrMcCoy
DrMcCoy reviewed Nov 11, 2018
View changes
src/keybif.cpp
Parser parser(argv[0], "BioWare KEY/BIF archive packer",
"",
returnValue,
makeEndArgs(&archiveOpt, &filesOpt));

This comment has been minimized.

Sign in to view
@DrMcCoy

DrMcCoy Nov 11, 2018
Member

Broken alignment
@DrMcCoy
DrMcCoy reviewed Nov 11, 2018
View changes
man/keybif.1 Outdated
.Sh DESCRIPTION
.Nm
packs files together into a BioWare KEY/BIF archive, found in many
BioWare Games as files with the extensions .key, .bif and .bzf where .bzf is

This comment has been minimized.

Sign in to view
@DrMcCoy

DrMcCoy Nov 11, 2018
Member

Comma before the "where"
@DrMcCoy
DrMcCoy reviewed Nov 11, 2018
View changes
man/keybif.1 Outdated
.It Ar keyfile
The .key file to create
.It Ar files
One or more files to pack together. When a .bif/.bzf file is occurs, the following

This comment has been minimized.

Sign in to view
@DrMcCoy

DrMcCoy Nov 11, 2018
Member

"is occurs"?
@Nostritius Nostritius force-pushed the Nostritius:tools_keybif branch from 3115288 to 417650b Nov 12, 2018
@Nostritius
Copy link
Contributor Author

@Nostritius Nostritius commented Nov 12, 2018

The issues should now be fixed
@DrMcCoy
Copy link
Member

@DrMcCoy DrMcCoy commented Nov 12, 2018

Please fix this warning too: https://travis-ci.org/xoreos/xoreos-tools/jobs/453918651#L2841
Nostritius added 8 commits Oct 14, 2018
@Nostritius
AURORA: Add a writer for BIF files
44cf1d4
@Nostritius
TESTS: Add unit test for BIF writer
c7dd0a0
@Nostritius
COMMON: Add lzma compression
137ef8f
@Nostritius
AURORA: Add writer for BZF files
a5603ec
@Nostritius
TESTS: Add unit test for BZF writer
66734a0
@Nostritius
KEYBIF: Add initial keybif tool
62a42ae
@Nostritius
DOCS: Add keybif README hint
bee2d83
@Nostritius
MAN: Add keybif tool man page
Loading status checks…
6a824bf
@Nostritius
Copy link
Contributor Author

@Nostritius Nostritius commented Nov 13, 2018

The warning should now be fixed
@Nostritius Nostritius force-pushed the Nostritius:tools_keybif branch from 417650b to 6a824bf Nov 13, 2018
@DrMcCoy
Copy link
Member

@DrMcCoy DrMcCoy commented Nov 13, 2018

Merged as 2ba8961...1ff924f, thanks! :)
@DrMcCoy DrMcCoy closed this Nov 13, 2018
@Nostritius Nostritius deleted the Nostritius:tools_keybif branch Nov 28, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@DrMcCoy DrMcCoy

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
@Nostritius @DrMcCoy