Skip to content
This repository has been archived by the owner on Jun 7, 2022. It is now read-only.

Commit

Permalink
Merge pull request #93 from darrenldl/dev
Browse files Browse the repository at this point in the history
Multihash overhaul, help/warning messages improvement
  • Loading branch information
darrenldl committed Dec 12, 2018
2 parents adc7797 + 930e4e0 commit f38eb92
Show file tree
Hide file tree
Showing 19 changed files with 690 additions and 203 deletions.
4 changes: 2 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@ script:
- rm -rf target/cov
- bash install_kcov.sh
- bash cov.sh
- bash cov_tests/dev_tests.sh
- bash tests/dev_tests.sh
- bash cov_tests/dev_tests.sh

addons:
apt:
Expand All @@ -67,7 +67,7 @@ addons:
- libiberty-dev

after_success:
- bash kcov_run_tests_and_upload_reports.sh
- bash kcov_upload_reports.sh

before_deploy:
- sh ci/before_deploy.sh
Expand Down
138 changes: 107 additions & 31 deletions Cargo.lock

Large diffs are not rendered by default.

3 changes: 2 additions & 1 deletion Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -45,14 +45,15 @@ doc = false

[dependencies]
reed-solomon-erasure = { version = "3.1", features = ["pure-rust"] }
ring = "0.13.2"
blake2_c = "0.3"
rand = "0.6.1"
smallvec = "0.6"
nom = "4.0"
chrono = "0.4"
clap = "2.30.0"
ctrlc = "3.1"
sha-1 = "0.8.1"
sha2 = "0.8.0"

[dev-dependencies]
quickcheck = "0.7"
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ The design of block set interleaving arrangement in RS enabled versions is heavi

#### Libcrc code

The crcccitt code is translated from the C implementation in [libcrc](https://github.com/lammertb/libcrc) and are under the same MIT License as used by libcrc and as stated in libcrc source code, the license text of the crcccitt.c is copied over to `crcccitt/build.rs`, `crcccitt/src/lib.rs`, `build.rs` and `src/crc-ccitt.rs` as well
The crcccitt code is translated from the C implementation in [libcrc](https://github.com/lammertb/libcrc) and are under the same MIT License as used by libcrc and as stated in libcrc source code, the license text of the crcccitt.c is copied over to `crcccitt/build.rs`, `crcccitt/src/lib.rs`, `build.rs` and `src/crc_ccitt.rs` as well

The C source code of crcccitt copied directly from libcrc are under the MIT License as used by libcrc, the files are in ```libcrc_crcccitt```

Expand Down
2 changes: 1 addition & 1 deletion SBX_FORMAT.md
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ Supported crypto hashes since 1.0.0 are

Metadata block (block 0) can be disabled

## For versions : 17(0x11), 18(0x12), 19(0x13)
## For versions : 17 (0x11), 18 (0x12), 19 (0x13)

Overall similar to above specs.

Expand Down
18 changes: 17 additions & 1 deletion cov_tests/dev_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ cd cov_tests
test_failed=0

echo "Generating test data"
dd if=/dev/urandom of=dummy bs=$[1024 * 2] count=1 &>/dev/null
./gen_dummy.sh
# truncate -s 10m dummy

# version tests
Expand All @@ -25,6 +25,22 @@ if [[ $? != 0 ]]; then
test_failed=$[$test_failed+1]
fi

echo "========================================"
echo "Starting version tests (stdin as encode input)"
echo "========================================"
./version_tests_encode_stdin.sh
if [[ $? != 0 ]]; then
test_failed=$[$test_failed+1]
fi

echo "========================================"
echo "Starting version tests (stdout as decode output)"
echo "========================================"
./version_tests_decode_stdout.sh
if [[ $? != 0 ]]; then
test_failed=$[$test_failed+1]
fi

# nometa tests
echo "========================================"
echo "Starting nometa tests"
Expand Down
2 changes: 2 additions & 0 deletions cov_tests/gen_dummy.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
#!/bin/bash
dd if=/dev/urandom of=dummy bs=$[1024 * 2] count=1 &>/dev/null
6 changes: 3 additions & 3 deletions cov_tests/version_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ for ver in ${VERSIONS[*]}; do
fi
done

# Check all
# Check all of them
for ver in ${VERSIONS[*]}; do
echo -n "Checking version $ver container"
output=$(kcov_rsbx check --json --verbose dummy$ver.sbx)
Expand Down Expand Up @@ -76,9 +76,9 @@ for ver in ${VERSIONS[*]}; do
echo -n "Comparing decoded version $ver container data to original"
cmp dummy dummy$ver
if [[ $? == 0 ]]; then
echo "==> Okay"
echo " ==> Okay"
else
echo "==> NOT okay"
echo " ==> NOT okay"
exit_code=1
fi
done
Expand Down
86 changes: 86 additions & 0 deletions cov_tests/version_tests_decode_stdout.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
#!/bin/bash

source kcov_rsbx_fun.sh

exit_code=0

VERSIONS=(1 2 3 17 18 19)

# Encode in all 6 versions
for ver in ${VERSIONS[*]}; do
echo -n "Encoding in version $ver"
output=$(kcov_rsbx encode --json --sbx-version $ver -f dummy dummy$ver.sbx \
--rs-data 10 --rs-parity 2)
if [[ $(echo $output | jq -r ".error") != null ]]; then
echo " ==> Invalid JSON"
exit_code=1
fi
if [[ $(echo $output | jq -r ".stats.sbxVersion") == "$ver" ]]; then
echo " ==> Okay"
else
echo " ==> NOT okay"
exit_code=1
fi
done

# Check all of them
for ver in ${VERSIONS[*]}; do
echo -n "Checking version $ver container"
output=$(kcov_rsbx check --json --pv 2 --verbose dummy$ver.sbx 2>/dev/null)
if [[ $(echo $output | jq -r ".error") != null ]]; then
echo " ==> Invalid JSON"
exit_code=1
fi
if [[ $(echo $output | jq -r ".stats.numberOfBlocksFailedCheck") == 0 ]]; then
echo " ==> Okay"
else
echo " ==> NOT okay"
exit_code=1
fi
done

# Show all
for ver in ${VERSIONS[*]}; do
echo -n "Checking show output for $ver container"
output=$(kcov_rsbx show --json --pv 1 dummy$ver.sbx 2>/dev/null)
if [[ $(echo $output | jq -r ".error") != null ]]; then
echo " ==> Invalid JSON"
exit_code=1
fi
if [[ $(echo $output | jq -r ".blocks[0].sbxContainerVersion") == $ver ]]; then
echo " ==> Okay"
else
echo " ==> NOT okay"
exit_code=1
fi
done

# Decode all of them
for ver in ${VERSIONS[*]}; do
echo -n "Decoding version $ver container"
output=$(kcov_rsbx decode --json --verbose dummy$ver.sbx - 2>&1 > dummy$ver)
if [[ $(echo $output | jq -r ".error") != null ]]; then
echo " ==> Invalid JSON"
exit_code=1
fi
if [[ $(echo $output | jq -r ".stats.sbxVersion") == "$ver" ]]; then
echo " ==> Okay"
else
echo " ==> NOT okay"
exit_code=1
fi
done

# Compare to original file
for ver in ${VERSIONS[*]}; do
echo -n "Comparing decoded version $ver container data to original"
cmp dummy dummy$ver
if [[ $? == 0 ]]; then
echo " ==> Okay"
else
echo " ==> NOT okay"
exit_code=1
fi
done

exit $exit_code
87 changes: 87 additions & 0 deletions cov_tests/version_tests_encode_stdin.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
#!/bin/bash

source kcov_rsbx_fun.sh

exit_code=0

VERSIONS=(1 2 3 17 18 19)

# Encode in all 6 versions
for ver in ${VERSIONS[*]}; do
echo -n "Encoding in version $ver"
output=$(cat dummy | \
kcov_rsbx encode --json --sbx-version $ver -f - dummy$ver.sbx \
--rs-data 10 --rs-parity 2)
if [[ $(echo $output | jq -r ".error") != null ]]; then
echo " ==> Invalid JSON"
exit_code=1
fi
if [[ $(echo $output | jq -r ".stats.sbxVersion") == "$ver" ]]; then
echo " ==> Okay"
else
echo " ==> NOT okay"
exit_code=1
fi
done

# Check all of them
for ver in ${VERSIONS[*]}; do
echo -n "Checking version $ver container"
output=$(kcov_rsbx check --json --pv 2 --verbose dummy$ver.sbx 2>/dev/null)
if [[ $(echo $output | jq -r ".error") != null ]]; then
echo " ==> Invalid JSON"
exit_code=1
fi
if [[ $(echo $output | jq -r ".stats.numberOfBlocksFailedCheck") == 0 ]]; then
echo " ==> Okay"
else
echo " ==> NOT okay"
exit_code=1
fi
done

# Show all
for ver in ${VERSIONS[*]}; do
echo -n "Checking show output for $ver container"
output=$(kcov_rsbx show --json --pv 1 dummy$ver.sbx 2>/dev/null)
if [[ $(echo $output | jq -r ".error") != null ]]; then
echo " ==> Invalid JSON"
exit_code=1
fi
if [[ $(echo $output | jq -r ".blocks[0].sbxContainerVersion") == $ver ]]; then
echo " ==> Okay"
else
echo " ==> NOT okay"
exit_code=1
fi
done

# Decode all of them
for ver in ${VERSIONS[*]}; do
echo -n "Decoding version $ver container"
output=$(kcov_rsbx decode --json --verbose -f dummy$ver.sbx dummy$ver)
if [[ $(echo $output | jq -r ".error") != null ]]; then
echo " ==> Invalid JSON"
exit_code=1
fi
if [[ $(echo $output | jq -r ".stats.sbxVersion") == "$ver" ]]; then
echo " ==> Okay"
else
echo " ==> NOT okay"
exit_code=1
fi
done

# Compare to original file
for ver in ${VERSIONS[*]}; do
echo -n "Comparing decoded version $ver container data to original"
cmp dummy dummy$ver
if [[ $? == 0 ]]; then
echo " ==> Okay"
else
echo " ==> NOT okay"
exit_code=1
fi
done

exit $exit_code
File renamed without changes.
20 changes: 13 additions & 7 deletions src/cli_decode.rs
Original file line number Diff line number Diff line change
Expand Up @@ -48,16 +48,22 @@ pub fn decode<'a>(matches : &ArgMatches<'a>) -> i32 {
"Warning :";
"";
" Since output is stdout, rsbx can only output data chunks in the";
" same order as the stored SBX blocks.";
" anticipated encoding order.";
"";
" In other words, if the SBX blocks are not correctly ordered, then";
" the output data will be incorrect.";
" For versions with no RS enabled (version 1, 2, 3), this means rsbx";
" outputs data chunks in the same order as the blocks are stored.";
"";
" You may fix this by sorting the SBX container using the rsbx sort";
" command before decoding to stdout.";
" For versions with RS enabled (version 17, 18, 19), this means rsbx";
" first guesses the burst resistance level, then reads using the block";
" set interleaving pattern and outputs the data chunks.";
"";
" rsbx also tries to avoid outputting the data padding at the end of";
" the container at a best effort basis, but may fail.";
" rsbx also tries to strip the data padding at the end of the container";
" at a best effort basis, but does not provide any guarantees.";
"";
" If the ordering matches the anticipated ordering, output of rsbx to";
" stdout should match the one produced in output to file mode. If the";
" ordering is not as anticipated, you may fix it by sorting the SBX";
" container using the rsbx sort command first.";
"";
);
}
Expand Down

0 comments on commit f38eb92

Please sign in to comment.