Merge pull request #142 from darrenldl/dev

Changed encode mode defaults, version bump, code refactoring

CHANGELOG.md

@@ -1,3 +1,11 @@
+# Changelog
+
+## 4.0.0
+
+- Changed "Uid" to "UID" in encode help messages for consistency
+- Changed default archiving options
+  - Changed from using SBX version 1 to using SBX version 17 with data parity ratio of 10:2 and burst error resistance level of 20 by default
+
 ## 3.0.0
 
 - Changed decode mode to use only file portion of stored file name in SBX container

Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name    = "blkar"
-version = "3.0.0"
+version = "4.0.0"
 authors = ["Darren Ldl <darrenldldev@gmail.com>"]
 edition = "2018"
 build   = "build.rs"

Makefile

@@ -0,0 +1,19 @@
+SRCFILES = src/*.rs \
+	src/bin/*.rs \
+	src/rs_codec/*.rs \
+	src/sbx_block/*.rs
+
+.PHONY: all
+all : bin
+
+.PHONY: bin
+bin :
+	cargo build
+
+.PHONY: format
+format :
+	rustfmt $(SRCFILES)
+
+.PHONY: test
+test :
+	cd tests && ./dev_tests.sh

README.md

@@ -6,45 +6,46 @@
 [![Coverage Status](https://coveralls.io/repos/github/darrenldl/blockyarchive/badge.svg?branch=master)](https://coveralls.io/github/darrenldl/blockyarchive?branch=master)
 [![Crates](https://img.shields.io/crates/v/blkar.svg)](https://crates.io/crates/blkar)
 [![dependency status](https://deps.rs/repo/github/darrenldl/blockyarchive/status.svg)](https://deps.rs/repo/github/darrenldl/blockyarchive)
-[![ko-fi](https://www.ko-fi.com/img/donate_sm.png)](https://ko-fi.com/H2H7OXZN)
 [![Gitter chat](https://badges.gitter.im/blockyarchive/gitter.png)](https://gitter.im/blockyarchive/community)
 
 [Documentation](https://github.com/darrenldl/blockyarchive/wiki)
 
-Blockyarchive/blkar (formerly rust-SeqBox) is a comprehensive utility for creating, rescuing, and general handling of SeqBox archives, with optional forward error correction.
+Blockyarchive/blkar (pronounced "bloc-kar") is a comprehensive utility for creating, rescuing, and general handling of SeqBox archives, with optional forward error correction
 
-SeqBox is a single-file archive format designed by [Marco Pontello](https://github.com/MarcoPon) that facilitates sector level data recovery for when file system metadata is corrupted/missing, while the archive itself still exists as a normal file on file system.
+SeqBox is a single-file archive format designed by [Marco Pontello](https://github.com/MarcoPon) that facilitates sector level data recovery for when file system metadata is corrupted/missing, while the archive itself still exists as a normal file on file system
 
-Please visit the official [SeqBox](https://github.com/MarcoPon/SeqBox) repo for the original implementation and technical details on this.
+Please visit the official [SeqBox](https://github.com/MarcoPon/SeqBox) repo for the original implementation and technical details on this
+
+Blockyarchive/blkar was formerly known as rust-SeqBox/rsbx prior to renaming
 
 ## Comparison to the original SeqBox implementation/design
 
-The original SeqBox implementation and format do not support repairing of data, only sector level recoverability.
+The original SeqBox implementation and format do not support repairing of data, only sector level recoverability
 
-Blockyarchive allows repairs to be made by adding forward error correction (Reed-Solomon erasure code) to extended versions of SeqBox format, and also allows arranging the blocks in a burst error resistant pattern.
+Blockyarchive allows repairs to be made by adding forward error correction (Reed-Solomon erasure code) to extended versions of SeqBox format, and also allows arranging the blocks in a burst error resistant pattern
 
-Blockyarchive is also more robust compared to the original SeqBox implementation, as it does not assume the SBX container to be well formed, and makes as few assumptions about the SBX container as possible, if at all.
+Blockyarchive is also more robust compared to the original SeqBox implementation, as it does not assume the SBX container to be well formed, and makes as few assumptions about the SBX container as possible
 
-blkar is overall based around [osbx](https://github.com/darrenldl/ocaml-SeqBox), but much more optimized.
+blkar is overall based around [osbx](https://github.com/darrenldl/ocaml-SeqBox), but much more optimized
 
 ## Features overall
 
 - Data recovery that does not depend on file system metadata (sector level recovery)
   - This allows data recovery even when data is fragmented and out of order
 - Supports error correction (via Reed-Solomon erasure code)
-- Supports burst sector error resistance
+- Supports burst (sector) error resistance
 - JSON mode
-  - Output information in JSON format instead of human readable text
+  - Outputs information in JSON format instead of human readable text, easy integration with scripts
 
 ## Goals
 
-As blkar is to be used largely as a backup utility, security/robustness of the code will be prioritised over apparent performance.
+As blkar is to be used largely as a backup utility, security/robustness of the code will be prioritised over apparent performance
 
 ## Status
 
 This project has reached its intended feature completeness, so no active development for new features will occur. However, this project is still actively looked after, i.e. I will respond to PRs, issues, and emails, will consider feature requests, respond to bug reports quickly, and so on.
 
-In other words, this is a completed project with respect to its original scope, but it is not abandoned.
+In other words, this is a completed project with respect to its original scope, but it is not abandoned
 
 ## Getting started
 
@@ -60,6 +61,10 @@ cargo install blkar
 
 The [wiki](https://github.com/darrenldl/blockyarchive/wiki) contains comprehensive guides and resources.
 
+## Note on Rust to Bash ratio
+
+Just to avoid confusion, blkar is written purely in Rust, Bash is only used to write tests
+
 ## Got a question?
 
 Feel free to join the [Gitter chat](https://gitter.im/blockyarchive/community) if you've got a question. You can email me directly as well.
@@ -84,7 +89,7 @@ I would like to thank [Marco](https://github.com/MarcoPon) (the official SeqBox
 
 I would like to thank [Ming](https://github.com/mdchia/) for his feedback on the documentation, UX design, and several other general aspects of the osbx project, of which most of the designs are carried over to blkar, and also his further feedback on this project as well.
 
-The design of the readable rate in progress report text is copied from [Arch Linux pacman](https://wiki.archlinux.org/index.php/Pacman)'s progress bar design.
+The design of the readable rate in progress report text is copied from [Arch Linux pacman](https://wiki.archlinux.org/index.php/Pacman)'s progress bar design
 
 The design of block set interleaving arrangement in RS enabled versions is heavily inspired by [Thanassis Tsiodras's design of RockFAT](https://www.thanassis.space/RockFAT.html). The interleaving provides resistance against burst sector errors.
 

src/bin/main.rs

@@ -4,7 +4,7 @@ use clap::*;
 extern crate blkar_lib;
 use blkar_lib::*;
 
-fn real_main () -> i32 {
+fn real_main() -> i32 {
     let matches = App::new("blkar")
         .version(env!("CARGO_PKG_VERSION"))
         .author("Darren Ldl <darrenldldev@gmail.com>")
@@ -19,31 +19,23 @@ fn real_main () -> i32 {
         .subcommand(cli_calc::sub_command())
         .get_matches();
 
-    if      let Some(matches) = matches.subcommand_matches("encode") {
+    if let Some(matches) = matches.subcommand_matches("encode") {
         cli_encode::encode(matches)
-    }
-    else if let Some(matches) = matches.subcommand_matches("decode") {
+    } else if let Some(matches) = matches.subcommand_matches("decode") {
         cli_decode::decode(matches)
-    }
-    else if let Some(matches) = matches.subcommand_matches("rescue") {
+    } else if let Some(matches) = matches.subcommand_matches("rescue") {
         cli_rescue::rescue(matches)
-    }
-    else if let Some(matches) = matches.subcommand_matches("show") {
+    } else if let Some(matches) = matches.subcommand_matches("show") {
         cli_show::show(matches)
-    }
-    else if let Some(matches) = matches.subcommand_matches("repair") {
+    } else if let Some(matches) = matches.subcommand_matches("repair") {
         cli_repair::repair(matches)
-    }
-    else if let Some(matches) = matches.subcommand_matches("check") {
+    } else if let Some(matches) = matches.subcommand_matches("check") {
         cli_check::check(matches)
-    }
-    else if let Some(matches) = matches.subcommand_matches("sort") {
+    } else if let Some(matches) = matches.subcommand_matches("sort") {
         cli_sort::sort(matches)
-    }
-    else if let Some(matches) = matches.subcommand_matches("calc") {
+    } else if let Some(matches) = matches.subcommand_matches("calc") {
         cli_calc::calc(matches)
-    }
-    else {
+    } else {
         exit_with_msg!(ok json_printer::JSONPrinter::new(false, output_channel::OutputChannel::Stdout)
                        => "Invoke with -h or --help for help message\n");
     }

src/cli_calc.rs

@@ -147,12 +147,12 @@ pub fn calc<'a>(matches: &ArgMatches<'a>) -> i32 {
         if burst == 0 {
             print_block!(
                 "    The container can tolerate {} SBX block corruptions", par;
-                "    in any block set.";
+                "    in each block set.";
                 "";
                 "    A block set consists of {} blocks ({} bytes).", (data + par), (data + par) * block_size;
                 "";
                 "    In total, {} blocks({} bytes) may be corrupted in", par, par * block_size;
-                "    any block set.";
+                "    each block set.";
             );
         } else {
             if burst == 1 {
@@ -169,7 +169,7 @@ pub fn calc<'a>(matches: &ArgMatches<'a>) -> i32 {
             let super_block_set_size = (data + par) * burst;
 
             print_block!("    The container can tolerate {} burst SBX block corruptions in", par;
-                         "    any super block set ({} interleaved block sets).", burst;
+                         "    each super block set ({} interleaved block sets).", burst;
                          "";
                          "    A block set consists of {} blocks ({} bytes).", block_set_size, block_set_size * block_size;
                          "";
@@ -178,10 +178,10 @@ pub fn calc<'a>(matches: &ArgMatches<'a>) -> i32 {
                          "    Each burst error may be up to {} blocks ({} bytes) in size.", burst, burst * block_size;
                          "";
                          "    In total, {} sets of {} consecutive blocks ({} bytes) may be", par, burst, burst * block_size;
-                         "    corrupted in any super block set.";
+                         "    corrupted in each super block set.";
                          "";
                          "    The sets of corrupted blocks may be connected as well, so the";
-                         "    largest single burst error that can be tolerated in any super";
+                         "    largest single burst error that can be tolerated in each super";
                          "    block set is {} blocks({} bytes) in size.", par * burst, par * burst * block_size;
                          "";
                          "    Note that the actual tolerance depends on the behaviour of";

src/cli_decode.rs

@@ -17,6 +17,7 @@ pub fn sub_command<'a, 'b>() -> App<'a, 'b> {
         .arg(in_file_arg().help("SBX container to decode"))
         .arg(out_arg().help(
             "Decoded file name. Supply - to use stdout as output. Use ./- for files named -.
+If output is stdout, progress text is outputted to stderr instead.
 If OUT is not provided, then the original file name stored in the SBX container
 (STOREDNAME) is used if present (only the file part of STOREDNAME is used). If
 OUT is provided and is a directory, then the output file is stored as OUT/STOREDNAME

src/cli_encode.rs

@@ -52,7 +52,7 @@ This means this option has no effect for version 17, 18, 19.",
         .arg(sbx_version_arg())
         .arg(only_pick_uid_arg().long("uid").help(
             "Alternative file UID in hex (by default UID is randomly generated).
-Uid must be exactly 6 bytes (12 hex digits) in length.",
+UID must be exactly 6 bytes (12 hex digits) in length.",
         ))
         .arg(rs_data_arg())
         .arg(rs_parity_arg())
@@ -93,21 +93,7 @@ pub fn encode<'a>(matches: &ArgMatches<'a>) -> i32 {
         }
     }
 
-    let version = get_version!(matches, json_printer);
-
-    let data_par_burst = if ver_uses_rs(version) {
-        // deal with RS related options
-        let data_shards = get_data_shards!(matches, version, json_printer);
-        let parity_shards = get_parity_shards!(matches, version, json_printer);
-
-        check_data_parity_shards!(data_shards, parity_shards, json_printer);
-
-        let burst = get_burst_or_zero!(matches, json_printer);
-
-        Some((data_shards, parity_shards, burst))
-    } else {
-        None
-    };
+    let (version, data_par_burst) = get_ver_data_par_burst_w_defaults!(matches, json_printer);
 
     let in_file = get_in_file!(accept_stdin matches, json_printer);
     let out = match matches.value_of("out") {

src/cli_macros.rs

@@ -257,6 +257,42 @@ macro_rules! get_parity_shards {
     }}
 }
 
+macro_rules! get_ver_data_par_burst_w_defaults {
+    (
+        $matches:expr, $json_printer:expr
+    ) => {{
+        use crate::sbx_specs::string_to_ver;
+        match $matches.value_of("sbx_version") {
+            None => (Version::V17, Some((10, 2, 20))),
+            Some(x) => {
+                let version =
+                    match string_to_ver(&x) {
+                        Ok(v)   => v,
+                        Err(()) => {
+                            exit_with_msg!(usr $json_printer => "Invalid SBX version");
+                        }
+                    };
+
+                let data_par_burst = if ver_uses_rs(version) {
+                    // deal with RS related options
+                    let data_shards = get_data_shards!($matches, version, $json_printer);
+                    let parity_shards = get_parity_shards!($matches, version, $json_printer);
+
+                    check_data_parity_shards!(data_shards, parity_shards, $json_printer);
+
+                    let burst = get_burst_or_zero!($matches, $json_printer);
+
+                    Some((data_shards, parity_shards, burst))
+                } else {
+                    None
+                };
+
+                (version, data_par_burst)
+            }
+        }
+    }}
+}
+
 macro_rules! check_data_parity_shards {
     (
         $data_shards:expr, $parity_shards:expr, $json_printer:expr
@@ -287,7 +323,7 @@ macro_rules! get_burst_or_zero {
         match $matches.value_of("burst") {
             None    => 0,
             Some(x) => {
-                match usize::from_str(&x) {
+                match usize::from_str(x) {
                     Ok(x)  => x,
                     Err(_) => {
                         exit_with_msg!(usr $json_printer => "Failed to parse burst error resistance level");
@@ -326,7 +362,7 @@ macro_rules! get_burst_opt {
         match $matches.value_of("burst") {
             None    => None,
             Some(x) => {
-                match usize::from_str(&x) {
+                match usize::from_str(x) {
                     Ok(x)  => Some(x),
                     Err(_) => {
                         exit_with_msg!(usr $json_printer => "Failed to parse burst error resistance level");

src/cli_utils.rs

@@ -228,12 +228,14 @@ pub fn sbx_version_arg<'a, 'b>() -> Arg<'a, 'b> {
         .help(
             "SBX container version, one of :
                     | SBX block size | Reed-Solomon | Burst error resistance |
-(default)  1        |      512 bytes |  not enabled |          not supported |
+           1        |      512 bytes |  not enabled |          not supported |
            2        |      128 bytes |  not enabled |          not supported |
            3        |     4096 bytes |  not enabled |          not supported |
-          17 (0x11) |      512 bytes |      enabled |              supported |
+(default) 17 (0x11) |      512 bytes |      enabled |              supported |
           18 (0x12) |      128 bytes |      enabled |              supported |
-          19 (0x13) |     4096 bytes |      enabled |              supported |",
+          19 (0x13) |     4096 bytes |      enabled |              supported |
+
+Details of default option : sbx-version=17, rs-data=10, rs-parity=2, burst=20",
         )
 }
 

src/decode_core.rs

@@ -668,13 +668,9 @@ pub fn decode(
         return_if_ref_not_meta!(ref_block_pos, ref_block, "decode");
     }
 
-    let data_par_shards = if rs_enabled {
-        Some((
-            get_RSD_from_ref_block!(ref_block_pos, ref_block, "decode"),
-            get_RSP_from_ref_block!(ref_block_pos, ref_block, "decode"),
-        ))
-    } else {
-        None
+    let data_par_shards = match data_par_burst {
+        Some((data, parity, _)) => Some((data, parity)),
+        None => None,
     };
 
     let version = ref_block.get_version();

src/encode_core.rs

@@ -665,7 +665,12 @@ pub fn encode_file(param: &Param) -> Result<Stats, Error> {
     reporter.stop();
 
     stats.lock().unwrap().in_file_size = data_bytes_encoded;
-    stats.lock().unwrap().out_file_size = writer.get_file_size()?;
+    stats.lock().unwrap().out_file_size = file_utils::from_orig_file_size::calc_container_size(
+        param.version,
+        Some(param.meta_enabled),
+        param.data_par_burst,
+        data_bytes_encoded,
+    );
 
     let stats = stats.lock().unwrap().clone();
 

tests/clean.sh

@@ -1,7 +1,8 @@
 #!/bin/bash
 
-if [[ $PWD != */tests ]]; then
-  cd tests
+if [[ $PWD != */blockyarchive/tests ]]; then
+  echo "Please invoke clean.sh in the tests directory"
+  exit 1
 fi
 
 tests=(