Skip to content

Windows 32/64 binary,HW accelerated, ESXi, Linux, Free/Open BSD

Pre-release
Pre-release
Compare
Choose a tag to compare
@fcorbelli fcorbelli released this 25 Jan 11:44
· 205 commits to main since this release
57.1
1f3acfa
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

This is the first release (to be tested) of the new 57 series

Like any first release it has no bugs, but features :)

Remember: the more feedback (even negative) I receive, the greater the likelihood of improving zpaqfranz. And do no forget, please, the star on github or a review on sourceforge :)

The main difference is the internal refactoring (which can cause subtle problems in parameter/switch recognition), and especially the inclusion of a new metadata storage "package"
The new V3 (testable for now with whirlpool and highway) stores additional useful information (or rather will) and in the future also some sort of posix-style data
In short - to summarize - facilitate restoration to *nix of symlinks, proprietary users etc, similar to tar

So externally the changes look modest, but internally they are numerous

New hashers that can be used inside archive: whirlpool, highway 64/128/256

In addition to the supported control hashes, within the archives, such as XXHASH|SHA-1|SHA-2|SHA-3|MD5|XXH3|BLAKE3, you can now choose

  • whirlpool. It is a "slow" hash that creates very large footprints, but it is based on a completely different technology than the others. I like it very much
  • highway. It is a hash developed by two very good programmers the github which is actually not designed for use with large amounts of data (like zpaqfranz), but rather for (relatively) "small" packet indexing. In the case of zpaqfranz there are 3 different "versions" (actually it is the same, so there is no difference in speed) for 64, 128 and 256 bits. So it is (as you can understand) most useful for quick debugging with different-length hashes. The implementation is "straight" C (no AVX2 acceleration etc), and is not tested (at present) for use on BIG ENDIAN or sparc or "strange" systems. In fact a debug tool
zpaqfranz a z:\1.zpaq c:\nz -whirlpool
zpaqfranz a z:\1.zpaq c:\nz -highway64
zpaqfranz a z:\1.zpaq c:\nz -highway128
zpaqfranz a z:\1.zpaq c:\nz -highway256
zpaqfranz l z:\1.zpaq -checksum

The dir command is now better than... dir

As is well known, or maybe not, if the zpaqfranz executable is called "dir" it works, roughly, like the Windows dir
I use it so much on Linux and FreeBSD where the ls command doesn't look anything like what is needed for a storage manager (you need numerous other commands, hard to remember concatenations etc)

zpaqfranz dir c:\*.cpp /s /os
dir c:\*.cpp /s /os -n 100

will show all the "*.cpp" into c:\ (with recursion /s), ordered by size (/os) and limit at 100 (- n 100)

zfsbackup a bit evolved

During use with simple datasets, so far, zfsbackup seems to work better than expected. Clearly this is referred to systems with zfs (linux+openzfs, FreeBSD, Solaris)
The -kill switch will delete temporary files (otherwise you need to manually "purge" the /tmp folder)

sparc64: -DALIGNMALLOC

For sparc64 an experimental switch to try to align malloc(): must be used at compile time

Haiku OS https://www.haiku-os.org/

Yep, zpaq 7.15 is already in Haiku OS :)
zpaqfranz, not very tested, can be compiled on Haiku R1/beta4, 64 bit (gcc 11.2.0), hrev56721 (maybe the -pthread is redundand, but not a big deal)

g++ -O3 -Dunix zpaqfranz.cpp -o zpaqfranz  -pthread -static

TrueNAS

It is an appliance based on FreeBSD 13x, which, however, lacks the compiler.
zpaqfranz can run inside a GUI-made jail, or outside (i.e. in the normal /usr/local/bin). The second case, of course, enable any function (including zfs backups), but have to be "injected" manually (with a SSH session, for example). Maybe I'll do a little HOW-TO

Comment on the source

The curious will see that (partial) refactoring is "strange," as it does not use very convenient features (e.g., RTTI) that would make it more compact and elegant. This is because of the inability to get "modern" compilers, in short for backward compatibility They will also notice that the handling of Boolean flags is peculiar. The reason is of performance within highly CPU-bound loops as in zpaqfranz. They will note that sometimes maps are used where **unordered_**map would be more efficient. But I can't, because they simply don't exist on certain systems (!). In short, it is in best tradeoff I have found (so far) between conciseness, maintainability, and breadth of supported platforms. Sometimes even to_string or atoi does not exists :)

The binaries

In this very first release there are

  • zpaqfranz.exe (64bit Windows)
  • zpaqfranz32.exe (32bit Windows)
  • zpaqfranzhw.exe (64bit Windows w/HW SHA-1 acceleration via -hw switch, usually for AMD)
  • zpaqfranz_esxi (32bit vSphere "maybe-will-run")
  • zpaqfranz_freebsd (64bit statically linked)
  • zpaqfranz_linux (64bit statically linked)
  • zpaqfranz_openbsd (64bit statically linked)
  • zpaqfranz_qnap (QNAP NAS TS-431P3 Annapurna AL314)

Of course the "right" way I recommend is to download the source and compile directly from scratch.

I attach them because they are convenient for me to do quick tests on as many systems as I can get

Download zpaqfranz

Assets 11