v0.3.1

Not too much changed between v0.3.0 and v0.3.1, except for the addition of three new options in the command line.

• First, a memory limit option, so as to avoid bef killing your computer when it needs a lot of memory, instead just crashing.
• Second, a 'upgrade' option, that attempts to upgrade the size of the set of interleaved blocks into being approximately equal to the size given, or to the size of the input file if 0 is given. Useful when used in conjunction with minimize on small files, or with the seek option below on extremely large files.
• Third, a seek option, that is pretty much a hack in all honesty, that maps files to memory using mmap instead of using intermediate buffers. This is almost always slower than the default read/write dance, but is useful when you use some extremely large parameters that can't fit inside memory. It does have the catch of using an indeterminate amount of memory according to the kernel page cache (outside of my control) and having confounded performance characteristics, performing differently when under memory pressure, HDD vs SSD, etc. In some cases, when combined with the upgrade option, it can provide superior security than par2cmdline-turbo on extremely large files with better performance, but that is highly dependent on the environment and in most cases it performs worse.

v0.3.0

Bump in the major version, primarily for the deprecation of -s/--scan. Partially redid the deconstruct method to just scan into infinity rather than do the scan and skip dance. Also added support for SHA512, BLAKE2B, and CRC32C (from SSE4.2), in addition to preferring zlib-ng over zlib when its available. Really not much changed (despite the number of commits, in reality a lot of them were along the lines of 'oops').

Same as before, bef-full contains almost every dependency statically linked but is otherwise dynamically linked, bef-static has everything statically linked but with xxhash as its only dependency.

v0.2.4

Another performance release. This release decreases memory usage and performance by sharing global buffers instead of creating and freeing them each time, as well as removing redundant memcpys and buffers. Consequently, single threaded encoding performance almost doubled and multi threaded went up ~50%. Decoding performance only saw a modest ~10% improvement. By default the AVX2 accelerated fec-vand goes at ~1.5GiB/s on my laptop on a single thread. So the default has returned to being single threaded.

Also I've made some simple binaries (on Debian bullseye and Alpine 3.19) for folks that don't wanna compile it, but I recommend (especially in light of some recent events) to download the source tarballs and compile it yourself. Both i386 and amd64 versions are available.

bef-full have almost every dependency statically linked, except ISA-L and Jerasure as they depend on dlopen for liberasurecode. Standard stuff like libc and libgomp are not statically linked. This should support just about everything. The i386 version does not have BLAKE3 support.

bef-static is the absolute minimum, with the only dependency being xxhash, fully statically compiled on Alpine Linux.

v0.2.3

Lots changed between v0.2.2 and v0.2.3.

• OpenMP multithreading support was added for both construction and deconstruction. A -T/--threads flag was added to control specifically the number of threads bef should use.
• Minimize flag was added so you don't have to play with block sizes for small files
• CM256CC, OpenFEC, Leopard, and Wirehair support have been added as parity backends
• Multiple bugs, particularly with decoding, have been fixed

All in all, I say this is a pretty good release before attempting to add any format extensions. Future parity backends to look at would be Klaus' Reed Solomon package for Golang (though I have never written a line of Go in my life). Also of course the convolutional code support I haven't felt bothered to even try to implement.

v0.2.2

This version is mostly the same as before, as I haven't felt bothered to add new features to the format. Main differences are vastly improved performance, due to a couple optimizations in both the internal library and the stolen zfec code. For both the liberasurecode and zfec backends, throughput has roughly doubled, though mind you that is with AVX2 optimizations. Also added in a verbosity flag.

Current benchmarks (from my terrible laptop CPU) roughly are (from bef -c -p $preset -P $parity -i /dev/zero | pv >/dev/null), not including cauchy ones as they're roughly equal in performance anyways

• fec-vand: 870MiB/s
• fec-vand (paranoid preset): 470MiB/s
• jerasure-vand: 700MiB/s
• jerasure-vand (paranoid preset): 300MiB/s
• liberasurecode-vand: 600MiB/s
• liberasurecode-vand (paranoid preset): 70MiB/s
• intel-vand: 670MiB/s
• intel-vand (paranoid preset): 500MiB/s

Ideas for future release v0.3 would be to look back into the concept of convolutional code support, see if I can make a wrapper around GNU Radio or AFF3CT or something like that, and add full convolutional code support as the first extension to the format. This would provide protection against random noise (say every other byte has a bit flip) at the cost of a vastly larger file size, so it would be a good option for some folks.

Outside of that, I don't have many ideas for improvement, it's a pretty simple protocol and its supposed to be simple. The next best thing would be to add an optional seekable mode that acts more like PAR2 rather than what it acts like now, but I feel like par2cmdline-turbo is good enough at that and I should expand more into being a good unix-style pipe program instead.

Benchmarks in the future will come from a container on a Xeon machine, as I'm fairly convinced this laptop is on its way out...

v0.2.1

Hotfix for a buffer overflow in v0.2 that makes it segfault on Alpine Linux. Also fixes 32bit support for large files.

v0.2

This version comes as a breaking release to v0.1 as indicated by the README for the last day or two. It attempts to fix the primary problem of information loss/insertion completely destroying the poor format. Also it changes the default interleave amount to 5, adds a select few presets to choose from beyond the library defaults, and hopefully clears up some misunderstandings I saw in the HN Thread in the documentation and README.

v0.1

With this I can finally start shilling my software! With this version, tests have been added, a few final bugs have been fixed, etc. There's really not much that has changed since v0.04 except the README.

However, it is the start of a new era! With this, no more tinkering with data structs any longer! It is set in stone forevermore! Hopefully at least, if a major problem is found tomorrow, considering the sole userbase of this software package right now is me and me alone, I may make a rare, hypocritical, incompatible change. Forgive me!

v0.04

Moving fast today, v0.04 came just a few hours later, as the command line interface wasn't terribly hard to do. Now all that's left is adding in some tests and bug crunching (I hate how they taste).

v0.03

Implemented interleaving support, decided against convolutional code support. After this, I'll be making the command line utility and writing some user documentation for v0.04. Then I'll do some final bug crunching and cleaning up and release v0.1 if things go well.