Using and extending the code

Kanzi provides a Writer and a Reader as main constructs to compress and decompress data blocks.

See doc kanzi-go/v2/io

Compressing/Decompressing data

Here is how to compress/decompress a block to/from a file using RLT+TEXT as transform, Huffman as entropy codec, using a block size of 1 MB, 4 jobs and a checksum:

package main

import (
	"os"
	kio "github.com/flanglet/kanzi-go/v2/io"
)

func Compress(block []byte) (int, error) {
	// Create an io.WriteCloser
	output, err := os.Create("compressed.knz")

	if err != nil {
		return 0, err
	}

	// Create a Writer
        w, err := kio.NewWriter(output, "RLT+TEXT", "HUFFMAN", 1024*1024, 4, true, 0, false)

	if err != nil {
		return 0, err
	}

        // Compress block
	written, err := w.Write(block)

	if err == nil {
		// Close Writer
	        err = w.Close()
	}

	return written, err
}

func Decompress(block []byte) (int, error) {
	// Create an io.ReadCloser
        input, err := os.Open("compressed.knz")

	if err != nil {
		return 0, err
	}

	// Create a Reader
        r, err := kio.NewReader(input, 4)

	if err != nil {
		return 0, err
	}

	// Decompress block
        read, err := r.Read(block)

	if err == nil {
		// Close Reader
                err = r.Close()
	}

	return read, err
}

Implementing a new transform

Here is how to implement and add a new transform to Kanzi.

• Step 1: write the transform code

For example:

type SuperDuperTransform struct {
}

// NewSuperDuperTransform creates a new instance of SuperDuperTransform
func NewSuperDuperTransform() (*SuperDuperTransform, error) {
	this := &SuperDuperTransform{}
	return this, nil
}

// NewSuperDuperTransformWithCtx creates a new instance of SuperDuperTransform using a
// configuration map as parameter.
func NewSuperDuperTransformWithCtx(ctx *map[string]any) (*SuperDuperTransform, error) {
	this := &SuperDuperTransform{}
	return this, nil
}

// Forward applies the function to the src and writes the result
// to the destination. Returns number of bytes read, number of bytes
// written and possibly an error.
func (this *SuperDuperTransform) Forward(src, dst []byte) (uint, uint, error) {
        // Ensure enough room in the destination buffer
	if n := this.MaxEncodedLen(len(src)); len(dst) < n {
		return 0, 0, fmt.Errorf("Output buffer is too small - size: %d, required %d", len(dst), n)
	}

        for i := range(src) {
		dst[i] = src[i] ^ 0xAA
        }

	return uint(len(src)), uint(len(src)), nil
}

// Inverse applies the reverse function to the src and writes the result
// to the destination. Returns number of bytes read, number of bytes
// written and possibly an error.
func (this *SuperDuperTransform) Inverse(src, dst []byte) (uint, uint, error) {
        for i := range(src) {
		dst[i] = src[i] ^ 0xAA
        }

	return uint(len(src)), uint(len(src)), nil
}

// MaxEncodedLen returns the max size required for the encoding output buffer
func (this SuperDuperTransform) MaxEncodedLen(srcLen int) int {
	return srcLen
}

Always provide a constructor with a context: the context map contains all the application wide information (such as block size, number of jobs, input & output names, etc ...). Always implement the ByteTransform interface and do not create more goroutines than the number of jobs provided in the context. Implement Forward and Inverse methods as well as MaxEncodedLen(int). Do not write to stdio or stderr.

• Step 2: Register the transform in transform/Factory.go

Add the type, say

 SUPERDUPER_TYPE = uint64(63)

Let us say you use the name "SUPERDUPER" for the transform. Update the following methods:

func newToken(ctx *map[string]interface{}, functionType uint64) (kanzi.ByteTransform, error) 
func getByteFunctionNameToken(functionType uint64) string
func getByteFunctionTypeToken(name string) uint64 

• step 3: Update the help message in app/Kanzi.go

In printHelp(), add the SUPERDUPER transform to the list in the -t option section.

• This is it. For example, run

Kanzi -i foo.txt -f -t SUPERDUPER -j 2 -v 4