Poor man's delta compression for .xopp files (in Python)

[charstorm]

Hello folks,

Let me start by thanking all the people who contributed/contributing to Xournal++. This software makes life a lot easy when it comes to writing notes on my graphic tablet. Nice work!

I made a bunch of notes and I wanted to keep them in github so that others can edit those files at a later stage. The default .xopp seems to be a binary format. So I did some digging and saw that its just gzipped XML file. So I thought of unzipping it and checking in the XML file. But I was worried about the file size. I searched for some compression tools. But nothing worked well.

Then I looked inside the XML file. Saw that most of the space was taken by raw x, y values in the element. So I took those x,y values, split them (x and y channels), rounded them to ints with a scaling (like 32, to capture some precision), then encoded their differences in successive values with a custom base64 encoding. This gave a good enough compression for my use case.

This is the script

The original data looks like:

<stroke tool="pen" ts="0" fn="" color="#00c0ffff" width="1.41" capStyle="round">1294.9999 58.905594 1295.3359 59.207993 1295.6719 59.207993 1296.0559 59.207993 ... 56.702393</stroke>

With delta encoding, it looks like:

<strk prev="1">+9oG+4+3+5+5+6+b+9+7+7+1+3+3+6+8|+qc-c-d-d-c-c+0+7+b+h+9+a+a+d+9</strk>

This is what the results look like:

There wasn't any noticeable difference in the output when I reconstructed the original document.

Thank you!

[eldipa]

Nice hack! A previous optimization similar for reducing the digits was merged a few years ago and the idea of using delta compression is definitely the way to go.

In a analysis that I did there is a plot under the title "Talking of strokes, how are theirs x-y coords distributed?" It is shown that all the x and y coordinates are in the range [0-800] so we are dealing with always small numbers very similar to each other, hence delta encoding works very nice.

You may want to go a little deeper with frame of reference. Delta encoding is definitely something that will be implemented in the new file format (a work in progress still).

And about the .xopp file format, I documented it a few months ago. You may find it useful for more hacking.