Skip to content
A time series + geographical coordinates compression
Go
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
testdata
.deepsource.toml
.drone.yml
.gitignore
README.md
delta.go
go.mod
go.sum
tsd.go
tsd_test.go

README.md

Build Status

TSD

TSD is a time series storage specialized in compressing increasing timestamp and geo spacial data (lat, lng).

It performs better than Snappy & LZ4, on this specific usage without any advanced optimizations but storing deltas and storage size reduction.

Size: 64776 Snappy 61007    LZ4 57225       TSD 23948
Size: 18864 Snappy 18870    LZ4 18267       TSD 9883
Size: 15768 Snappy 12721    LZ4 11860       TSD 4775
Size: 7764  Snappy 5440     LZ4 5148        TSD 2122
Size: 9168  Snappy 7251     LZ4 6933        TSD 3018

Here is a blog post I wrote to describe this project

Usage

It implements BinaryMarshaler and BinaryUnmarshaler

ts := New()
ts.Push(1201984833, 39.91071, 116.50962)
ts.Push(1201985433, 39.91588, 116.52231)
ts.Push(1201986033, 39.91445, 116.56444)
b, _ := ts.MarshalBinary()
fmt.Println(hex.EncodeToString(b))
47a4d541003ce61f00b1c792290258020504f5290258ff711075

Format is as follow

header ts uint32,   lat float32, lng float32
      32                  32           32

info,  time delta dyn,   lat delta dyn, lng delta dyn
 8            n,               n,             n

n can be 0, an int8, int16 or int32.

Optimization

The Facebook's Gorilla paper would perform better but my needs were a bit different.

I needed a time window larger than the limit imposed by the paper (14 bits ~ 4 hours), and a way to store 2 values lat, lng that are closed to the previous one (moving devices).
The timestamp storage is almost the same as the paper but for the values a delta of delta and appropriate storage per entry are good enough and perform better than dict compression.

You can’t perform that action at this time.