Skip to content
/ Hyckle Public

A memory-friendly warpper of pickle with better compression.

License

Notifications You must be signed in to change notification settings

zylo117/Hyckle

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

12 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Hyckle

Introduction

Hyckle is a high-level warpper of pickle that provides data serialization for all python-pickleable object with the support of local read & write (like H5), better compression (85% smaller than original pickle) and more memory-efficient. Also it's a drop-in replacement for python dict and list.

Hyckle是pickle的一个高层封装,可以为所有可pickle的python变量提供数据序列化, 同时支持像H5一样的局部读写,拥有更好的压缩率(比原生pickle小了85%),并更节省内存。 同时,它可以用于直接替换python的字典和列表。


Installation

pip install hyckle

Demo

Basic Usage

# create or reload a hyckle file. 
# This will create a new hyckle file if not exists or reload a existed hyckle file.
hk = Hyckle('helloworld.hyckle')

# use hyckle as a dict
hk['test_0'] = any_pickleable_object_0
hk['test_1'] = any_pickleable_object_1
hk['test_2'] = any_pickleable_object_2
hk['test_3'] = any_pickleable_object_3
# or
hk.add('test_0', any_pickleable_object_0)
hk.add('test_1',any_pickleable_object_1)
hk.add('test_2',any_pickleable_object_2)
hk.add('test_3',any_pickleable_object_3)

# use hyckle as a list
hk.append(any_pickleable_object_4)
hk.append(any_pickleable_object_5)

# check the length of hyckle
length = len(hk)

# use hyckle as a iterable object
for obj in hk:
    do_something(obj)

# get certain object by key: str
obj_0 = hk.get(key)
# or using getitem method
obj_1 = hk[key]

# get certain object by index: int
obj_2 = hk[index]

# get certain items by slice
objs = hk[233:666]

# remove key in hyckle
hk.remove('test_0')

# flush to disk. 
# Normally hyckle will auto-flush to disk every K item adding (K = buffersize)
hk.flush()

# close a hyckle file. 
# After final reading/writing hyckle, you should close it to ensure data integrity.
hk.close()

Advanced Usage

# create a hyckle with custom compression,
# lzma is much slower but smaller
hk = Hyckle('helloworld.hyckle', compression='lzma')

# or use gzip (default) with different compression level,
# bigger number means smaller filesize.
hk = Hyckle('helloworld.hyckle', compression=9)

# sppedup I/O using bigger buffer.
# Noted, bigger buffersize means allocating more memory.
hk = Hyckle('helloworld.hyckle', buffer_size=128)

About

A memory-friendly warpper of pickle with better compression.

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages