Skip to content
a collection of toolkit lib for distributed system development in python
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
argchecker Close #181. add argchecker module Apr 10, 2018
aws Squashed aws/thirdpart/errorcode.json master:aws/errorcode.json b1aab… Apr 12, 2018
awssign add example of awssign add_post_auth Sep 11, 2018
cacheable refine tests Sep 2, 2018
cachepool cachepool: safety update self.stat May 15, 2018
cgrouparch add close zk fun Oct 29, 2018
csvutil add csvutil module Jul 9, 2018
daemonize daemonize/test: extend sleep time to tolerate slow VM. Aug 8, 2018
dictutil dictutil: add test of intern string Jan 29, 2019
ectypes fixup: ServerStatus.def_status return string, instead of tupe Jul 17, 2019
etcd etcd: test should pull docker image automatically Aug 8, 2018
fsutil linux always use 512 Byte as sector size May 20, 2019
heap heap: add exception NotFound Oct 10, 2017
http http client add set_timeout Mar 14, 2019
httpmultipart Fix #406. modify readme.md Sep 11, 2018
humannum humannum supports negative number Sep 29, 2018
jobq jobq: adjust format Apr 29, 2018
jobscheduler fix doc Apr 18, 2018
logcollector add no merge in log collector May 5, 2019
logutil refine tests Sep 2, 2018
mathtoy mathtoy: plot: add argument "rangey" to specify range of y May 5, 2019
mime mime uses thirdpart mimes.json. Oct 21, 2017
modutil simplify modutil test Sep 8, 2017
mysqlconnpool ututil: move port listening check to ututil.py Aug 25, 2018
mysqlutil add function: make_sharding Dec 25, 2018
net refine inner ip patterns Jan 25, 2019
p3json p3json: fix p3json.tool: outfile with the same path with infile shoul… Feb 27, 2019
portlock portlock cli tool to run sth only once: python2 -m pykit.portlock --l… Oct 1, 2017
priorityqueue refine tests Sep 2, 2018
proc proc.command add option: shell Jul 13, 2018
profiling mes -> msg Apr 8, 2017
rangeset rangeset: Range support operator "&" Apr 12, 2019
ratelimiter add ratelimiter impl Nov 29, 2017
redisutil redis proxy cli add delete && hdel Jun 14, 2019
request design class request for http request Oct 12, 2018
script M script/t echo test info Sep 2, 2018
shell shell module refine Jan 21, 2019
strutil M strutil/README.md Nov 12, 2018
threadutil threadutil.start_* support arg `after=0.1` to let thread start after … May 8, 2018
timeutil timeutil: add parse_to_ts() to convert time string to timestamp Nov 14, 2018
txutil txutil API changes May 14, 2018
utfjson refine tests Sep 2, 2018
utfyaml add module utfyaml Apr 21, 2018
wsjobd refine tests Sep 2, 2018
zktx zktx use lock node to save the modified record value Apr 3, 2019
zkutil using kazooclient access zk Apr 4, 2019
.gitignore Initial commit Dec 7, 2016
.gitsubrepo simplify gitsubrepo config Apr 16, 2018
ISSUE_TEMPLATE.md add issue and pull request template Aug 22, 2018
LICENSE Initial commit Dec 7, 2016
PULL_REQUEST_TEMPLATE.md add issue and pull request template Aug 22, 2018
README.md Fix 112. finish multipart/form-data module Jul 26, 2018
__init__.py 添加pykit打包需要的文件及打包发布脚本 Jul 16, 2018
config.md fsutil: add iostat() for collecting iostat on path or device. May 28, 2018
config.py refine inner ip patterns Jan 25, 2019
dep-graph.jpg update dep-graph.jpg Jul 9, 2018
dep-graph.sh add module dependency May 12, 2018
git-subrepo Squashed git-subrepo master:git-subrepo c5229c61a () Apr 12, 2018
pack-release.sh 添加pykit打包需要的文件及打包发布脚本 Jul 16, 2018
requirements.txt kazoo upgrade version to 2.6.1 Jul 4, 2019
setup.py 添加pykit打包需要的文件及打包发布脚本 Jul 16, 2018
stopwatch.py Squashed stopwatch.py master:stopwatch.py cc34eae71 Jun 7, 2017
utdocker.py refine tests Sep 2, 2018
ututil.py refine tests Sep 2, 2018

README.md

Table of Content

Name

pykit: A collection of python libs those are used in project s2: Storage-Service at baishancloud.com

Status

This library is in beta phase.

It has been used heavily in our object storage service, as a foundamental library of our devops platform.

Description

Module List

There is a README.md for each module.

name description
argchecker Validates arguments against the schema
aws AWS error codes and so on
awssign Sign a request by using aws signature version 4
cacheable Cache data which access frequently
cachepool Reusable object cache in process
cgrouparch Build cgroup directory tree, add set cgroup pid
cluster Some helper function for the server in a cluster
csvutil Utility functions for CSV file loading and conversion
daemonize Start, stop or restart a daemon process
dictutil Dictionary helper utility
ectypes Utility functions for the server
etcd etcd client
fsutil File-system Utilities
heap Min heap
http HTTP/1.1 client
httpmultipart Utility functions to make headers or make body reader about multipart
humannum Convert number to human readable number string
jobq Process serial of input elements with several functions concurrently and sequentially
jobscheduler Run jobs at certain time
logcollector Collect logs of interest locally
logutil Utility functions to create logger or make log message
mime Utility functions to handle mime type
modutil Submodule Utilities
mysqlconnpool Mysql connection pool with MySQLdb in python
mysqlutil Mysql related datatype, operations
net Network utility
portlock cross process lock
priorityqueue Priority queue
proc Utility to create sub process
rangeset Segmented range.
ratelimiter rate limiter
redisutil For using redis more easily.
shell Set different command arguments to execute different functions
strutil A collection of helper functions used to manipulate string
threadutil Utility functions for better management of threads
timeutil Support specify time format output and get current ts, ms, us api etc
utfjson Force json.dump and json.load in utf-8 encoding
utfyaml Force yaml.dump and yaml.load in utf-8 encoding
wsjobd Job daemon based on websocket protocol
zktx Transaction implementation on Zookeeper
zkutil Utility functions for zookeeper

Module dependency

Install

Just clone it and copy it into your project source folder.

cd your_project_folder
git clone https://github.com/baishancloud/pykit.git

Usage

from pykit import jobq

def add1(args):
    return args + 1

def printarg(args):
    print args

jobq.run([0, 1, 2], [add1, printarg])
# > 1
# > 2
# > 3

Configuration

See config.md

Test

Run one of following to test all, a module, a TestCase or a function.

./script/t
./script/t zkutil
./script/t zkutil.test
./script/t zkutil.test.test_zkutil
./script/t zkutil.test.test_zkutil.TestZKUtil
./script/t zkutil.test.test_zkutil.TestZKUtil.test_lock_id

See Details

For developer

There are several scripts for developers. See script.

Author

Zhang Yanpo (张炎泼) drdr.xp@gmail.com

Copyright and License

The MIT License (MIT)

Copyright (c) 2015 Zhang Yanpo (张炎泼) drdr.xp@gmail.com

You can’t perform that action at this time.