Skip to content

Latest commit

 

History

History
193 lines (139 loc) · 7.12 KB

README.md

File metadata and controls

193 lines (139 loc) · 7.12 KB
Raw

Table of Content

Name

pykit: A collection of python libs those are used in project s2: Storage-Service at baishancloud.com

Status

This library is in beta phase.

It has been used heavily in our object storage service, as a foundamental library of our devops platform.

Description

Module List

There is a README.md for each module.

name description
argchecker Validates arguments against the schema
aws AWS error codes and so on
cacheable Cache data which access frequently
cachepool Reusable object cache in process
daemonize Start, stop or restart a daemon process
dictutil Dictionary helper utility
etcd etcd client
fsutil File-system Utilities
heap Min heap
http HTTP/1.1 client
humannum Convert number to human readable number string
jobq Process serial of input elements with several functions concurrently and sequentially
logutil Utility functions to create logger or make log message
mime Utility functions to handle mime type
modutil Submodule Utilities
mysqlconnpool Mysql connection pool with MySQLdb in python
mysqlutil Mysql related datatype, operations
net Network utility
portlock cross process lock
priorityqueue Priority queue
proc Utility to create sub process
rangeset Segmented range.
ratelimiter rate limiter
redisutil For using redis more easily.
shell Set different command arguments to execute different functions
strutil A collection of helper functions used to manipulate string
threadutil Utility functions for better management of threads
timeutil Support specify time format output and get current ts, ms, us api etc
utfjson Force json.dump and json.load in utf-8 encoding
wsjobd Job daemon based on websocket protocol
zkutil Utility functions for zookeeper

Install

Just clone it and copy it into your project source folder.

cd your_project_folder
git clone https://github.com/baishancloud/pykit.git

Usage

from pykit import jobq

def add1(args):
    return args + 1

def printarg(args):
    print args

jobq.run([0, 1, 2], [add1, printarg])
# > 1
# > 2
# > 3

Configuration

pykit provides a way to setup config for it. Config module tries to import pykitconfig in which a user sets config. Example:

> cat pykitconfig.py
uid = 2
gid = 3

> cat foo.py
from pykit import fsutil
fsutil.write_file('bar', '123') # write_file sets file uid and gid to 2 and 3.

Supported config

  • uid: specifies default user-id when file created, directory made.
  • gid: specifies default group-id when file created, directory made.
  • log_dir: specifies default base_dir when logger created.
  • cat_stat_dir: specifies default stat_dir for all log cat class instances.

See the README.md of sub modules for detail.

Test

Run one of following to test all, a module, a TestCase or a function.

./script/t.sh
./script/t.sh zkutil
./script/t.sh zkutil.test
./script/t.sh zkutil.test.test_zkutil
./script/t.sh zkutil.test.test_zkutil.TestZKUtil
./script/t.sh zkutil.test.test_zkutil.TestZKUtil.test_lock_data

For developer

There are several scripts for developers. See script.

Config

pykit.config is used internally by pykit modules.

How it works

It try to load pykitconfig from a top level module of the project.

Then it load predefined config via getattr(pykitconfig, key, default).

So module can get config via config.xxx by import it.

How to use

  • Add predefined config in config.py

    uid=_get('uid')
gid=_get('gid')

  • Import config in you own module

    from pykit import config

  • Handle user define config in you code

    uid=uid or config.uid
gid=gid or config.gid

  • Update the configuration supported config in this doc

Author

Zhang Yanpo (张炎泼) drdr.xp@gmail.com

Copyright and License

The MIT License (MIT)

Copyright (c) 2015 Zhang Yanpo (张炎泼) drdr.xp@gmail.com