Use public cloud SDKs to measure object store CRUD operations times (S3, google cloud storage, azure blobs etc.)
Switch branches/tags
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.gitignore
LICENSE.txt
README.md
awsexercizer.py
azureexercizer.py
datamaker.py
dirstats.ipynb
exercizer.py
gcpexercizer.py
libcloud_deprecated_objectexercizer.py
requirements.txt

README.md

Object Bench

What is this?

Object Bench is a benchmarking tool to create, upload, download and delete object store buckets (containers) and objects (blobs) into some public cloud stores (currently AWS, Azure, Google Cloud). At this time it supports uploading a directory (its constituent files being treated as multiple objects) to a single bucket.

Usage

All credentials are read-off set OS environment variables. Please obtain the required variables from the cloud provider. To see which environment variables are needed please see the "main" block of the corresponding "*exercizer.py" code file in this repository.

Install the requirements (preferably into a python virtual environment):

pip install -r requirements.txt

After putting the credentials into environment variables (using e.g. the source statement below), run the benchmark. For example:

$ source {secure_folder}/credentials.sh
$ python awsexercizer.py
Upload objects
{'/tmp/smalldir/MAINTAINERS': (1.1708409786224365, 446794),
 '/tmp/smalldir/cpuinfo': (0.6111950874328613, 1748),
 '/tmp/smalldir/downloads/MAINTAINERS': (0.5888941287994385, 446794),
 '/tmp/smalldir/downloads/cpuinfo': (0.4392850399017334, 1748),
 '/tmp/smalldir/downloads/iomem': (0.23023605346679688, 1043),
 '/tmp/smalldir/downloads/meminfo': (0.4946460723876953, 1307),
 '/tmp/smalldir/iomem': (0.3935070037841797, 1043),
 '/tmp/smalldir/level1/iomem': (0.5325939655303955, 1043),
 '/tmp/smalldir/meminfo': (0.1788921356201172, 1307),
 '/tmp/smalldir/uploads/MAINTAINERS': (0.7265870571136475, 446794),
 '/tmp/smalldir/uploads/cpuinfo': (0.2738769054412842, 1748),
 '/tmp/smalldir/uploads/iomem': (0.22115206718444824, 1043),
 '/tmp/smalldir/uploads/meminfo': (0.4289848804473877, 1307)}
Download objects
{'/tmp/smalldir/MAINTAINERS': (17.39839506149292, 446794),
 '/tmp/smalldir/cpuinfo': (0.2229619026184082, 1748),
 '/tmp/smalldir/downloads/MAINTAINERS': (20.07686996459961, 446794),
 '/tmp/smalldir/downloads/cpuinfo': (0.5823531150817871, 1748),
 '/tmp/smalldir/downloads/iomem': (0.520056962966919, 1043),
 '/tmp/smalldir/downloads/meminfo': (0.22242498397827148, 1307),
 '/tmp/smalldir/iomem': (0.22393798828125, 1043),
 '/tmp/smalldir/level1/iomem': (0.21981501579284668, 1043),
 '/tmp/smalldir/meminfo': (0.6320850849151611, 1307),
 '/tmp/smalldir/uploads/MAINTAINERS': (48.98902893066406, 446794),
 '/tmp/smalldir/uploads/cpuinfo': (0.32395100593566895, 1748),
 '/tmp/smalldir/uploads/iomem': (0.7771360874176025, 1043),
 '/tmp/smalldir/uploads/meminfo': (1.2843818664550781, 1307),
 '_listObjectsInContainer': (0.17245697975158691,
                             'Container objects listing time')}
Delete buckets
{'blobtester': 2.95768404006958, 'operation': 'Deleted'}
(objbench) root@sachin-udtp:~/bbbcode/objbench#

In the above run each uploaded file/downloaded object has a tuple (time in seconds, bytes).