Miner is generic engine and framework for processing and analyzing data written in python
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
bin
lib
m
tests
.gitattributes
.gitignore
INSTALL
LICENSE
README.md
miner
miner.bat
miner_globals.py
miner_main.py
miner_upgrade.py
miner_version.py
pyreadlineconfig.py
readline.ini
readline_replacement.py
tool-description.json

README.md

Miner

Miner is generic engine and interactive framework for processing and analyzing data. It defines simple yet powerful query language for data processing, see some examples below. Miner is written in python, allowing easy integration with proprietary data formats and extending its functionality.

Applications

Miner can be used for wide variety of tasks like analyzing logs of web servers, financial data, post processing responses from database queries and even processing network captures. It can be also used to perform simulations and tests.

Query Examples

Consider you have log from http server in csv format with something like following:

path,time
/index.html,10
/login,1000
/heavy_script?param=1,10000

With miner you can easily find user requests that took maximum time:

READ log.csv | TOP 5 time | STDOUT

The output will be something like:

path                   time
----------------------------
/heavy_script?param=5  50000
/heavy_script?param=4  40000
/heavy_script?param=3  30000
/heavy_script?param=2  20000
/heavy_script?param=1  10000

You can see the total time consumed by different resources

READ log.csv | SELECT path.split('?',1)[0] as resource, time | FOR DISTINCT resource SELECT count(True) as numRequests, sum(time) | STDOUT
resource       numRequests     time
------------------------------------
/heavy_script  10              100000
/index.html    5               50
/login         5               5000

And many many others interesting facts that you can learn without writing single line of code.

Miner provides the possibility to work with plain data without need to insert it to database or processing it in another way.

Windows Binaries

Miner distribution includes python implementation of following linux utilities:

  • curl
  • gzip, gunzip, zcat
  • md5sum
  • tar
  • cat
  • grep