help to design map/reduce jobs (SQL, Pig)
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.circleci
_doc
_unittests
src
.gitattributes
.gitignore
.landscape.yml
.local.jenkins.lin.yml
.local.jenkins.win.yml
.travis.yml
HISTORY.rst
LICENSE.txt
README.rst
appveyor.yml
build_script.bat
requirements.txt
setup.py

README.rst

README

Build status Build Status Windows https://circleci.com/gh/sdpython/pysqllike/tree/master.svg?style=svg GitHub Issues MIT License Requirements Status https://codecov.io/github/sdpython/pysqllike/coverage.svg?branch=master

Links:

Description

Writing a map/reduce job (using PIG for example), usually requires to switch from local files to remote files (on Hadoop). On way to work is extract a small sample of the data which will be processed by a map/reduce job. The job is then locally developped. And when it works, it is run on a parallized environment.

The goal of this extension is allow the implementation of this job using Python syntax as follows:

def myjob(input):
    iter = input.select (input.age, input.nom, age2 = input.age2*input.age2)
    wher = iter.where( (iter.age > 60).Or(iter.age < 25))
    return where

input = IterRow (None, [ {"nom": 10}, {"jean": 40} ] )
output = myjob(input)

When the job is ready, it can be translated into a PIG job:

input = LOAD '...' USING PigStorage('\t') AS (nom, age);
iter = FOREACH input GENERATE age, nom, age*age AS age2 ;
wher = FILTER iter BY age > 60 or age < 25 ;
STORE wher INTO '...' USING PigStorage();

It should also be translated into SQL.

Functionalities

  • not yet ready

Design

This project contains various helper about logging functions, unit tests and help generation.

  • a source folder: src
  • a unit test folder: _unittests, go to this folder and run run_unittests.py
  • a _doc folder: _doc, it will contains the documentation
  • a file setup.py to build and to install the module
  • a file make_help.py to build the sphinx documentation