Skip to content
master
Go to file
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 

README.md

genX

Robust push/pull/graph generator primitives for Python

Introduction

Python supports (lazy) generator expressions and functions:

nums = (i for i in range(100))
evens = (i for i in nums if i % 2 == 0)
squared_evens = (i * i for i in evens)

print(list(squared_evens)[0:5])

image

This is useful because we only iterate over each value a single time while maintaining composability and expresiveness of the individual components. At the end of this code block all of the generators have been depleted.

There are, however, some limitations. First consider the case of evens needing to be passed into 2 expressions rather than just one.

nums = (i for i in range(100))
evens = (i for i in nums if i % 2 == 0)
squared_evens = (i * i for i in evens)
cubed_evens = (i * i * i for i in evens)

print(list(squared_evens)[0:5])
print(list(cubed_evens))

image

Oops! The expressions are lazy but not smart enough to save each iteration for both squared_evens and cubed_evens. There are some workarounds of course but you start to lose out on the composability benefits depending on what you do.

Another limitation is that traditional generator expressions rely on a pull model, i.e. you are pulling results from some sort of iterable. They cannot easily address a push model, or when you want to push a value or iterable of values into a lazily linked computational chain.

This library attempts to address all 3 use cases with as intuitive of a syntax as regular Python generators.

Implementation is heavily inspired by the concept of transducers (most notably from Clojure). This is a somewhat opionated implementation of just some of the concepts of transducers + some directed graph niceties. For heavy duty applications you may want to consider a proper flow-based DAG such as Luigi.

Other projects doing something similar:

About

Robust push/pull/graph generator primitives for Python

Resources

Releases

No releases published

Packages

No packages published
You can’t perform that action at this time.