Addition: functools.pipe

[dg-pb]

Feature or enhancement

Proposal:

See discourse for a complete proposal.

This is one of basic components for functional programming.

It would be useful as it in various cases, such as predicate composition.

Also the user could inherit from it to implement operator-based pipeline syntax, which is not much worse than some of the proposals for specialised full-fledged pipeline statements.

So I propose a minimal pipe implementation.

class pipe:
    def __init__(self, *funcs):
        self.funcs = funcs

    def __call__(self, obj, /, *args, **kwds):
        if not (funcs := self.funcs):
            assert not args and not kwds
            return obj
        first, *rest = funcs
        if args or kwds:
            obj = first(obj, *args, **kwds)
        else:
            obj = first(obj)
        for func in rest:
            obj = func(obj)
        return obj
    
    def __get__(self, obj, objtype=None):
        if obj is None:
            return self
        return types.MethodType(self, obj)

Features:

1. pipe supports full signature of the first function.
2. Having __get__ it behaves in line with def and partial. (see examples below)
3. Behaves as identity function without input functions

There are other possible additions to it, but user can implement most of those in inherited classes.

Implementation side is straight forward (including inspect.signature) so the only question is whether it is desirable to have this.

Other languages that have it (or considered it):

1. JavaScript
2. https://docs.rs/pipelines/latest/pipelines/
3. https://isocpp.org/files/papers/n3534.html

It is the same concept as in JavaScript, but differs from Rust and C++ proposal in a way that it does not intend to provide a full pipelining framework, but rather simple utility function that can be modularly combined with other tools.

E.g. Rust example can be achieved by implementing independent functions that do parallel work:

from functools import partial

class pipe(functools.pipe):
   """Convenience subclass with operator overloading"""
    __ror__ = functools.__call__
    def __rshift__(self, func):
        return self.__class__(*self.funcs, *(funcs,))

# Build the first 10 fibonacci numbers in parallel,
#    then double them:
parallel_map = map    # Not parallel :)

def fibonacci(n):
    return 1 if n<2 else fibonacci(n-1) + fibonacci(n-2)

result = range(10) | (pipe()
    >> partial(parallel_map, fibonacci)
    >> partial(map, lambda x: x*x)
    >> list
)
print(result)    # [1, 1, 4, 9, 25, 64, 169, 441, 1156, 3025]

# Input not being tied to `pipe` object (as it is in Rust)
#     allows re-using of pipelines for different inputs:
pipeline = pipe(
    partial(parallel_map, fibonacci),
    partial(map, lambda x: x*x),
    list
)
pipeline([20, 40])       # [119814916, 27416783093579881]
pipeline(range(14, 16))  # [372100, 974169]

Couple of additional benefits:
A) Convenience for pipe-like method modifications.

class A:
    a = 1
    def factorial_plus(self, n):
        return math.factorial(n + self.a)

    neg_factorial_plus = pipe(factorial_plus, operator.neg)

B) pipe() without arguments acts as identity function.

Personally, I would use it instead of defining my own lambda as it would most likely be more performant and also I would like to rely on one implementation for all identity function needs.

The way I see it, the benefits could potentially outweigh the costs:

• Benefits: multi-purpose utility function with different applications, which is fun to play with.
• Costs: It would be the least complex object in functools. Implementation is straight forward without any foreseeable obstacles and there is nothing new to invent here.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

https://discuss.python.org/t/functools-pipe/69744

[sobolevn]

Please, provide type hints for this feature. We would have to type in typeshed somehow. You can take a look at https://github.com/dry-python/returns/blob/master/returns/_internal/pipeline/pipe.pyi

[nineteendo]

You wrote def instead of class.

[skirpichev]

I have already discussed this feature proposal on Discourse

You did. But I don't see that this proposal got any support from core developers. @sobolevn, may we count you?

I think this issue should be closed, until idea got some support.

[sobolevn]

Yes, let's close it for now. What would I expect to see in the discussion of this feature?

• Typing. Type safety would be a strong point to add this feature. Because regular function calls are type safe. So, if you switch from type safe code to unsafe one - that's a minus.
• Performance. Please, compare your proposal with direct function calls. It should not be much slower. Maybe some parts can be written in C to make this faster?
• DSL discussion.
• Async support? This is optional, but it would be great to read the discussion about this topic. Will it work? Do we need any specific changes in the design?

Thank you for your idea! The idea itself seems interesting, but it needs some more work, before we can jump to the implementation.

[dg-pb]

@sobolevn, I think I have crowded this issue with examples of use cases, which somewhat obscured what the actual proposal is.

Proposal itself is to simply add class pipe, which in essence is simply:

def pipe(*funcs):
    def inner(arg):
        for f in funcs:
            arg = f(arg)
        return arg
    return inner

It:

1. has a couple of quirks: __get__ and fullargspec of first function
2. is implemented in C precisely for performance of __call__

It does not include:

1. any DSL
2. async support (I think async should implement its own async version, I can not see how class can have both async and sync call).

Also, I have a question regarding typing.
Could you give an example of C implemented class which has type safe implementation?