Add functools.partialmethod

[ssadler]

BPO	4331
Nosy	@rhettinger, @jcea, @ncoghlan, @abalkin, @jackdied, @bitdancer, @vajrasky
Files	partialbug.py: Demonstration of bug issue4331.patch: Patch - Add descriptor to partial object + tests functools.partial-descrget.patch 4331.v1.patch 4331.v2.patch issue4331_partialmethod.diff: Proposed implementation of partialmethod

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2013-11-03.06:43:39.618>
created_at = <Date 2008-11-16.06:32:58.493>
labels = ['extension-modules', 'type-feature', 'library']
title = 'Add functools.partialmethod'
updated_at = <Date 2013-11-04.13:34:43.919>
user = 'https://bugs.python.org/ssadler'

bugs.python.org fields:

activity = <Date 2013-11-04.13:34:43.919>
actor = 'ncoghlan'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2013-11-03.06:43:39.618>
closer = 'ncoghlan'
components = ['Extension Modules', 'Library (Lib)']
creation = <Date 2008-11-16.06:32:58.493>
creator = 'ssadler'
dependencies = []
files = ['12017', '15800', '25016', '32405', '32409', '32436']
hgrepos = []
issue_num = 4331
keywords = ['patch']
message_count = 32.0
messages = ['75928', '75942', '75945', '75946', '97470', '98490', '98675', '99776', '99813', '99956', '99957', '156712', '181582', '181658', '182943', '182946', '190642', '190645', '190648', '201111', '201279', '201310', '201366', '201377', '201451', '201453', '201592', '201633', '201816', '201998', '202084', '202135']
nosy_count = 14.0
nosy_names = ['rhettinger', 'jcea', 'ncoghlan', 'belopolsky', 'ironfroggy', 'jackdied', 'Christophe Simonis', 'ssadler', 'eckhardt', 'r.david.murray', 'Alexander.Belopolsky', 'alonho', 'python-dev', 'vajrasky']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue4331'
versions = ['Python 3.4']

[ssadler]

Calling a function created by _functools.partial as a method raises an
exception:

"TypeError: method_new() takes exactly n non-keyword arguments (0 given)"

Where method_new is the function passed into partial() and n is the
number of arguments it expects.

This does not happen when using a python version of partial().

Strangely, in the circumstance that I originally encountered the bug,
there was one instance that I was doing this and it _DID WORK_. The
function being passed into partial() was the same as in the place where
it was failing. The only significant difference that I could see was
that the input function to partial() was being imported, rather than
being defined in the same namespace as it was used I was unable to
reproduce it in my test case (attatched).

Tested on 2.6 and 2.5.2

[ssadler]

A short update, I believe that the reason that it was working in one
instance was because of some abstractions by a base class (Django model,
get_absolute_url).

[ironfroggy]

I don't think this is any kind of bug, it is simply a product of only
function objects being decorated automatically as methods. Your python
version works because it is, in fact, a function. _functools.partial
objects are not functions, but simply callable objects.

[rhettinger]

Reclassifying as a feature request.
A descriptor could be added to partial()
so that it too would have automatic
method binding just like pure python functions.

[ChristopheSimonis]

I followed the advice of Raymond and implement a descriptor on partial.

[AlexanderBelopolsky]

Christophe,

It looks like your patch goes out of its way to avoid creating nested partials. This is a worthwhile goal and I think it should be done in partial_new so that partial(partial(f, x), y) returns partial(f, x, y).

If fact, I was surprised to learn that current partial implementation does not behave this way:

>>> partial(partial(f, 1), 2).func
<functools.partial object at 0x100435af8>

Does anyone know the reason for the current behavior? It is possible that I am missing some subtlety related to keyword arguments.

[AlexanderBelopolsky]

Please see bpo-7830 for a related patch.

[jackdied]

I'm having some trouble wrapping my head around this one. It isn't obvious to me that
my_method(*args):
print(args)
class A():
meth = partial(my_method, 'argA')
ob = A()
ob.meth('argB')

should print (<A object at 0x1234>, 'argA', 'argB') and not
('argA', <A object at 0x1234>, 'argB')

The patch seems to prefer the first form but if you are using a partial shouldn't you expect 'argA' to always be the first argument to the partial-ized function?

[bitdancer]

I would expect the second and would view the first as a bug.

[jackdied]

We talked about it at sprints and the semantics are ambiguous and there are alternatives.

Ambiguous:
def show_funcs(*args): print(args)
class A():
run = partial(1)
ob = A()
ob.run(2,3)
Should this print (self, 1, 2, 3) or (1, self, 2, 3)? And what about
partial(ob.run, 2)(3)

Alternatives: partial is a convenience function not an optimization (it doesn't offer a speedup. So you can write a lambda or named function that has the exact semantics you want without suffering a speed penalty.

So unless there are a lot of good use cases with obvious behavior, we should refuse the temptation to guess and leave partial as-is.

[jackdied]

correction:
run = partial(1)
should have been
run = partial(show_funcs, 1)

[anacrolix]

I've attached a patch that implements the descriptor protocol for functools.partial with minimum changes.

[eckhardt]

Just for the record, the behaviour is documented, unfortunately in the very last line of the functools documentation: "Also, partial objects defined in classes behave like static methods and do not transform into bound methods during instance attribute look-up."

Concerning how exactly they should behave during that lookup, I'd use the least surprising variant, namely that they are not treated differently from other functions: The first parameter is implicitly "self".

[anacrolix]

What's preventing this from being committed and closed?

[eckhardt]

There is at least one thing that is missing in the patch, it lacks the necessary tests. The partialbug.py demonstrates the issue, it could be used as a base. However, even then, there is still one thing that is problematic: The fact that partial() returns something that behaves like a static method is documented and changing that is not backward compatible.

I still think that something like this should become part of Python though. Jack Diederich argues that you can use lambda to achieve the same, but that is not always true. If you want to bind an argument to the current value of a variable instead of a constant, lambda fails. You need the closure created by a function call to bind those variables inside a local function. Having a dedicated function for that is IMHO preferable to people copying the Python-only equivalent of partial() to achieve the same effect or even inventing their own.

[bitdancer]

See also bpo-11470.

[ncoghlan]

I don't believe it is reasonable to change the behaviour of partial at this late stage of the game. It's documented as behaving like staticmethod (albeit by not implementing the descriptor protocol at all), so that's no longer something we can change. If bpo-11470 is added, then we'll just implement the staticmethod-like behaviour explicitly rather than leaving it as implicit.

More importantly, the acceptance of PEP-443's functools.singledispatch makes it more desirable than ever to support partial binding for method implementations *even when the descriptor protocol is not involved*.

Accordingly, I suggest converting this proposal to a separate functools.partialmethod API that:

1. When called directly, passes the first positional argument as the first positional argument of the underlying function (providing call time binding of self, just like a normal function)
2. When retrieved from a class, returns itself
3. When retrieved from an instance, returns an appropriate bound method object (providing method lookup time binding of self, just like a normal function)

functools.partial will then continue to behave as it always has (thus posing no backwards compatibility risks), while the new partialmethod implementation should work with both class descriptor protocol based dispatch (through point 3) *and* the new functools.singledispatch mechanism (through point 1).

[ncoghlan]

Any associated tests may also want check that wrapping classmethod around a partialmethod generates a well behaved class method, and ditto for property.

If singledispath, classmethod, partialmethod and class and instance attribute access all work correctly, then we can be absolutely certain the results is behaving just like an ordinary function does :)

[anacrolix]

This sounds excellent Nick.

[ncoghlan]

To clarify the current state of this:

• I'm still in favour of adding this feature for Python 3.4
• a suitable patch is still needed, as the currently attached patches modify the existing functools.partial object, rather than adding a separate "partialmethod" API
• a Python implementation would be fine

The following prototype should work as a starting point to be elaborated into a full patch with docs and tests:

class partialmethod(functools.partial):
    def __get__(self, obj, cls):
        if obj is None:
            return self
        return functools.partial(self.func,
                                 *((obj,) + self.args),
                                 **self.keywords)
    def __call__(*args, **kwds):
        self, *args = args
        call_kwds = {}
        call_kwds.update(self.keywords)
        call_kwds.update(kwds)
        return self.func(self,
                         *(self.args + args),
                         **call_kwds)

class C:
    def example(self, *args, **kwds):
        print(self, args, kwds)
    fails = functools.partial(example, 1, 2, 3, x=1)
    works = partialmethod(example, 1, 2, 3, x=1)

>>> C().fails()
1 (2, 3) {'x': 1}
>>> C().works()
<__main__.C object at 0x7f91cefeea90> (1, 2, 3) {'x': 1}
>>> C().fails(4, 5, 6)
1 (2, 3, 4, 5, 6) {'x': 1}
>>> C().works(4, 5, 6)
<__main__.C object at 0x7f91cefeea10> (1, 2, 3, 4, 5, 6) {'x': 1}

[alonho]

I just want to make sure I understand the semantics concerning class methods, the following example demonstrates a usage similar to regular methods as much as possible:

class A(object):
    def add(self, x, y):
        print(self)
        return x + y
    add10 = partialmethod(add, 10)
    add10class = classmethod(partialmethod(add, 10))

assert A().add10(5) == 15 # prints <main.A object at 0x1097e1390>
assert A.add10class(5) == 15 # prints <class '__main__.A'>

Another option would be to return a class-bound partial from the __get__ method. It's not as consistent as the first example but perhaps nicer:

class A(object):
    def add(self, x, y):
        print(self)
        return x + y
    add10 = partialmethod(add, 10)

assert A().add10(5) == 15 # prints <main.A object at 0x1097e1390>
assert A.add10(5) == 15 # prints <class '__main__.A'>

Is the first option what you had in mind?

[ncoghlan]

On 26 Oct 2013 05:28, "alon horev" <report@bugs.python.org> wrote:

Is the first option what you had in mind?

That's actually an interesting question. I was going to say yes, but then I
realised it would be better to just "do the right thing" when the
underlying object was a classmethod descriptor, rather than composing them
the other way around.

That view means we should be delegating __get__ to the underlying
descriptor and responding appropriately to the result. And for __call__ we
then can't play games at all, since what my sketch does would be wrong when
wrapping staticmethod.

We also need to make sure the descriptor does the right thing when
@AbstractMethod is involved.

[alonho]

Here's another attempt at a consistent api with regular methods.
I'm contemplating whether partialmethod should support __call__. Given the fact partial is used to bind positional arguments, it will do the 'wrong' thing when calling the partialmethod directly and will shift all positional arguments (example at the last line of the snippet).

I also think staticmethod in this context is useless but agree consistency is a thing (you could just use partial instead).

If consistency hadn't held us back, the first proposal of partialmethod, working both for instances and classes, would have been most elegant.

from functools import partial

class partialmethod(object):

    def __init__(self, func, *args, **keywords):
        self.func = func
        self.args = args
        self.keywords = keywords

    def __call__(self, *args, **keywords):
        call_keywords = {}
        call_keywords.update(self.keywords)
        call_keywords.update(keywords)
        return self.func(*(self.args + args), **call_keywords)
    
    def __get__(self, obj, cls):
        return partial(self.func.__get__(obj, cls), *self.args, **self.keywords)

class A(object):
    def add(self, x, y):
        print(self)
        return x + y
    add10 = partialmethod(add, 10)
    add10class = partialmethod(classmethod(add), 10)
    add10static = partialmethod(staticmethod(add), 'self', 10)

assert A().add10(5) == 15 # prints <main.A object at 0x1097e1390>
assert A.add10class(5) == 15 # prints <class '__main__.A'>
assert A.add10static(5) == 15 # prints self
assert A.add10(2, 3) == 5 # prints 10 because the first positional argument is self..

Once we approve of the API I'll provide a full fledged patch.

[ncoghlan]

I like your suggestion of not providing __call__(), as I don't see a way to make it work with arbitrary underlying descriptors, and neither classmethod nor staticmethod is callable.

In terms of usage, I think this approach will be OK, as in practice I expect @classmethod, etc, will be applied to the initial method definition as appropriate, so they won't need to be inline except in test cases like these.

Additional test cases needed:

A().add10class(5)
A().add10static(5)
A.add10(A(), 5)

All three should produce 15 as the result, and print the same thing as the following:

    A.add10class(5)
    A.add10static(5)
    A().add10(5)

A test method that uses a partial instance as the callable would also be helpful in assuring the edge cases are covered.

I believe it may be necessary to have a __get__ method something like:

    def _make_unbound_method(self, cls):
        def _unbound_method(*args, **kwds):
            call_keywords = self.keywords.copy()
            call_keywords.update(keywords)
            return self.func(*(self.args + args), **call_keywords)
        _unbound_method.__objclass__ = cls
        return _unbound_method


    def __get__(self, obj, cls):
        get = getattr(self.func, "__get__")
        if get is None:
            if obj is None:
                return self._make_unbound_method(cls)
            callable = self.func
        else:
            callable = get(obj, cls)
            if callable is self.func:
                return self._make_unbound_method(cls)
        return partial(callable, *self.args, **self.keywords)

[alonho]

I think the following test demonstrates the API we're looking for.

1. Am I missing any functionality?
2. How does partialmethod relate to @absolutemethod?

from functools import partial

class partialmethod(object):

    def __init__(self, func, *args, **keywords):
        # func could be a descriptor like classmethod which isn't callable,
        # so we can't inherit from partial (it verifies func is callable)
        if isinstance(func, partialmethod):
            # flattening is mandatory in order to place cls/self before all other arguments
            # it's also more efficient since only one function will be called
            self.func = func.func
            self.args = func.args + args
            self.keywords = {}
            self.keywords.update(func.keywords)
            self.keywords.update(keywords)
        else:
            self.func = func
            self.args = args
            self.keywords = keywords
    
    def _make_unbound_method(self):
        def _method(*args, **keywords):
            keywords.update(self.keywords)
            cls_or_self, *rest = args
            call_args = (cls_or_self,) + self.args + tuple(rest)
            return self.func(*call_args, **keywords)
        return _method

    def __get__(self, obj, cls):
        get = getattr(self.func, "__get__", None)
        if get is None:
            if obj is None:
                return self._make_unbound_method()
            return partial(self._make_unbound_method(), obj) # returns a bound method
        else:
            callable = get(obj, cls)
            if callable is self.func:
                return self._make_unbound_method()
        return partial(callable, *self.args, **self.keywords)

class A(object):
    def add(self, x, y):
        return self, x + y

    add10 = partialmethod(add, 10)
    add10class = partialmethod(classmethod(add), 10)
    add10static = partialmethod(staticmethod(add), 'self', 10)

    return15 = partialmethod(add10, 5) # nested partialmethod
    
    add2partial = partial(add, x=2)
    return12 = partialmethod(add2partial, y=10) # partialmethod over partial
    
def test():
    cls = A
    instance = cls()

assert instance.add10class(5) == (cls, 15)
assert cls.add10class(5) == (cls, 15)

assert instance.add10static(5) == ('self', 15)
assert cls.add10static(5) == ('self', 15)

assert instance.add10(5) == (instance, 15)
assert cls.add10(instance, 5) == (instance, 15)

assert instance.return15() == (instance, 15)
assert cls.return15(instance) == (instance, 15)

assert instance.return12() == (instance, 12)
assert cls.return12(instance) == (instance, 12)

test()

[ncoghlan]

On 27 Oct 2013 22:17, "alon horev" <report@bugs.python.org> wrote:

alon horev added the comment:

I think the following test demonstrates the API we're looking for.

1. Am I missing any functionality?

The keyword arg handling is backwards for unbound methods (the call time
kwds should override the preset ones).

Otherwise, looks good to me.

2. How does partialmethod relate to @absolutemethod?

Do you mean @AbstractMethod?

We need a __isabstractmethod__ property implementation that delegates the
question to the underlying descriptor.

See http://docs.python.org/dev/library/abc#abc.abstractmethod for details.

[alonho]

Adding a patch with tests and documentation. Please feel free to comment on anything: my English, coding/testing style.

I'm pretty sure the documentation can be better but it turns out I speak better Python than English.

Two decisions I've made and unsure of:

1. I didn't use @wraps or copied attributes from the wrapped function (doc, __dict__) to the partialmethod object. This is intentionally so partial and partialmethod would have similar semantics.
2. I've implemented a __repr__ although in all cases __get__ returns a partial object or bound method. I consider it nice for debugging when looking at an object's __dict__.

[alonho]

I've changed the test according to the code review. Thanks

[ncoghlan]

Updated patch based on Alon's last patch.

The major functional change is to ensure __self__ is set appropriately on any bound methods returned by the descriptor.

I also updated the docs and docstring, and added a What's New entry (as well as rewording the existing entry for functools.singledispatch)

There were a few other cosmetic changes, with the most noticeable being moving the partialmethod implementation and tests adjacent to the existing ones for functools.partial.

Assuming nobody pokes any significant holes in this idea and implementation in the meantime, I'll commit this before beta 1.

[python-dev]

New changeset 46d3c5539981 by Nick Coghlan in branch 'default':
Issue bpo-4331: Added functools.partialmethod
http://hg.python.org/cpython/rev/46d3c5539981

[vajrasky]

Should we add partialmethod to __all__ for consistency?

[ncoghlan]

Indeed, added to __all__ in http://hg.python.org/cpython/rev/ac1685661b07