Keyword arguments and generalized unpacking for C++ API

[dean0x7d]

General idea

A Python function can be called with the syntax:

foo(a1, a2, *args, ka=1, kb=2, **kwargs)

This PR adds support for the equivalent syntax in C++:

foo(a1, a2, *args, "ka"_a=1, "kb"_a=2, **kwargs)

In addition, generalized unpacking is implemented, as per PEP 448, which allows calls with multiple * and ** unpackings in Python:

bar(*args1, 99, *args2, 101, **kwargs1, kz=200, **kwargs2)

and in C++:

bar(*args1, 99, *args2, 101, **kwargs1, "kz"_a=200, **kwargs2)

Impact on compile time

The new functionality is SFINAEd-off into a separate instantiation path from the existing positional-only arguments. Thus, the compile time will not increase for existing code. When (and only when) keywords and/or generalized unpacking are used, a little more template metaprogramming is in play, but it's not really any more complicated than the existing def/arg machinery for naming arguments and assigning default values. There are no compile time slowdowns as far as I've seen.

Python version compatibility

PEP 448 is implemented only in Python >= 3.5, but the C++ API allows these kinds of calls for any Python version. Limiting it by version is also possible, but I don't think it's necessary. The function calls are still perfectly valid, it's just that:

f(**kwargs1, **kwargs2)  # Python 3.5

would look like:

kwargs1.update(kwargs2)  # Python < 3.5
f(**kwargs1)

Applications

This PR also implements a few functions using this new functionality.

Python's print function is replicated in the C++ API including the keyword arguments sep, end, file, flush. E.g.:

py::print(v1, v2, "sep"_a=" -- ");

The str.format method is added. Together with the _s UDL for str, this allows the following syntax in C++:

auto str1 = "Hello, {}! Your number is {}"_s.format("World", 42);
// or
auto str2 = "Hello, {name}! Your number is {number}"_s.format("number"_a=42, "name"_a="World");

The py::dict class also gets a keyword constructor which replicates its Python counterpart:

auto d = dict("number"_a=42, "name"_a="World");

Feedback?

I hope this is not considered too much of an abuse of C++. Let me know if this is OK so far.

[wjakob]

this type alias is a good idea -- should have added that much earlier :)

[wjakob]

For consistency, it would be nice to this make_caster alias in all places where type_caster<typename intrinsic_type<type>::type> is currently used (quite a few I think)

[wjakob]

After a brief look, this seems like very a nice idea (and I like the print function). I'll have to set a aside a bit more time to look at all the metatemplate code but for now it seems very reasonable.

Is there a symmetrical change that would also have to be done to function call handlers, or are these feature-complete as far as PEP 448 goes?

[dean0x7d]

Is there a symmetrical change that would also have to be done to function call handlers, or are these feature-complete as far as PEP 448 goes?

As far as I can tell, everything is already fine in the other direction. Generalized unpacking is just syntax sugar for the interpreter. The multiple tuples/dicts are merged and the C API only sees the usual pair of args/kwargs.

[aldanor]

This is pretty cool. A few lines in the changelog would be nice

[dean0x7d]

Updated docs with the new call syntax. Updated changelog with all the new features.
The py::dict class also got a keyword constructor which replicates its Python counterpart:

auto d = dict("number"_a=42, "name"_a="World");

MSVC had an issue which required a slightly ugly workaround. It's not too bad, but perhaps somebody has a better solution.

[wjakob]

Hmm, this change (pointer instead of an explicit copy of value) seems problematic to me. It means that arg_t cannot receive something that is constructed on the fly, e.g.

m.def("fun", py::arg("arg") = MyArg(1, 2, 3));

[dean0x7d]

Temporary objects (even ones which are not bound to a const&) are only destroyed after the full expression is evaluated. A full expression is one which is not contained within any other expression (so essentially, temp object lifetime is extended to a semicolon).

This makes the following line perfectly safe:

f("arg"_a=MyArg(1, 2, 3));

On the other hand,

auto a = "arg"_a=MyArg(1, 2, 3);
f(a);

would not be safe, but this is currently a compile time error because unpacking_collector only accepts rvalues: arg<T>&&. (Although, I do need to add this safeguard to process_attribute as well.)

That said, I think the ideal implementation of arg_t would be to hold a py::object instead of T:

struct arg_t : arg {
    template <typename T>
    arg_t(const char *name, const T &value, const char *descr = nullptr)
        : arg(name), value(make_caster<T>::cast(value, return_value_policy::automatic)), descr(descr) { }
    object value;
    const char *descr;
}

The to-python cast would then be done right away instead of being delayed. I didn't make this change originally because default arguments are collected with the automatic policy while function calls use automatic_reference by default. Would there be any downside to always using automatic for call argument collection? (Users should be able to override by doing a manual py::cast(value, policy) when assigning an argument.)

[wjakob]

Right -- that makes sense. Is the restriction to rvalues unavoidable? If it's easy to also support your example

auto a = "arg"_a=MyArg(1, 2, 3); f(a);

with minor changes then that would be nice. Switching to py::object for the underlying argument storage in this case is fine to me. The only reason for the current state of affairs is because of the way in which def() arguments were processed previously.

Regarding value policies: this unlikely to be a serious concern for default arguments constructed at module load time, and it's always safe to make a copy (which is what will happen for const T& instances that have not been previously observed by pybind11).

[dean0x7d]

Switching to py::object would be the best case. There would be no restrictions on py::arg_t then. As a bonus, the template code would also be simplified since py::arg_t would no longer be a template class (it would only have a template constructor). py::arg_t should create it's py::object using the automatic policy which looks like the only reasonable choice.

There is just one small inconsistency that this would introduce: Keyword arguments would then be passed to function calls with the automatic policy, while positional arguments would use automatic_reference (which is the function call default). I don't think this a big issue, but maybe you have some thoughts.

[wjakob]

I don't think it is a big issue. In any case, the cast to a py::object could also be done manually with any kind of policy, which is then assigned to py:arg_t.

[wjakob]

I think this one is almost ready to go. I added a few comments. The only major gotcha is the pointer-vs-value storage in arg_t, can you take a look at that? Maybe I am just missing something though.

[jagerman]

Could these be written as: template <typename...> using always_true = std::true_type; and likewise for always_false?

[dean0x7d]

Yes, that was the initial implementation, however GCC 4.8 had an issue with it in one spot (which is weird since even MSVC compiled it correctly). I'll need to investigate a bit more to see what's up.

[wjakob]

Let me know when you think this is ready to merge.

[dean0x7d]

OK, if everything looks good with the changes to py::arg this should be ready to go. See the last commit message for py::arg details.

For the docs, I added a quick section for py::print, but everything is still in Advanced. I think the reorganization is better left for a separate PR. Better docs for the type wrappers can also be added at that point.

I addressed all the comments and squashed the smaller changes (remove dict.get, improved comments) with their respective commits to keep the git history relatively tidy.

[wjakob]

very compact 👍

[wjakob]

Awesome, merged! 🎉