Add basic support for tag-based static polymorphism

docs/advanced/classes.rst

@@ -999,3 +999,94 @@ described trampoline:
     requires a more explicit function binding in the form of
     ``.def("foo", static_cast<int (A::*)() const>(&Publicist::foo));``
     where ``int (A::*)() const`` is the type of ``A::foo``.
+
+Custom automatic downcasters
+============================
+
+As explained in :ref:`inheritance`, pybind11 comes with built-in
+understanding of the dynamic type of polymorphic objects in C++; that
+is, returning a Pet to Python produces a Python object that knows it's
+wrapping a Dog, if Pet has virtual methods and pybind11 knows about
+Dog and this Pet is in fact a Dog. Sometimes, you might want to
+provide this automatic downcasting behavior when creating bindings for
+a class hierarchy that does not use standard C++ polymorphism, such as
+LLVM [#f4]_. As long as there's some way to determine at runtime
+whether a downcast is safe, you can proceed by specializing the
+``pybind11::detail::polymorphic_type_hook`` template:
+
+.. code-block:: cpp
+
+    enum class PetKind { Cat, Dog, Zebra };
+    struct Pet {
+        const PetKind kind;
+        int age = 0;
+      protected:
+        Pet(PetKind _kind) : kind(_kind) {}
+    };
+    struct Dog : Pet {
+        Dog() : Pet(PetKind::Dog) {}
+        std::string sound = "woof!";
+        std::string bark() const { return sound; }
+    };
+
+    namespace pybind11 { namespace detail {
+        template<> struct polymorphic_type_hook<Pet> {
+            static const void *get(const Pet *src, const std::type_info*& type) {
+                // note that src may be nullptr
+                if (src && src->kind == PetKind::Dog) {
+                    type = &typeid(Dog);
+                    return static_cast<const Dog*>(src);
+                }
+                return src;
+            }
+        };
+    }} // namespace pybind11::detail
+
+When pybind11 wants to convert a C++ pointer of type ``Base*`` to a
+Python object, it calls ``polymorphic_type_hook<Base>::get()`` to
+determine if a downcast is possible. The ``get()`` function should use
+whatever runtime information is available to determine if its ``src``
+parameter is in fact an instance of some class ``Derived`` that
+inherits from ``Base``. If it finds such a ``Derived``, it sets ``type
+= &typeid(Derived)`` and returns ``static_cast<const Derived*>(src)``.
+Otherwise, it just returns ``src``, leaving ``type`` at its default
+value of nullptr. It's OK to return a type that pybind11 doesn't know
+about; in that case, no downcasting will occur, and the original
+``src`` pointer will be used with its static type ``Base*``.
+
+It is critical that the return value and ``type`` argument of
+``get()`` agree with each other: if ``type`` is set to something
+non-null, the returned pointer must point to the start of an object
+whose type is ``type``. If the hierarchy being exposed uses only
+single inheritance, a simple ``return src;`` will achieve this just
+fine, but in the general case, you must cast ``src`` to the
+appropriate derived-class pointer before allowing it to be cast to
+``void*``.
+
+pybind11's standard support for downcasting objects whose types
+have virtual methods is implemented using ``polymorphic_type_hook`` too:
+
+.. code-block:: cpp
+
+    template <typename itype>
+    struct polymorphic_type_hook<itype, enable_if_t<std::is_polymorphic<itype>::value>>
+    {
+        static const void *get(const itype *src, const std::type_info*& type) {
+            type = src ? &typeid(*src) : nullptr;
+            return dynamic_cast<const void*>(src);
+        }
+    };
+
+This uses the standard C++ ability to determine the most-derived type
+of a polymorphic object using ``typeid()`` and to cast a base pointer
+to that most-derived type (even if you don't know what it is) using
+``dynamic_cast<void*>``.
+
+.. [#f4] https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html
+
+.. seealso::
+
+    The file :file:`tests/test_tagbased_polymorphic.cpp` contains a
+    more complete example, including a demonstration of how to provide
+    automatic downcasting for an entire class hierarchy without
+    writing one get() function for each class.

docs/classes.rst

@@ -228,8 +228,8 @@ just brings them on par.
 
 .. _inheritance:
 
-Inheritance and automatic upcasting
-===================================
+Inheritance and automatic downcasting
+=====================================
 
 Suppose now that the example consists of two data structures with an
 inheritance relationship:
@@ -298,7 +298,7 @@ inheritance relationship. This is reflected in Python:
 
     >>> p = example.pet_store()
     >>> type(p)  # `Dog` instance behind `Pet` pointer
-    Pet          # no pointer upcasting for regular non-polymorphic types
+    Pet          # no pointer downcasting for regular non-polymorphic types
     >>> p.bark()
     AttributeError: 'Pet' object has no attribute 'bark'
 
@@ -330,11 +330,11 @@ will automatically recognize this:
 
     >>> p = example.pet_store2()
     >>> type(p)
-    PolymorphicDog  # automatically upcast
+    PolymorphicDog  # automatically downcast
     >>> p.bark()
     u'woof!'
 
-Given a pointer to a polymorphic base, pybind11 performs automatic upcasting
+Given a pointer to a polymorphic base, pybind11 performs automatic downcasting
 to the actual derived type. Note that this goes beyond the usual situation in
 C++: we don't just get access to the virtual functions of the base, we get the
 concrete derived type including functions and attributes that the base type may

include/pybind11/cast.h

@@ -774,9 +774,41 @@ template <typename T1, typename T2> struct is_copy_constructible<std::pair<T1, T
     : all_of<is_copy_constructible<T1>, is_copy_constructible<T2>> {};
 #endif
 
+// polymorphic_type_hook<itype>::get(src, tinfo) determines whether the object pointed
+// to by `src` actually is an instance of some class derived from `itype`.
+// If so, it sets `tinfo` to point to the std::type_info representing that derived
+// type, and returns a pointer to the start of the most-derived object of that type
+// (in which `src` is a subobject; this will be the same address as `src` in most
+// single inheritance cases). If not, or if `src` is nullptr, it simply returns `src`
+// and leaves `tinfo` at its default value of nullptr.
+//
+// The default polymorphic_type_hook just returns src. A specialization for polymorphic
+// types determines the runtime type of the passed object and adjusts the this-pointer
+// appropriately via dynamic_cast<void*>. This is what enables a C++ Animal* to appear
+// to Python as a Dog (if Dog inherits from Animal, Animal is polymorphic, Dog is
+// registered with pybind11, and this Animal is in fact a Dog).
+//
+// You may specialize polymorphic_type_hook yourself for types that want to appear
+// polymorphic to Python but do not use C++ RTTI. (This is a not uncommon pattern
+// in performance-sensitive applications, used most notably in LLVM.)
+template <typename itype, typename SFINAE = void>
+struct polymorphic_type_hook
+{
+    static const void *get(const itype *src, const std::type_info*&) { return src; }
+};
+template <typename itype>
+struct polymorphic_type_hook<itype, enable_if_t<std::is_polymorphic<itype>::value>>
+{
+    static const void *get(const itype *src, const std::type_info*& type) {
+        type = src ? &typeid(*src) : nullptr;
+        return dynamic_cast<const void*>(src);
+    }
+};
+
 /// Generic type caster for objects stored on the heap
 template <typename type> class type_caster_base : public type_caster_generic {
     using itype = intrinsic_t<type>;
+
 public:
     static constexpr auto name = _<type>();
 
@@ -795,30 +827,25 @@ template <typename type> class type_caster_base : public type_caster_generic {
 
     // Returns a (pointer, type_info) pair taking care of necessary RTTI type lookup for a
     // polymorphic type.  If the instance isn't derived, returns the non-RTTI base version.
-    template <typename T = itype, enable_if_t<std::is_polymorphic<T>::value, int> = 0>
     static std::pair<const void *, const type_info *> src_and_type(const itype *src) {
-        const void *vsrc = src;
         auto &cast_type = typeid(itype);
         const std::type_info *instance_type = nullptr;
-        if (vsrc) {
-            instance_type = &typeid(*src);
-            if (!same_type(cast_type, *instance_type)) {
-                // This is a base pointer to a derived type; if it is a pybind11-registered type, we
-                // can get the correct derived pointer (which may be != base pointer) by a
-                // dynamic_cast to most derived type:
-                if (auto *tpi = get_type_info(*instance_type))
-                    return {dynamic_cast<const void *>(src), const_cast<const type_info *>(tpi)};
-            }
+        const void *vsrc = polymorphic_type_hook<itype>::get(src, instance_type);
+        if (instance_type && !same_type(cast_type, *instance_type)) {
+            // This is a base pointer to a derived type. If the derived type is registered
+            // with pybind11, we want to make the full derived object available.
+            // In the typical case where itype is polymorphic, we get the correct
+            // derived pointer (which may be != base pointer) by a dynamic_cast to
+            // most derived type. If itype is not polymorphic, we won't get here
+            // except via a user-provided specialization of polymorphic_type_hook,
+            // and the user has promised that no this-pointer adjustment is
+            // required in that case, so it's OK to use static_cast.
+            if (const auto *tpi = get_type_info(*instance_type))
+                return {vsrc, tpi};
         }
         // Otherwise we have either a nullptr, an `itype` pointer, or an unknown derived pointer, so
         // don't do a cast
-        return type_caster_generic::src_and_type(vsrc, cast_type, instance_type);
-    }
-
-    // Non-polymorphic type, so no dynamic casting; just call the generic version directly
-    template <typename T = itype, enable_if_t<!std::is_polymorphic<T>::value, int> = 0>
-    static std::pair<const void *, const type_info *> src_and_type(const itype *src) {
-        return type_caster_generic::src_and_type(src, typeid(itype));
+        return type_caster_generic::src_and_type(src, cast_type, instance_type);
     }
 
     static handle cast(const itype *src, return_value_policy policy, handle parent) {

tests/CMakeLists.txt

@@ -57,6 +57,7 @@ set(PYBIND11_TEST_FILES
   test_smart_ptr.cpp
   test_stl.cpp
   test_stl_binders.cpp
+  test_tagbased_polymorphic.cpp
   test_virtual_functions.cpp
 )
 

tests/test_tagbased_polymorphic.cpp

@@ -0,0 +1,138 @@
+/*
+    tests/test_tagbased_polymorphic.cpp -- test of detail::polymorphic_type_hook
+
+    Copyright (c) 2018 Hudson River Trading LLC <opensource@hudson-trading.com>
+
+    All rights reserved. Use of this source code is governed by a
+    BSD-style license that can be found in the LICENSE file.
+*/
+
+#include "pybind11_tests.h"
+#include <pybind11/stl.h>
+
+struct Animal
+{
+    enum class Kind {
+        Unknown = 0,
+        Dog = 100, Labrador, Chihuahua, LastDog = 199,
+        Cat = 200, Panther, LastCat = 299
+    };
+    static const std::type_info* type_of_kind(Kind kind);
+    static std::string name_of_kind(Kind kind);
+
+    const Kind kind;
+    const std::string name;
+
+  protected:
+    Animal(const std::string& _name, Kind _kind)
+        : kind(_kind), name(_name)
+    {}
+};
+
+struct Dog : Animal
+{
+    Dog(const std::string& _name, Kind _kind = Kind::Dog) : Animal(_name, _kind) {}
+    std::string bark() const { return name_of_kind(kind) + " " + name + " goes " + sound; }
+    std::string sound = "WOOF!";
+};
+
+struct Labrador : Dog
+{
+    Labrador(const std::string& _name, int _excitement = 9001)
+        : Dog(_name, Kind::Labrador), excitement(_excitement) {}
+    int excitement;
+};
+
+struct Chihuahua : Dog
+{
+    Chihuahua(const std::string& _name) : Dog(_name, Kind::Chihuahua) { sound = "iyiyiyiyiyi"; }
+    std::string bark() const { return Dog::bark() + " and runs in circles"; }
+};
+
+struct Cat : Animal
+{
+    Cat(const std::string& _name, Kind _kind = Kind::Cat) : Animal(_name, _kind) {}
+    std::string purr() const { return "mrowr"; }
+};
+
+struct Panther : Cat
+{
+    Panther(const std::string& _name) : Cat(_name, Kind::Panther) {}
+    std::string purr() const { return "mrrrRRRRRR"; }
+};
+
+std::vector<std::unique_ptr<Animal>> create_zoo()
+{
+    std::vector<std::unique_ptr<Animal>> ret;
+    ret.emplace_back(new Labrador("Fido", 15000));
+
+    // simulate some new type of Dog that the Python bindings
+    // haven't been updated for; it should still be considered
+    // a Dog, not just an Animal.
+    ret.emplace_back(new Dog("Ginger", Dog::Kind(150)));
+
+    ret.emplace_back(new Chihuahua("Hertzl"));
+    ret.emplace_back(new Cat("Tiger", Cat::Kind::Cat));
+    ret.emplace_back(new Panther("Leo"));
+    return ret;
+}
+
+const std::type_info* Animal::type_of_kind(Kind kind)
+{
+    switch (kind) {
+        case Kind::Unknown: break;
+
+        case Kind::Dog: break;
+        case Kind::Labrador: return &typeid(Labrador);
+        case Kind::Chihuahua: return &typeid(Chihuahua);
+        case Kind::LastDog: break;
+
+        case Kind::Cat: break;
+        case Kind::Panther: return &typeid(Panther);
+        case Kind::LastCat: break;
+    }
+
+    if (kind >= Kind::Dog && kind <= Kind::LastDog) return &typeid(Dog);
+    if (kind >= Kind::Cat && kind <= Kind::LastCat) return &typeid(Cat);
+    return nullptr;
+}
+
+std::string Animal::name_of_kind(Kind kind)
+{
+    std::string raw_name = type_of_kind(kind)->name();
+    py::detail::clean_type_id(raw_name);
+    return raw_name;
+}
+
+namespace pybind11 {
+namespace detail {
+    template <typename itype>
+    struct polymorphic_type_hook<itype, enable_if_t<std::is_base_of<Animal, itype>::value>>
+    {
+        static const void *get(const itype *src, const std::type_info*& type)
+        { type = src ? Animal::type_of_kind(src->kind) : nullptr; return src; }
+    };
+}
+}
+
+TEST_SUBMODULE(tagbased_polymorphic, m) {
+    py::class_<Animal>(m, "Animal")
+        .def_readonly("name", &Animal::name);
+    py::class_<Dog, Animal>(m, "Dog")
+        .def(py::init<std::string>())
+        .def_readwrite("sound", &Dog::sound)
+        .def("bark", &Dog::bark);
+    py::class_<Labrador, Dog>(m, "Labrador")
+        .def(py::init<std::string, int>(), "name"_a, "excitement"_a = 9001)
+        .def_readwrite("excitement", &Labrador::excitement);
+    py::class_<Chihuahua, Dog>(m, "Chihuahua")
+        .def(py::init<std::string>())
+        .def("bark", &Chihuahua::bark);
+    py::class_<Cat, Animal>(m, "Cat")
+        .def(py::init<std::string>())
+        .def("purr", &Cat::purr);
+    py::class_<Panther, Cat>(m, "Panther")
+        .def(py::init<std::string>())
+        .def("purr", &Panther::purr);
+    m.def("create_zoo", &create_zoo);
+};

tests/test_tagbased_polymorphic.py

@@ -0,0 +1,20 @@
+from pybind11_tests import tagbased_polymorphic as m
+
+
+def test_downcast():
+    zoo = m.create_zoo()
+    assert [type(animal) for animal in zoo] == [
+        m.Labrador, m.Dog, m.Chihuahua, m.Cat, m.Panther
+    ]
+    assert [animal.name for animal in zoo] == [
+        "Fido", "Ginger", "Hertzl", "Tiger", "Leo"
+    ]
+    zoo[1].sound = "woooooo"
+    assert [dog.bark() for dog in zoo[:3]] == [
+        "Labrador Fido goes WOOF!",
+        "Dog Ginger goes woooooo",
+        "Chihuahua Hertzl goes iyiyiyiyiyi and runs in circles"
+    ]
+    assert [cat.purr() for cat in zoo[3:]] == ["mrowr", "mrrrRRRRRR"]
+    zoo[0].excitement -= 1000
+    assert zoo[0].excitement == 14000