[BUG]: `py::cast()` from `std::shared_ptr` throws with custom holder

[pobrn]

Required prerequisites

• Make sure you've read the documentation. Your issue may be addressed there.
• Search the issue tracker and Discussions to verify that this hasn't already been reported. +1 or comment there if it has.
• Consider asking first in the Gitter chat room or in a Discussion.

What version (or hash if on master) of pybind11 are you using?

3.0.4

Problem description

The libcamera project provides python bindings via pybind11. One particular type has a private destructor and is managed via std::shared_ptr. It has a custom holder to work around the need for a public destructor.

This has worked so far, but now it does work not with pybind11 3.0.4. Specifically, calling py::cast() on an std::shared_ptr of that type in question throws:

>>> import libcamera
>>> cm = libcamera.CameraManager.singleton()
>>> cm.cameras
RuntimeError: Unable to convert std::shared_ptr<T> to Python when the bound type does not use std::shared_ptr or py::smart_holder as its holder type

As far as I can see this has been added by #6008. So I'm wondering, is this exception diagnosing a real issue, or the above PR missed some cases?

It seems to work with py::smart_holder, but this code predates that, so it wasn't used.

holder type

template<typename T>
class PyCameraSmartPtr
{
public:
	using element_type = T;

	PyCameraSmartPtr()
	{
	}

	explicit PyCameraSmartPtr(T *)
	{
		throw std::runtime_error("invalid SmartPtr constructor call");
	}

	explicit PyCameraSmartPtr(std::shared_ptr<T> p)
		: ptr_(std::move(p))
	{
	}

	T *get() const { return ptr_.get(); }

	operator std::shared_ptr<T>() const { return ptr_; }

private:
	std::shared_ptr<T> ptr_;
};

PYBIND11_DECLARE_HOLDER_TYPE(T, PyCameraSmartPtr<T>)

// ...

auto pyCamera = py::class_<Camera, PyCameraSmartPtr<Camera>>(m, "Camera");

Reproducible example code

Is this a regression? Put the last known working version here if it is.

3.0.2 (?)

[rwgk]

@virtuald is there a chance that you could look into this issue?

[virtuald]

Sure.

[virtuald]

I asked GPT to reproduce this in the pybind11 unit tests, and it did. I looked a bit at it too, and I asked GPT to write an explanation of why your code worked previously without any problems. It came to the same conclusion that I did, so I'll paste it's explanation here:

---

GPT's explanation

Most likely: it “worked” by accident, because the old code relied on a latent UB path that happened to be benign for this holder.

What happened before #6008:

• Returning std::shared_ptr always went through the same path, even if T was bound with a different holder type.
• For a custom holder, pybind11 ended up treating the address of a std::shared_ptr as though it were the address of the bound holder object.

That is not type-safe in C++, but your holder has a very friendly shape:

  class PyCameraSmartPtr {                                                                                                 
      std::shared_ptr<T> ptr_;                                                                                             
      ...                                                                                                                  
  };                                                                                                                       

So in practice, on common compilers/ABIs:

• the holder object is basically just a wrapper around one std::shared_ptr
• copying/destruction behave the same way as copying/destructing that member
• no extra state, base classes, vtable, padding-sensitive logic, etc.

So the old unsafe path could appear to work perfectly.

Why they may never have seen issues:

1. The holder layout is simple
   • Single std::shared_ptr member.
   • No extra bookkeeping to get corrupted.
2. The usage pattern is mild
   • Just converting shared ownership into Python objects.
   • No odd inheritance / aliasing / mixed-holder edge cases.
3. UB often stays invisible
   • Especially when object layout happens to match what the code wrongly assumes.
   • Different compiler, STL, optimization level, sanitizer, or tiny holder change could break it.

Why #6008 changed behavior:

• That PR fixed a real crash/UB class elsewhere by adding a guard: pybind11 now refuses to convert std::shared_ptr to Python unless the bound holder is known-safe (std::shared_ptr or smart_holder).
• Your case was one of the “used to work accidentally, but isn’t explicitly supported” cases caught by that guard.

So: the user likely never saw problems because their holder was a layout-compatible wrapper in practice, even though
pybind11 was using it through an unsupported path.

---

The question then becomes what to do about this, if anything. If smart holder works for you, I think the easiest way to proceed is to replace your custom holder with the smart holder instead.

GPT has theorized about some ways to fix this, but they seem involved and require an ABI bump. #6065 is one way it found to fix it without an ABI bump, but that approach is a little hard to follow and I don't think we should do that.

[rwgk]

A quick general remark, JIC it helps you make decisions:

We can afford an ABI bump: master has one already. There was no release yet after the bump, so bumping more would be reasonable.

I'll look more closely tomorrow.

[virtuald]

#6066 is the abi-bumping version. It's directionally correct, but I think it can be simplified. I'll have to look at it a different evening.

[rwgk]

Generated with GPT-5.5 1M Extra High

TL;DR: Me and the agent completely agree with @virtuald

I'll focus on 6066 next.

---

Issue 6064 / PR 6065 / PR 6066 Assessment

Prompt

Could you please fetch views of:

issue 6064

PR 6065

PR 6066

Could you please carefully analyze the original issue, the two solution, summarize them for me, and give me your opinion of how we should proceed?

Response

I fetched #6064, #6065, #6066, and also checked #6008 for context.

Summary

#6064 is a real regression for libcamera-style bindings: a type with a private destructor is bound with a custom holder that wraps std::shared_ptr<T>. Since #6008, py::cast(std::shared_ptr<T>) rejects types whose registered holder is not std::shared_ptr or py::smart_holder.

The old behavior was not generally safe. It could treat a std::shared_ptr<T> object as if it were the custom holder object. For libcamera's holder this happened to work because the holder is basically just a std::shared_ptr<T> wrapper, but that was layout-dependent UB. So #6008 diagnosed a real class of bugs, but it also blocked a supportable case: custom holders explicitly constructible from std::shared_ptr<T>.

PR Comparison

#6065 fixes it without an ABI bump by using a thread-local payload and passing a sentinel through the existing init_instance holder pointer path. It is clever, but I agree with Dustin's concern: it is hard to reason about and makes an already subtle object-construction path more implicit.

#6066 adds explicit callback plumbing to type_record / type_info: if a bound holder is constructible from std::shared_ptr<T>, py::cast(std::shared_ptr<T>) can construct the registered custom holder from an erased aliasing shared pointer. This is the cleaner design direction because the capability is recorded directly in type metadata instead of tunneled through TLS.

Both PRs currently share the same test.

Recommendation

Proceed with #6066 as the basis, not #6065.

Before merging, I would simplify and harden #6066:

• Factor the duplicated policy/object-creation logic shared with smart_holder_from_shared_ptr.
• Decide explicitly whether to bump PYBIND11_INTERNALS_VERSION; the fetched patch changes type_info, and the file itself says that requires a bump.
• Keep the regression test.
• Add/confirm a negative test that an incompatible custom holder still throws, preserving the safety fix from #6008.

For a released 3.0.x patch where an ABI bump is unacceptable, I would prefer recommending py::smart_holder as the workaround over merging the #6065 TLS approach. For master, where an ABI bump is acceptable, #6066 is the right path.

[rwgk]

To address this point in my previous comment:

Decide explicitly whether to bump PYBIND11_INTERNALS_VERSION; the fetched patch changes type_info, and the file itself says that requires a bump.

Cursor GPT-5.5 1M Extra High analysis:

---

This definitely needs a PYBIND11_INTERNALS_VERSION bump.

Trace:

1. struct internals is the cross-extension shared registry. It stores type_info * in multiple maps:

/// Internal data structure used to track registered instances and types.
/// Whenever binary incompatible changes are made to this structure,
/// `PYBIND11_INTERNALS_VERSION` must be incremented.
struct internals {
#if PYBIND11_INTERNALS_VERSION >= 12
    fast_type_map<type_info *> registered_types_cpp_fast;
#endif

    // std::type_index -> pybind11's type information
    type_map<type_info *> registered_types_cpp;
    // PyTypeObject* -> base type_info(s)
    std::unordered_map<PyTypeObject *, std::vector<type_info *>> registered_types_py;

2. type_info itself is explicitly documented as internals-version-sensitive:

/// Additional type information which does not fit into the PyTypeObject.
/// Changes to this struct also require bumping `PYBIND11_INTERNALS_VERSION`.
struct type_info {
    PyTypeObject *type;
    const std::type_info *cpptype;
    size_t type_size, type_align, holder_size_in_ptrs;
    void *(*operator_new)(size_t);
    void (*init_instance)(instance *, const void *);
    void (*dealloc)(value_and_holder &v_h);
    void (*init_instance_from_shared_ptr)(instance *, const std::shared_ptr<void> *) = nullptr;

3. When a py::class_ is registered, pybind11 allocates a type_info, fills it, then publishes the pointer into internals:

/* Register supplemental type information in C++ dict */
auto *tinfo = new detail::type_info();
tinfo->type = reinterpret_cast<PyTypeObject *>(m_ptr);
tinfo->cpptype = rec.type;
// ...
tinfo->init_instance = rec.init_instance;
tinfo->dealloc = rec.dealloc;
tinfo->init_instance_from_shared_ptr = rec.init_instance_from_shared_ptr;
// ...

with_internals([&](internals &internals) {
    auto tindex = std::type_index(*rec.type);
    tinfo->direct_conversions = &internals.direct_conversions[tindex];
    auto &local_internals = get_local_internals();
    if (rec.module_local) {
        local_internals.registered_types_cpp[rec.type] = tinfo;
    } else {
        internals.registered_types_cpp[tindex] = tinfo;
#if PYBIND11_INTERNALS_VERSION >= 12
        internals.registered_types_cpp_fast[rec.type] = tinfo;
#endif
    }

4. Python-type lookup also stores the same pointer in registered_types_py:

internals.registered_types_py[reinterpret_cast<PyTypeObject *>(m_ptr)] = {tinfo};

5. Later casts/loaders retrieve the shared type_info * through get_type_info(...), which consults local then global internals:

PYBIND11_NOINLINE detail::type_info *get_type_info(const std::type_info &tp,
                                                   bool throw_if_missing = false) {
    PYBIND11_LOCK_INTERNALS(get_internals());
    if (auto *ltype = get_local_type_info_lock_held(tp)) {
        return ltype;
    }
    if (auto *gtype = get_global_type_info_lock_held(tp)) {
        return gtype;
    }

And the global lookup returns pointers from internals.registered_types_cpp(_fast):

inline detail::type_info *get_global_type_info_lock_held(const std::type_info &tp) {
    detail::type_info *type_info = nullptr;
    auto &internals = get_internals();
#if PYBIND11_INTERNALS_VERSION >= 12
    auto &fast_types = internals.registered_types_cpp_fast;
#endif
    auto &types = internals.registered_types_cpp;
// ...
    auto it = types.find(std::type_index(tp));
    if (it != types.end()) {
// ...
        type_info = it->second;
    }
    return type_info;
}

6. The shared internals capsule name includes PYBIND11_INTERNALS_VERSION, so bumping it prevents extensions compiled with different type_info layouts from sharing the same internals object:

#define PYBIND11_INTERNALS_ID                                                                     \
    "__pybind11_internals_v" PYBIND11_TOSTRING(PYBIND11_INTERNALS_VERSION)                        \
        PYBIND11_COMPILER_TYPE_LEADING_UNDERSCORE PYBIND11_PLATFORM_ABI_ID "__"

#define PYBIND11_MODULE_LOCAL_ID                                                                  \
    "__pybind11_module_local_v" PYBIND11_TOSTRING(PYBIND11_INTERNALS_VERSION)                     \
        PYBIND11_COMPILER_TYPE_LEADING_UNDERSCORE PYBIND11_PLATFORM_ABI_ID "__"

Bottom line: type_info is heap-allocated per registered class, but the pointer is published through struct internals and consumed across pybind11 extension boundaries. Adding a field changes the layout expected by code compiled against the headers, so PYBIND11_INTERNALS_VERSION should be bumped.