fix: DECREFing PyObject*s from Python C API calls in v1.

[jpivarski]

A memory leak in Coffea was traced to this: we were calling Python C API functions (just PyUnicode_DecodeUTF8) that returned new refs and then passing them directly to pybind11 py::object (or its subclasses') constructors. I had thought that the py::object constructor would steal this reference and implicitly call Py_DECREF on it when the py::object enters its destructor, but this is evidently not the case. (The memory leak in #2310 toggles exactly with adding Py_DECREF into parameters2dict.)

In this PR, I have added Py_DECREF calls as late as possible in every branch of the program flow on the raw PyObject* objects directly returned by PyUnicode_DecodeUTF8. I got all of these cases, and there are no other Python C API function calls that return PyObject* in the codebase. All of the other Python C API functions called,

• PyObject_CheckBuffer
• PyBytes_Check
• PyBytes_Size
• PyBytes_AsString

return C types, not PyObject*. (The Python documentation says the char* that PyBytes_AsString returns is owned by Python and must not be released. In our code, we copy it immediately.)

This PR also updates the version number in anticipation of an Awkward v1 bug-fix release, and it fixes a test that now fails a NumPy deprecation warning (no relationship to the actual bug; it's just that main-v1 is an old branch).

There is no equivalent action to be performed on Awkward main because none of this stuff exists there.

[jpivarski]

Check: are the following memory leaks now fixed?

• Memory leak in ArrayBuilder/GrowableBuffer #567
• Massive memory leak with awkward.Array #1127
• Memory leak when iterating over array #1280
• Unbounded memory use when iterating over RecordArray #2275

I don't remember why I thought that #567 was an unresolved memory leak. (The first part of that issue addressed a different leak in ArrayBuilder; afterward, I had thought we had hints of another one, but the conversation that's there doesn't look conclusive to me now.)

[jpivarski]

@agoose77 I disabled tests of ak.to_/from_rdataframe and JAX. The RDataFrame tests were testing the last ak._v2.* version, and the JAX tests were testing the v1 version, neither of which are supported anymore. Hopefully, the remaining tests will pass.

Do you expect that enough of the infrastructure will work to do a PyPI 1.10.3 release the normal way—by creating a release in GitHub targeting the main-v1 branch (after this gets merged)?

What would be necessary to get it from PyPI into conda-forge? Does that pick up any new PyPI releases, even if they're out of order?

[agoose77]

I think the PyPI release should work just fine with the wheels.yml workflow. I'm not so sure about how we would release a patch version for an old major version on conda-forge. I think it should be possible, but let me do some digging.

[agoose77]

I checked on Matrix, and it looks like we just need to maintain a v1 branch on our feedstock. That should take care of it. Once we make the PyPI release, I can revert back to the v1 feedstock for a branch to make the release.

[jpivarski]

Alternative to using the py:: namespace in front of reinterpret_steal:

#include <pybind11/pytypes.h>

But I like the namespace to denote all things pybind11.

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (main-v1@198cf61). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 2b6d805 differs from pull request most recent head 7161d3d. Consider uploading reports for the commit 7161d3d to get more accurate results

Additional details and impacted files

[agoose77]

Did we get rid of this because it's flukey?

[jpivarski]

Yes. It's been failing in the last digits on only some platforms. This is a test of the not-completely-done v2 inside ak._v2.*, so I don't think we need to work very hard to save it.

[jpivarski]

In principle, we could remove all of ak._v2 and tests/v2 from main-v1, but I don't want to make such a large diff in this PR.

[agoose77]

I'd prefer to keep some of the tests so we don't fully regress, but I agree this is transitional code and should be treated as such. We didn't make long-term guarantees about v2-in-v1.