Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Struct members of cdef'd classes don't get destroyed #3226

Closed
david-cortes opened this issue Nov 7, 2019 · 9 comments · Fixed by #3228
Closed

Struct members of cdef'd classes don't get destroyed #3226

david-cortes opened this issue Nov 7, 2019 · 9 comments · Fixed by #3228
Milestone

Comments

@david-cortes
Copy link

If I create a cdef'd class in a cython file having a struct as class member, and then instantiate that class in python, the structs do not get destroyed when the python object is deleted. Adding an explicit call to the struct destructor in __dealloc__ will do the job however.

Example:
cppheader.hpp:

#include <vector>

typedef struct MyObject {
	int val;
	std::vector<double> big_obj;
} MyObject;

typedef struct InfoHolder {
	std::vector<MyObject> v;
	int some_val;
} InfoHolder;

void increase_size(MyObject &my_obj)
{
	my_obj.big_obj.resize(10000000, 1);
}

void force_destroy(InfoHolder &info_holder)
{
	info_holder.~InfoHolder();
}

cyfile.pyx:

from libcpp.vector cimport vector

cdef extern from "cppheader.hpp":
	ctypedef struct MyObject:
		int val
		vector[double] big_obj
	ctypedef struct InfoHolder:
		vector[MyObject] v
		int some_val
	void increase_size(MyObject &my_obj)
	void force_destroy(InfoHolder &info_holder)

cdef class PyInfoHolder:
	cdef InfoHolder info_holder
	def __init__(self):
		self.info_holder.v.resize(1);
		increase_size(self.info_holder.v.at(0))	
	def say_hello(self):
		print("hello")

pyfile.py:

from cyfile import PyInfoHolder

for i in range(100):
	py_obj = PyInfoHolder()
	py_obj.say_hello()
print("end")

Watch the memory usage when running python pyfile.py, and then compare against the following alternative cython class:

cdef class PyInfoHolder:
	cdef InfoHolder info_holder
	def __init__(self):
		self.info_holder.v.resize(1);
		increase_size(self.info_holder.v.at(0))
	def __dealloc__(self):
		force_destroy(self.info_holder)		
	def say_hello(self):
		print("hello")

(For some reason, the structs also cannot call their destructor explicitly within cython - e.g. self.info_holder.~InfoHolder() fails to compile)

@scoder scoder added the C++ label Nov 7, 2019
@scoder
Copy link
Contributor

scoder commented Nov 7, 2019

Cython doesn't handle structs in C++ any different from structs in C.
If you can, use a C++ class instead.

Does this seem worth turning into a feature?

@scoder scoder added the feature label Nov 7, 2019
@david-cortes
Copy link
Author

Yes, would be a nice addition to auto-destruct all attributes of a cdef'd class, especially considering that it's not possible to call the struct destructor in Cython unless it's wrapped in a C++ function.

Also, does it mean that if I instantiate a cdef'd class with POD types like int, double, etc. that those would also not be freed when the python object is deleted?

@da-woods
Copy link
Contributor

da-woods commented Nov 9, 2019

I've had a go at fixing it #3228. My view is it's somewhat useful: it's convention that C++ structs should only be POD, but it is only a convention so to be safe they should be destructed. It should cost almost nothing to do it (especially when they are just POD) but it could save mystery memory leaks

@david-cortes There's no issue with memory being freed built-in types like int and double - no destructor will be called, but the memory is allocated and deallocated with the Python object.

@david-cortes
Copy link
Author

@da-woods : related question: with the changes in the PR you created, what would happen if some struct member is re-assigned? e.g.

self.some_struct = SomeStruct(n = 1)
self.some_struct = SomeStruct(n = 2)

I think this currently also produces a memory leak.

@da-woods
Copy link
Contributor

da-woods commented Nov 10, 2019

@david-cortes I believe it calls the SomeStruct constructor to create a temporary then calls the copy assignment operator, then when the temporary falls out of scope (might be instant or might later in the function) the temporary will be destructed by default - this is the existing behaviour. It might possibly be worth testing for this, but I'm pretty sure it already did the right thing.

The issue was really only for structs in cdef classes, where the cdef class is allocated with malloc and freed with free. For pretty much every other case normal C++ scope rules handle appropriately.

@david-cortes
Copy link
Author

@da-woods : Was testing the assignments in the PR you had created and in the current master branch, and I'm finding some strange behavior:

#include <vector>
#include <iostream>

typedef struct MyObject {
	int val;
	std::vector<double> big_obj;
	~MyObject() {std::cout << "I'm destructed (val:" << this->val << ")" << std::endl;}
} MyObject;

void modify_val(int &val)
{
	val *= 2;
}
from libcpp.vector cimport vector

cdef extern from "cppheader.hpp":
	ctypedef struct MyObject:
		int val
		vector[double] big_obj
	void modify_val(int &val)

cdef class PyInfoHolder:
	cdef MyObject my_object

	def __init__(self):
		self.my_object.val = -1
		
		self.my_object = MyObject()
		self.my_object.val = 1
		modify_val(self.my_object.val)
		print("this is val: ", self.my_object.val)

		self.my_object = MyObject()
		self.my_object.val = 2
		modify_val(self.my_object.val)
		print("this is val: ", self.my_object.val)

		self.my_object = MyObject()
		self.my_object.val = 3
		modify_val(self.my_object.val)
		print("this is val: ", self.my_object.val)	

	def say_hello(self):
		print("hello")
from cyfile import PyInfoHolder

for i in range(3):
	py_obj = PyInfoHolder()
	py_obj.say_hello()
	print("---done with this object---")
print("end")
this is val:  2
this is val:  4
this is val:  6
I'm destructed (val:2075964704)
hello
---done with this object---
this is val:  2
this is val:  4
this is val:  6
I'm destructed (val:2075964704)
hello
---done with this object---
this is val:  2
this is val:  4
this is val:  6
I'm destructed (val:2075964704)
hello
---done with this object---
end

Should I be worried about those outputs? Is it the same issue?

@da-woods
Copy link
Contributor

The output you show is without my PR, so it's missing the destruction of the MyObject in the cdef class. With my PR these are destructed correctly

If you dig into the C++ code it first creates an uninitialized MyObject temporary:

MyObject __pyx_t_1;

then for all the self.my_object = MyObject() assignments it just assigns from that temporary:

__pyx_v_self->my_object = __pyx_t_1;

The destructor calls you see is that temporary being destroyed at the end of __init__, with the arbitrary number being because it was uninitialized.

You could possibly argue that this is slightly odd code to generate. However, it's valid and there's no memory leaks or real bugs to fix here. I'm not sure a patch to treat structs and C++ classes completely identically would be practical, and it'd be hard not to break existing code (e.g. struct->dict conversion). I'll leave real decisions on this to the actual maintainers, but personally I'm not going to do anything more on this.

Remember that you can always just wrap MyObject as cdef cppclass MyObject if you want Cython to treat it as a C++ class. This'll work fine (without changing the C++ source).

@scoder scoder added this to the 3.0 milestone Sep 4, 2020
scoder pushed a commit that referenced this issue Sep 4, 2020
In C++ mode, structs can contain C++ classes. Therefore structs
should have their destructors called to ensure any class
contained is also destructed.

Also, a bit more thorough about ensuring constructor is generated where necessary.

Closes GH-3226.
sairam4123 added a commit to sairam4123/cython that referenced this issue Nov 4, 2020
* Fix test in Py2.

* Check for exceptions also when @returns() is used, not only for "->" return type annotations. (cythonGH-3664)

When you use Python type annotations, it would be weird if you lost Python exception propagation semantics along the way, just by compiling the code. So the default behaviour is "except? -1" here for C integer types.

Arguably, this would also be a better default for the decorator case.

See cython#3625 (comment)

* Add tests that exception propagation works with the "@returns()" decorator.

* Update changelog.

* Allow selecting specific doctests in test modules with "-k pattern", instead of always running all tests.

* Correct the positions reported for errors in f-strings.
Closes cython#3674

* Fix f-string error positions in nogil test.

* Clarify backwards incompatible special method change in changelog.

* Fix many indentation and whitespace issues throughout the code base (cythonGH-3673)

… and enforce them with pycodestyle.

* Allow selecting specific doctests in test modules with "-k pattern", instead of always running all tests.

* Prepare release of 0.29.20.

* Update changelog.

* Avoid calling PyUnicode_FromUnicode() in Py3.
Closes cython#3677

* Update changelog.

* Minor code simplification.

* Always bind Cython functions

* Update changelog.

* Avoid an "unused variable" warning for code that gets compiled out in Py3.

* Minor code simplification.

* Revert "Always bind Cython functions"

This reverts commit 6677326.

* Update changelog.

* Revert "Limited API updates and cleanup for cython#2056. cythonGH-3635)"

This reverts commit 02bb311.

* Revert "Invoke binop super method via direct slot access."

This reverts commit bcb9387.

* Revert "Add support for pow operator."

This reverts commit d849fb2.

* Revert "Python-style binary operation methods."

This reverts commit e6a8124.

* Re-add "c_api_binop_methods" directive for backwards compatibility after reverting cython#3633 and force it to "False".
Closes cython#3688.

* Change the default of the "c_api_binop_methods" directive to False. (cythonGH-3644)

This is a backwards incompatible change that enables Python semantics for special methods by default, while the directive allows users to opt out of it and go back to the previous C-ish Cython semantics.

See cython#2056

* Always bind Cython functions in Py3 (cythonGH-3683)

Rebased 1bb26b9 for cython 0.29.x,
and made conditional to Python 3.  This does not solve the original staticmethod problem
in Python 2 but it does resolve classmethod problems in Python 3.9.  Therefore, it unbreaks
other packages on Python 3.9, improves overall state for Python 3 and does not introduce
regressions in Python 2.

* Limit the scope of a local variable in a helper function.

* Avoid a C compiler warning about a constant condition.

* Remove dead code.

* Update changelog.

* Add support for C++ scoped enums with "enum class" and "enum struct" (cythonGH-3640)

Closes cython#1603.

* Add missing "PyUnicode_GET_LENGTH" to unicode.pxd (cythonGH-3692)

* Add missing "PyUnicode_GET_LENGTH" to unicode.pxd (cythonGH-3692)

* Fix usage of deprecated Py_UNICODE API.

* Add safety fix to avoid reading a character from the empty string.

* Fix prefix of internal function name.

* Fix definition of "__Pyx_CyFunction_USED", which should only be #defined and not have a value.

* Avoid using "tp_name" when CYTHON_COMPILING_IN_LIMITED_API (cythonGH-3693)

* exec() did not allow recent Python syntax features in Py3.8+ due to https://bugs.python.org/issue35975
Closes cython#3695

* Avoid using the "tp_iternext" slot when CYTHON_USE_TYPE_SLOTS is disabled.

* Give the "__Pyx_PyObject_GetIterNext" helper macro a more explanatory name.

* Avoid unused variable in PyPy etc.

* Avoid a call to PyTuple_GET_ITEM() to get the item array pointer if CYTHON_ASSUME_SAFE_MACROS is disabled.
See cython#3701

* Enable travis for all branches.

* Fix indentation counter for module init function.

* Fix FunctionState handling for module cleanup function.

* Validate that all temps were correctly released at the end of a function.

* Keep reference to module dict around also in limited mode (cythonGH-3707)

`PyModule_GetDict` is part of the limited API so we can keep a reference
to the module dict around regardless of limited mode being enabled or not.

* Validate and fix temp releasing (cythonGH-3708)

* Fix a temp leak in the type init code.
* Fix temp leaks in fused types initialisation code.
* Correctly release the buffer index temps allocated for index calculations.
* Make tests fails hard if a temp variable is not released at the end of a generated function.
* Fix temp leak in switch statement code.
* Make end-to-end tests fail on refnanny output.
* Fix result temp leak in PyTypeTestNode.
* Fix result temp leak in external type/function import code and enable the refnanny check for them.
* Fix temp leak when try-return-finally is used in generators.
* Make it explicit when an allocated temp is not meant to be reused.
* Fix temp leak when assigning to the real/imag attributes of complex numbers.
* Fix temp leak when assigning to a memoryview slice.
* Clean up "num_threads" result temp in parallel section, not only in prange loop.
* Fix temp leak in Pythran buffer setitem code.
* Simplify NumPyMethodCallNode since it does not need the Python function anymore. Previously, it generated code that needlessly looked up the Python function without actually using it.
* Fix temp leak when deleting C++ objects.
* Add a test that actually reusing temps when deleting C++ objects works correctly.

* Test runner: disable keep-alive output in --debug mode to keep a potential pdb console clean.

* Fix argument name usage in finally blocks (cythonGH-3713)

Fixes cython#3712 (hopefully) by reverting a small part of bbef4d7

* Fix argument name usage in finally blocks (cythonGH-3713)

Fixes cython#3712 (hopefully) by reverting a small part of bbef4d7

* Document version-tagged pxd files (cythonGH-3587)

* Implement generic optimized loop iterator with indexing and type inference for memoryviews (cythonGH-3617)

* Adds bytearray iteration since that was not previously optimised (because it allows changing length during iteration).
* Always set `entry.init` for memoryviewslice.

* Update changelog.

* Make end-to-end tests fail on refnanny output.

* Fix FunctionState handling for module cleanup function.

* Update change log.

* Really only use PyUnicode_FromUnicode() when needed (cythonGH-3697)

* Really only use PyUnicode_FromUnicode() for strings that contain lone surrogate, not for normal non-BMP strings and not for surrogate pairs on 16bit Unicode platforms.

See cython#3678

* Extend buildenv test to debug a MacOS problem.
* Add a test for surrogate pairs in Unicode strings.
* Limit PyUnicode_FromUnicode() usage to strings containing lone surrogates.
* Accept ambiguity of surrogate pairs in Unicode string literals when generated on 16bit Py2 systems.

* Disable testing against NumPy 1.19+ in the 0.29.x branch, which removed C-API features.

* Disable testing against NumPy 1.19.0 in the 0.29.x branch, which breaks a C-API call.

* Validate and fix temp releasing (cythonGH-3708) (cythonGH-3717)

* Validate and fix temp releasing (cythonGH-3708)
Backports 92147ba.

    * Fix a temp leak in the type init code.
    * Fix temp leaks in fused types initialisation code.
    * Correctly release the buffer index temps allocated for index calculations.
    * Make tests fails hard if a temp variable is not released at the end of a generated function.
    * Fix temp leak in switch statement code.
    * Make end-to-end tests fail on refnanny output.
    * Fix result temp leak in PyTypeTestNode.
    * Fix result temp leak in external type/function import code and enable the refnanny check for them.
    * Fix temp leak when try-return-finally is used in generators.
    * Make it explicit when an allocated temp is not meant to be reused.
    * Fix temp leak when assigning to the real/imag attributes of complex numbers.
    * Fix temp leak when assigning to a memoryview slice.
    * Clean up "num_threads" result temp in parallel section, not only in prange loop.
    * Fix temp leak in Pythran buffer setitem code.
    * Simplify NumPyMethodCallNode since it does not need the Python function anymore. Previously, it generated code that needlessly looked up the Python function without actually using it.
    * Fix temp leak when deleting C++ objects.
    * Add a test that actually reusing temps when deleting C++ objects works correctly.

* Improve syntax feature support of Cython CodeWriter (cythonGH-3514)

* Disable Py_UNICODE fallback for __Pyx_UnicodeContainsUCS4() in Py3.9 since Py_UNICODE is deprecated and PEP-393 unicode is practically required.

* Avoid calling PyUnicode_FromUnicode() in Py3 (cythonGH-3721)

See cython#3677

* Add missing name substitution in __Pyx_PyUnicode_Substring() utility code.

* Update changelog.

* Prepare release of 0.29.21.

* Really only use PyUnicode_FromUnicode() when needed (cythonGH-3697)

* Really only use PyUnicode_FromUnicode() for strings that contain lone surrogate, not for normal non-BMP strings and not for surrogate pairs on 16bit Unicode platforms.

See cython#3678

* Extend buildenv test to debug a MacOS problem.
* Add a test for surrogate pairs in Unicode strings.
* Limit PyUnicode_FromUnicode() usage to strings containing lone surrogates.
* Accept ambiguity of surrogate pairs in Unicode string literals when generated on 16bit Py2 systems.

* Disable Py_UNICODE fallback for __Pyx_UnicodeContainsUCS4() in Py3.9 since Py_UNICODE is deprecated and PEP-393 unicode is practically required.

* Fix test in 16-bit Unicode deployments.

* Update changelog.

* Always consider 0-sized arrays as C- and F-contiguous (cythonGH-3728)

Fixes cython#2093

* Always consider 0-sized arrays as C- and F-contiguous (cythonGH-3728)

Fixes cython#2093

* Update changelog.

* Add missing unordered_map template defaults (cythonGH-3686)

* Add missing unordered_map template defaults (cythonGH-3686)

* Update changelog.

* Improve test output in case of failures.

* Using Py_UNICODE to store lone surrogates makes Py3 join surrogate pairs on 16-bit Unicode platforms (Windows) when reading them back in, although we correctly processed them before.
Instead, we now use the "unicode_escape" codec to store byte strings, because it can return surrogate characters (which the other codecs cannot).

* Disable test in Py2.6.

* Update changelog.

* Use the more appropriate CYTHON_USE_TYPE_SLOTS guard for accessing the binop number slot.

* Fix typo in error message.

* Update the documentation on the arithmetic special methods and issue a "backwards compatibility" warning when the reversed method is not implemented.

See cython#2056

* Add the new "c_api_binop_methods" directive to the documentation.

* PyPy does not support PyType_GetSlot(). Use type slots instead.

* Reformat doc paragraph.

* Document C inline properties.

* Beautify example output.

* Use inline properties on the "PyComplex" builtin type declared in "cpython.complex" to provide C level access to the "real" and "imag" attributes (which Cython provides anyway for the 'undeclared' builtin type).

* Prevent compile error when the result of repr() is assigned to a "unicode" LHS with language_level=3.
Closes cython#3736

* Handle `for x in cpp_function_call()` (cythonGH-3667)

Fixes cython#3663

This ensures that rvalues here are saved as temps, while keeping the
existing behaviour for `for x in deref(vec)`, where the pointer for vec
is copied, meaning it doesn't crash if vec is reassigned.

The bit of this change liable to have the biggest effect is that I've
changed the result type of dereference(x) and x[0] (where x is a c++
type) to a reference rather than value type. I think this is OK because
it matches what C++ does. If that isn't a sensible change then I can
probably inspect the loop sequence more closely to try to detect this.

* Clarify changelog entry on ways to deal with the incompatible binop method change.

* Remove Google Analytics script from documentation to avoid tracking our users.

* Add donation banner to documentation.

* Create bug template

* Update issue templates

* Delete unused custom ticket template

* Add handshake emoji to donations banner to make it more visible.

* Don't create CReference in C (only c++) (cythonGH-3746)

* Fixed reference types being passed to getitemint (cythonGH-3755)

* Reorder test module to restore the "invalid - valid" order.

* Update changelog.

* Prepare release of 3.0a6.

* Add test comments on how "memslice.pyx" and "memoryview.pyx" relate.

* Print test dependency versions to help with test failure debugging.

* Add CI builds for different CPU architectures on travis.

* Update issue templates

* Add type "Py_hash_t" in pure Python mode.

* Update changelog.

* Clarify changelog entry.

* Remove dead code and dead comments from "numpy/__init__.pxd".

* Change "Py_intptr_t" declaration in numpy.pxd to what CPython uses as fallback. "int" seems overly lazy if it tends to be larger on many systems.

* Readability improvements in "numpy.pxd".

* Add a comment that numpy.pxd is maintained by the NumPy project.

* Try to fix NumPy test failures by not setting the "NPY_NO_DEPRECATED_API" #define for NumPy 1.19.[01].

* Use NumPy 1.18.x for testing on travis as long as 1.19.[01+] ships a numpy.pxd that is incompatible with Cython 3.0.

* Remove useless "extern" modifiers from cdef classes declared in "extern" sections.

* Do not depend on the default type inference in "cpython/array.pxd".

* Avoid unused C variable warning by moving declaration and usage under the same condition.
Closes cython#3763

* Use a generic shape accessor for pythranized array (cythonGH-3762)

This is a follow up to cython#3307

* Silence a C compiler warning about a constant condition.

* Remove an unused function that had been added for the now-deleted "__getbuffer__" implementation.

* Support PEP-560 ("__class_getitem__") for extension classes (cythonGH-3765)

* Add "make checks" target to run code checks.

* In bug template, ask for Python version in addition to Cython version

* Support simple, non-strided views of "cython.array".
Closes cython#3775

* Remove unused cimports.

* Update changelog.

* Set PYTHONHOME in embedding test to fix compilation issues in Py3.8/macOS.

* Fix unrelated test after changing MemoryView.pyx.

* Revert "Set PYTHONHOME in embedding test to fix compilation issues in Py3.8/macOS."

This reverts commit a333d6a.

The change did not resolve the test issue in travis.

* Fix Python 3.4 + MSVC issue with elaborated type specifier for enum (cythonGH-3782)

* Add more cimport_from_pyx tests (cythonGH-3786)

There's now a fairly wide range of valid syntax for declaring things in pyx files and it should all be supported when cimporting from them.

* Split known types into separate lines to let them have their own VCS history.

* Add a note on PayPal fees for small payments.

* Avoid merged-in code picking up directives from main module (cythonGH-3785)

Fixes cython#1071

* Fix cygdb (cythonGH-3542)

* Cython debugger documentation: Added link to an installation script.
* Got a new libpython.py from the cpython source distribution.
* Default language level in tests is now 3 instead of 2
* Migrated codefile from python 2 to python 3.
* Added testcase for the cy list command in cygdb.
* Temporarily removing test case that freezes gdb.
* Fixed a bug that broke several Cygdb tests.

The cython_debug/cython_debug_info_* files map the names of the C-functions generated by the Cython compiler to the names of the functions in the *.pyx source. If the function was defined using "def" (and not "cpdef" or "cdef") in the *.pyx source file, the C-function named in cython_debug/cython_debug_info_* used to be __pyx_pw_*, which is the name of the wrapper function and now it is __pyx_f_*, which is the name of the actual function. This makes some Cygdb tests pass that did not pass before.

* Better error messages: If a cygdb command raises, a traceback will be printed.
* Fixed a bug in cygdb.

The following now works:
1. Start cygdb
2. Type "cy exec" and hit enter
3. Type some other lines
4. Type "end" and hit enter.
-> These "other lines" will get executed

* Fixed a bug in cygdb: cy list now works outside of functions.
* Added print_hr_allmarkers function for easier debugging.
* Fixed a bug that broke cygdb:

cy break did not work if you put the breakpoint outside of a function if there was e.g. the following somewhere in your *.pyx file:
cdef class SomeClass():
    pass

* Added a Cygdb test for printing global variables.
* Fixing cygdb: Replaced cy print with a simple, working solution.
* If an exception in Cygdb occurs, a stacktrace will be printed.
* Fixed a bug that broke cy break -p
* Bugfix: The compiler now writes out correctly which cython linenumber and path corresponds to which c linenumber.
* Set language_level=2 in runtests.py

* Update changelog.

* Add a "gdb" test tag that depends on being able to run gdb.

* Update CPython "test_fstring" copy to Py3.9.

* Improved documentation for annotation typing (cythonGH-3799)

Mainly by moving it to a separate section to make it easier
to find, however also added a small amount of extra information
about some of the obvious limitations.

* Fix `return None` in CodeWriter. (cythonGH-3795)

* Rename "GCC_DIAGNOSTIC" macro to make it Cython specific and make it available to other utility code sections.

* Add a test for unpacking large PyLong values.

* Silence gcc diagnostics whenever -1 is cast to something user provided. (cythonGH-3803)

Fixes cythonGH-2749.

* Define extern `PyBUF_MAX_NDIM` (cythonGH-3811)

Ensure that Cython exposes `PyBUF_MAX_NDIM` from Python as part of
`cpython.buffer` to allow access to developers.

* Change test to use only integer calculations to prevent platform specific rounding issues, while keeping a reasonable distribution of integers across the PyLong digit ranges.

* Created a glossary and added one new entry (cythonGH-3810)

* Call destructors for structs in C++ (cythonGH-3228)

In C++ mode, structs can contain C++ classes. Therefore structs
should have their destructors called to ensure any class
contained is also destructed.

Also, a bit more thorough about ensuring constructor is generated where necessary.

Closes cythonGH-3226.

* Rename "GCC_DIAGNOSTIC" macro to make it Cython specific and make it available to other utility code sections.

* Silence gcc diagnostics whenever -1 is cast to something user provided. (cythonGH-3803)

Fixes cythonGH-2749.

* Allow cast to ctuple (cythonGH-3812)

* Make asyncio.iscoroutinefunction() recognise Cython compiled coroutines. (cythonGH-3427)

Python's asyncio.coroutines uses an object to tag objects as coroutine functions. We now read this object and use it to tag Cython compiled coroutines as well.

It also includes tests to make sure `asyncio.iscoroutinefunction()` works as expected.
This doesn't fix `inspect.iscouroutinefunction()` (which uses a flag that can trigger undesirable behaviour for cython functions).

Closes cython#2273

* Do not cover lines that were excluded in the coveragerc config file (cythonGH-3682)

Closes cython#3680.

* Add doc support for cpdef enum (cythonGH-3813)

* Update changelog.

* Add "check_size ignore" to all NumPy.pxd class declarations to silence the useless size warnings about them.
See numpy/numpy#432 (comment)
Also remove the useless "extern" modifiers from cdef classes declared in "extern" sections.

* Update changelog.

* Update changelog.

* Support utility code in headers (cythonGH-3779)

Specifically this allows public packed structs but may also allow other public declarations that require small amounts of utility code.

* Prevent overflowing hash values for "Machines.Node" due to arbitrarily large unsigned pointer values.
Closes cython#3840

* Extract "error_type" handling from the type creation functions.

* Remember in the AnnotationsWriter when a serialised expression contained unknown/unsupported nodes.

* Do not include "u" string prefix in annotations since tools that process them probably expect Py3 string semantics anyway.

* Keep AnnotationNode instead of the bare annotation expression in "entry.annotation" to get access to the string representation.

* Fix test after removing the 'u' prefix from unicode string annotation values.

* Add support for PEP 526 `__annotations__` in class body. (cythonGH-3829)

Closes cython#2552

* Some more glossary entries (cythonGH-3836)

* Rename test file to avoid ambiguity with the C "const" modifier.

* Join '*' and '**' parsing in declarators to avoid differences for 'const' parsing etc.

* Set language level in Cython's speed-up .pxd files since it no longer depends on the .py files that they correspond to.

* Declare "scanner.sy" as "unicode" string to optimise its usage.

* Use more recent C-API functions on tear-down of the embedding code.

* Improve some wordings in README.rst (cythonGH-3852)

* Restore Py2 compatibility in test.

* Replace useless comment.

* Fixed assorted crashes with fused types (cythonGH-3853)

Show a more detailed error for unspecializable fused return types.

* Add a more common and more versatile example to the Verbatim C-Code documentation.

* Fix RST typo.

* Make C code C89 again.

* Add support for forwarding references (cythonGH-3821)

See, for example, https://isocpp.org/blog/2012/11/universal-references-in-c11-scott-meyers

Closes cython#3814

* Avoid C compiler warnings about unused variables in test.

* Avoid C compiler warning about unused variable in test.

* Test Py3.7 and Py3.8 in C and C++ mode under appveyor.
Run both in the same job since the machines are quite fast with parallel processes, but setting one up is slow.

* Fix test compile failure in MSVC.

* Allow creation of wrappers for cdef functions with memoryviews (cythonGH-3856)

Fixes cython#3843

* Clean up and test type identifier escaping.
- hash() hashing lead to unpredictable random prefixes for long names across multiple runs
- use a single regex run instead of repeated calls to replace()

* Move a memoryview test out of the "run" test directory since memoryview tests tend to be slow and have their own test directory.

* Fix test compile failure in MSVC.

* Clarify the section on exception return values.

* Be a bit more paranoid about macro usage in the refnanny code.

* Looks like the usual macro on Windows is "_WIN32" and not "WIN32". Let's support both, to be on the safe side.

* Try to fix platform specific test once more.
"synchapi.h" is not supposed to be included directly, and "windows.h" can break ... other stuff.
Let's see what we can do.

* Fix ReST typo.

* docs: fix typos, minor clarification.

* Minor docs clarification on error return values. (cythonGH-3859)

* Replace deprecated Element.getiterator() with Element.iter(). (cythonGH-3864)

`xml.etree.ElementTree.Element.getiterator()` was deprecated in Python
2.7 & 3.2 and removed in the freshly released Python 3.9.

* Prefer Element.iter() over the deprecated Element.getiterator(). (cythonGH-3865)

`xml.etree.ElementTree.Element.getiterator()` was deprecated in Python
2.7 & 3.2 and removed in Python 3.9.

* Split a combined assert in two to avoid an unused C temp variable when assertions are disabled. (cythonGH-3870)

* Split a combined assert in two to avoid an unused C temp variable when assertions are disabled. (cythonGH-3870)

* Disable a test that fails in Py2 in 0.29.x since 'str' is 'unicode' in cython3, also in Python 2.

* Rename test file to avoid ambiguity with the C "const" modifier.

* Use more recent C-API functions on tear-down of the embedding code.

* Make type identifier escaping deterministic: hash() hashing lead to unpredictable random prefixes for long names across multiple Python runs.

* Clarify the section on exception return values.

* Fix ReST typo.

* docs: fix typos, minor clarification.

* Minor docs clarification on error return values. (cythonGH-3859)

* Add Python 3.9 to appveyor build.

* Resolve merge conflict.

* Removed `same_as` methods from Plex.Actions.Action (cythonGH-3847)

It seems to be unused and it looked like the existing implementation was faulty.

* Fix test after changing the hash method used for escaping long type descriptions.

* Add warning for common user mistake and fix crash relating to annotated attributes. (cythonGH-3850)

Closes cython#3830

* Improve error reporting when users mistakenly write "&&" or "||" instead of Python's "and" and "or" operators. (cythonGH-3858)

* Disable the embedding test on MacOS-X to get the builds green again. Too many PRs depend on the travis tests to give a green light.

* Simplify the output stream capturing for the C compiler runs by using a temp file instead of threads.

* Help reporters see the first paragraph of the bug report template.

* Detect _MSC_VER for __PYX_STD_MOVE_IF_SUPPORTED to support MSVC support even when "__cplusplus" is not set appropriately (cythonGH-3792)

* Fix memoryview casts involving fused types (cythonGH-3882)

I think this approach is more satisfactory than the old way it used to "work", where "fused_to_specific" was permanently added to the module scope containing the fused type (in this case the Cython scope), was applied in "Scope.lookup_type", but continued to have an effect on the scope forever.

Closes cython#3881

* Add O_DIRECT to posix/fcntl.pxd (cythonGH-3894)

Closes cython#3242

Co-authored-by: Stefan Behnel <stefan_ml@behnel.de>
Co-authored-by: Jeroen Demeyer <jeroen.k.demeyer@gmail.com>
Co-authored-by: Robert Bradshaw <robertwb@gmail.com>
Co-authored-by: Michał Górny <mgorny@gentoo.org>
Co-authored-by: Ashwin Srinath <3190405+shwina@users.noreply.github.com>
Co-authored-by: Thomas A Caswell <tcaswell@gmail.com>
Co-authored-by: Matthias Braun <matze@braunis.de>
Co-authored-by: da-woods <dw-git@d-woods.co.uk>
Co-authored-by: Tao He <sighingnow@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Clemens <chofreither@gmail.com>
Co-authored-by: will <timwilloney@gmail.com>
Co-authored-by: serge-sans-paille <serge.guelton@telecom-bretagne.eu>
Co-authored-by: Yuan <pyslvs@gmail.com>
Co-authored-by: Volker-Weissmann <39418860+Volker-Weissmann@users.noreply.github.com>
Co-authored-by: Tao He <linzhu.ht@alibaba-inc.com>
Co-authored-by: cf-natali <cf.natali@gmail.com>
Co-authored-by: jakirkham <jakirkham@gmail.com>
Co-authored-by: Tasha "Ren" Chin <tasha.s.chin@gmail.com>
Co-authored-by: Pedro Marques da Luz <pmdaluz@gmail.com>
Co-authored-by: Mathias Laurin <Mathias.Laurin+github.com@gmail.com>
Co-authored-by: matham <moiein2000@gmail.com>
Co-authored-by: Sean <stewart.d.sean@gmail.com>
Co-authored-by: Rajat Dash <49777622+DARKPROGRAMMER16@users.noreply.github.com>
Co-authored-by: ptype <ptype@users.noreply.github.com>
Co-authored-by: Nick Pope <nick.pope@flightdataservices.com>
Co-authored-by: Jeppe Dakin <jeppe_dakin@hotmail.com>
Co-authored-by: Zackery Spytz <zspytz@gmail.com>
@charlesbluca
Copy link

Would it be possible to backport this fix to 0.29.x? We are running into this issue in rapidsai/cudf#8935 and are currently restricted to cython >=0.29,<0.30

@da-woods
Copy link
Contributor

da-woods commented Aug 3, 2021

Would it be possible to backport this fix to 0.29.x? We are running into this issue in rapidsai/cudf#8935 and are currently restricted to cython >=0.29,<0.30

It's probably very easily worked around - just declare the struct type as a cppclass instead. It really doesn't matter to Cython whether the definition in C++ is a struct or a class.

The only thing you lose is the auto-conversation to dicts

rapids-bot bot pushed a commit to rapidsai/cudf that referenced this issue Aug 3, 2021
Closes #8935

Defines `packed_columns` in Cython as a `cppclass` instead of a `struct`, so that it is actually destroyed when a containing `PackedColumns` object is no longer referenced; see cython/cython#3226.

Authors:
  - Charles Blackmon-Luca (https://github.com/charlesbluca)

Approvers:
  - https://github.com/brandon-b-miller

URL: #8936
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants