Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] invalid tag name with iterators in --gdb mode #5552

Closed
tornaria opened this issue Jul 21, 2023 · 5 comments
Closed

[BUG] invalid tag name with iterators in --gdb mode #5552

tornaria opened this issue Jul 21, 2023 · 5 comments
Assignees

Comments

@tornaria
Copy link
Contributor

Describe the bug

$ cat example.pyx 
def use_an_iterator(iterator):
    return (x for x in iterator)
$ cython3 --version
Cython version 3.0.0
$ cython3 -3 example.pyx 
$ cython3 -3 --gdb example.pyx 

Error compiling Cython file:
------------------------------------------------------------
...
def use_an_iterator(iterator):
^
------------------------------------------------------------

example.pyx:1:0: Compiler crash in DebugTransform


Compiler crash traceback from this point on:
  File "Cython/Compiler/Visitor.py", line 182, in Cython.Compiler.Visitor.TreeVisitor._visit
  File "/usr/lib/python3.11/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 4033, in visit_ModuleNode
    self.visit_FuncDefNode(nested_funcdef)
  File "/usr/lib/python3.11/site-packages/Cython/Compiler/ParseTreeTransforms.py", line 4093, in visit_FuncDefNode
    self.tb.start(arg.name)
  File "/usr/lib/python3.11/site-packages/Cython/Debugger/DebugWriter.py", line 42, in start
    self.tb.start(name, attrs or {})
  File "src/lxml/saxparser.pxi", line 841, in lxml.etree.TreeBuilder.start
  File "src/lxml/saxparser.pxi", line 769, in lxml.etree.TreeBuilder._handleSaxStart
  File "src/lxml/apihelpers.pxi", line 179, in lxml.etree._makeSubElement
  File "src/lxml/apihelpers.pxi", line 1754, in lxml.etree._tagValidOrRaise
ValueError: Invalid tag name '.0'

Code to reproduce the behaviour:

def use_an_iterator(iterator):
    return (x for x in iterator)

Expected behaviour

No response

OS

linux

Python version

3.11.4

Cython version

No response

Additional context

This happens only if python lxml package is installed.

I reduced it to the simple function above, but this happens a lot in sagemath source code.

@da-woods
Copy link
Contributor

So the origin of this is: I've used function arguments with names like .0 and .1 as a way of passing arguments into generator expressions. This is by analogy with Python where

>>> [locals() for _ in range(1)]
[{'.0': <range_iterator object at 0x7f9a4732c150>, '_': 0}]

(exact behaviour likely somewhat version dependent).

It looks like these don't play well with the debug output writer.

It's probably fairly simple to fix either by renaming the arguments or by excluding them from the debug output. I don't know much about the --gdb so I don't really know what the appropriate solution is. But one of those solutions should work.

@h-vetinari
Copy link
Contributor

This issue is also affecting pyarrow, FWIW

@scoder
Copy link
Contributor

scoder commented Aug 30, 2023

We defined the XML format ourselves, I think, but changing it now brings in compatibility problems.

I think excluding these arguments is probably ok for now.

@oleksandr-pavlyk
Copy link
Contributor

I would like to look into fixing this issue, as it blocks building debug builds for certain projects based on scikit-build which passes --gdb argument to cython automatically.

@da-woods
Copy link
Contributor

da-woods commented Sep 8, 2023

@oleksandr-pavlyk I've assigned it to you. Hopefully it should be reasonably simple, but less us know how you get on.

@scoder scoder closed this as completed in 72172c6 Sep 14, 2023
AlenkaF pushed a commit to apache/arrow that referenced this issue Sep 21, 2023
### Rationale for this change

Cython 3.0.0 is the latest release. PyArrow should work with Cython 3.0.0. **Cython 3 is not enabled in this diff.**

### What changes are included in this PR?

* Don't use `vector[XXX]&&`
* Add a declaration for `postincrement`
  * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#c-postincrement-postdecrement-operator
* Ignore `C4551` warning (function call missing argument list) with MSVC
  * See also: cython/cython#4445
* Add missing `const` to `CLocation`'s static methods.
* Don't use `StopIteration` to stop generator
  * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-3-syntax-semantics
* non-extern `cdef` functions will now propagate python exceptions automatically unless explicitly labeled `noexcept`
* Function binding in cython is now enabled by default. Class methods that are used as wrappers for pickling should be converted to staticmethods.
* Numpydocs now validates more Cython 3 objects than Cython <3
  * Enum types are now being validated, and some unhelpful validation checks on Enums are now ignored
* Added a cython <3 nightly CI job

Note:
* Cython 3.0.0, 3.0.1, 3.0.2 has an issue when compiling with debug mode cython/cython#5552

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.

* Closes: #36730

Lead-authored-by: Dane Pitkin <dane@voltrondata.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
etseidl pushed a commit to etseidl/arrow that referenced this issue Sep 28, 2023
### Rationale for this change

Cython 3.0.0 is the latest release. PyArrow should work with Cython 3.0.0. **Cython 3 is not enabled in this diff.**

### What changes are included in this PR?

* Don't use `vector[XXX]&&`
* Add a declaration for `postincrement`
  * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#c-postincrement-postdecrement-operator
* Ignore `C4551` warning (function call missing argument list) with MSVC
  * See also: cython/cython#4445
* Add missing `const` to `CLocation`'s static methods.
* Don't use `StopIteration` to stop generator
  * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-3-syntax-semantics
* non-extern `cdef` functions will now propagate python exceptions automatically unless explicitly labeled `noexcept`
* Function binding in cython is now enabled by default. Class methods that are used as wrappers for pickling should be converted to staticmethods.
* Numpydocs now validates more Cython 3 objects than Cython <3
  * Enum types are now being validated, and some unhelpful validation checks on Enums are now ignored
* Added a cython <3 nightly CI job

Note:
* Cython 3.0.0, 3.0.1, 3.0.2 has an issue when compiling with debug mode cython/cython#5552

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.

* Closes: apache#36730

Lead-authored-by: Dane Pitkin <dane@voltrondata.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
dimpase referenced this issue in sagemath/sage Oct 10, 2023
JerAguilon pushed a commit to JerAguilon/arrow that referenced this issue Oct 23, 2023
### Rationale for this change

Cython 3.0.0 is the latest release. PyArrow should work with Cython 3.0.0. **Cython 3 is not enabled in this diff.**

### What changes are included in this PR?

* Don't use `vector[XXX]&&`
* Add a declaration for `postincrement`
  * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#c-postincrement-postdecrement-operator
* Ignore `C4551` warning (function call missing argument list) with MSVC
  * See also: cython/cython#4445
* Add missing `const` to `CLocation`'s static methods.
* Don't use `StopIteration` to stop generator
  * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-3-syntax-semantics
* non-extern `cdef` functions will now propagate python exceptions automatically unless explicitly labeled `noexcept`
* Function binding in cython is now enabled by default. Class methods that are used as wrappers for pickling should be converted to staticmethods.
* Numpydocs now validates more Cython 3 objects than Cython <3
  * Enum types are now being validated, and some unhelpful validation checks on Enums are now ignored
* Added a cython <3 nightly CI job

Note:
* Cython 3.0.0, 3.0.1, 3.0.2 has an issue when compiling with debug mode cython/cython#5552

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.

* Closes: apache#36730

Lead-authored-by: Dane Pitkin <dane@voltrondata.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
loicalleyne pushed a commit to loicalleyne/arrow that referenced this issue Nov 13, 2023
### Rationale for this change

Cython 3.0.0 is the latest release. PyArrow should work with Cython 3.0.0. **Cython 3 is not enabled in this diff.**

### What changes are included in this PR?

* Don't use `vector[XXX]&&`
* Add a declaration for `postincrement`
  * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#c-postincrement-postdecrement-operator
* Ignore `C4551` warning (function call missing argument list) with MSVC
  * See also: cython/cython#4445
* Add missing `const` to `CLocation`'s static methods.
* Don't use `StopIteration` to stop generator
  * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-3-syntax-semantics
* non-extern `cdef` functions will now propagate python exceptions automatically unless explicitly labeled `noexcept`
* Function binding in cython is now enabled by default. Class methods that are used as wrappers for pickling should be converted to staticmethods.
* Numpydocs now validates more Cython 3 objects than Cython <3
  * Enum types are now being validated, and some unhelpful validation checks on Enums are now ignored
* Added a cython <3 nightly CI job

Note:
* Cython 3.0.0, 3.0.1, 3.0.2 has an issue when compiling with debug mode cython/cython#5552

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.

* Closes: apache#36730

Lead-authored-by: Dane Pitkin <dane@voltrondata.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
### Rationale for this change

Cython 3.0.0 is the latest release. PyArrow should work with Cython 3.0.0. **Cython 3 is not enabled in this diff.**

### What changes are included in this PR?

* Don't use `vector[XXX]&&`
* Add a declaration for `postincrement`
  * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#c-postincrement-postdecrement-operator
* Ignore `C4551` warning (function call missing argument list) with MSVC
  * See also: cython/cython#4445
* Add missing `const` to `CLocation`'s static methods.
* Don't use `StopIteration` to stop generator
  * See also: https://cython.readthedocs.io/en/latest/src/userguide/migrating_to_cy30.html#python-3-syntax-semantics
* non-extern `cdef` functions will now propagate python exceptions automatically unless explicitly labeled `noexcept`
* Function binding in cython is now enabled by default. Class methods that are used as wrappers for pickling should be converted to staticmethods.
* Numpydocs now validates more Cython 3 objects than Cython <3
  * Enum types are now being validated, and some unhelpful validation checks on Enums are now ignored
* Added a cython <3 nightly CI job

Note:
* Cython 3.0.0, 3.0.1, 3.0.2 has an issue when compiling with debug mode cython/cython#5552

### Are these changes tested?

Yes.

### Are there any user-facing changes?

Yes.

* Closes: apache#36730

Lead-authored-by: Dane Pitkin <dane@voltrondata.com>
Co-authored-by: Sutou Kouhei <kou@clear-code.com>
Co-authored-by: Dane Pitkin <48041712+danepitkin@users.noreply.github.com>
Signed-off-by: AlenkaF <frim.alenka@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants