Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unicode/Bytes concatenation is inefficient #3453

Closed
da-woods opened this issue Mar 22, 2020 · 1 comment
Closed

Unicode/Bytes concatenation is inefficient #3453

da-woods opened this issue Mar 22, 2020 · 1 comment

Comments

@da-woods
Copy link
Contributor

da-woods commented Mar 22, 2020

CPython has a specific optimization when concatenating strings - it checks the reference count of the first operand and tries to concatenate in place if possible. This is done in ceval: https://github.com/python/cpython/blob/309d7cc5df4e2bf3086c49eb2b1b56b929554500/Python/ceval.c#L5354. For some specific cases this can make a big performance difference https://stackoverflow.com/questions/35787022/cython-string-concatenation-is-super-slow-what-else-does-it-do-poorly

I had an initial go at it here: #3451. However there's definite failure paths since it can NULL out variables that Cython isn't expecting to be NULL.

A couple of possible options:

  1. It might be possible to create something with that basically re-implements PyUnicode_Append but without clearing operand1. (i.e. remove this line https://github.com/python/cpython/blob/b146568dfcbcd7409c724f8917e4f77433dd56e4/Objects/unicodeobject.c#L11517)

  2. (probably easier) ensure that operand1 is always set to something on exit, even if it's a dummy value like an empty string. This could mostly be based on the current PR, but it would ocassionally lead to unexpected behaviour (mostly when exceptions are caught and handled)

      cdef unicode val = "X"
      try:
           val += "x"
      except:
           pass
      return val  # wouldn't crash, but would be an odd placeholder string.
    
@da-woods
Copy link
Contributor Author

da-woods commented Apr 2, 2020

Closed in #3451

@da-woods da-woods closed this as completed Apr 2, 2020
@scoder scoder added this to the 3.0 milestone Apr 2, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants