[BUG] unicode.split does not allow to pass None for sep

[navytux]

Describe the bug
I'm hitting the difference in behaviour in between CPython and Cython for unicode.split - with Cython passing sep=None explicitly raises TypeError. Please find details below:

To Reproduce
Code to reproduce the behaviour:

---- 8< ---- usplit.pyx

# cython: language_level=3

def mysplit(q):
    return unicode.split(q, None)

print(mysplit("hello world"))

Expected behavior

I expect it to behave the same as in Python - i.e. print ['hello', 'world']:

---- 8< ---- usplit_py.py

def mysplit(q):
    return str.split(q, None)

print(mysplit("hello world"))

$ python usplit_py.py 
['hello', 'world']

However what I get instead is the following exception that None could not be used for sep:

$ cythonize -i usplit.pyx 
Compiling /home/kirr/usplit.pyx because it changed.
[1/1] Cythonizing /home/kirr/usplit.pyx
running build_ext
building 'usplit' extension
creating /home/kirr/tmp3kckc5wa/home
creating /home/kirr/tmp3kckc5wa/home/kirr
x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kirr/src/wendelin/venv/py3.venv/include -I/usr/include/python3.9 -c /home/kirr/usplit.c -o /home/kirr/tmp3kckc5wa/home/kirr/usplit.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fwrapv -O2 -Wl,-z,relro -g -fwrapv -O2 -g -ffile-prefix-map=/build/python3.9-RNBry6/python3.9-3.9.2=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 /home/kirr/tmp3kckc5wa/home/kirr/usplit.o -o /home/kirr/usplit.cpython-39-x86_64-linux-gnu.so

$ python -c 'import usplit'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "usplit.pyx", line 6, in init usplit
    print(mysplit("hello world"))
  File "usplit.pyx", line 4, in usplit.mysplit
    return unicode.split(q, None)
TypeError: must be str, not NoneType

Environment (please complete the following information):

• OS: [Debian GNU/Linux 11]
• Python version [e.g. 3.9.2]
• Cython version [e.g. 0.29.27]

Thanks beforehand,
Kirill

[da-woods]

So it looks like we translate unicode.split to PyUnicode_split

cython/Cython/Compiler/Optimize.py

Lines 3614 to 3635 in d2d2ea3

	def _handle_simple_method_unicode_split(self, node, function, args, is_unbound_method):
	"""Replace unicode.split(...) by a direct call to the
	corresponding C-API function.
	"""
	if len(args) not in (1,2,3):
	self._error_wrong_arg_count('unicode.split', node, args, "1-3")
	return node
	if len(args) < 2:
	args.append(ExprNodes.NullNode(node.pos))
	self._inject_int_default_argument(
	node, args, 2, PyrexTypes.c_py_ssize_t_type, "-1")
	return self._substitute_method_call(
	node, function,
	"PyUnicode_Split", self.PyUnicode_Split_func_type,
	'split', is_unbound_method, args)
	PyUnicode_Join_func_type = PyrexTypes.CFuncType(
	Builtin.unicode_type, [
	PyrexTypes.CFuncTypeArg("str", Builtin.unicode_type, None),
	PyrexTypes.CFuncTypeArg("seq", PyrexTypes.py_object_type, None),
	])

The documentation for that (https://docs.python.org/3/c-api/unicode.html#c.PyUnicode_Split) implies that it treats sep==NULL as "split on whitespace". So we probably just want a tiny wrapper function to translate None to NULL and maybe also to special-case it when a None constant is passed.

[navytux]

Yes, @da-woods, thanks. I'm already doing my workarounds for other unicode methods - count, endswith, find etc for start and stop, e.g. as follows:

    def count(self, sub, start=None, end=None):                                                            
        # cython optimizes unicode.count to directly call PyUnicode_Count - cannot use None for start/stop        
        if start is None: start = 0                                                                        
        if end   is None: end   = PY_SSIZE_T_MAX                                                           
        return unicode.count(self, pyu(sub), start, end)                                                   

but for unicode.split I could not find a clean way to pass sep as NULL myself.

[da-woods]

If you have a list of the unicode methods that you know of that have similar bugs then that'd be very useful?

[navytux]

Sure, here is my current list:

• count
• endswith
• find
• index
• rfind
• rindex
• split (this one)
• startswith

[da-woods]

Your best workaround in the short-term would be to stop Cython from knowing that the object is unicode. So

def call_split(u: unicode):
   return (<object>u).split(None)

should at least prevent Cython from doing its optimization

[navytux]

@da-woods, thanks for the hint. For the reference I already worked it around as follows:

    def split(self, sep=None, maxsplit=-1):
        # cython optimizes unicode.split to directly call PyUnicode_Split - cannot use None for sep
        # and cannot also use object=NULL https://github.com/cython/cython/issues/4737
        if sep is None:
            v = unicode.split(self, maxsplit=maxsplit)
        else:
            v = unicode.split(self, pyu(sep), maxsplit)
        return list([pyu(_) for _ in v])

[scoder]

Agreed that Cython should handle this internally. Thanks for bringing it up.

[navytux]

You are welcome, @scoder. Thanks for feedback.

[scoder]

I looked into this. @navytux, apart from split(), the other methods you mentioned do not seem to accept None as argument except for the index range. That's what you meant, right? Supporting that is a bit tricky, because it involves two different coercions: 1) None -> 0 / PY_SSIZE_T_MAX and 2) object -> Py_ssize_t. I'll see what I can do.

[navytux]

@scoder, thanks for looking into it. Yes, for everything besides split I was talking about start/end parameters that specify interested index range.

[scoder]

I submitted a PR in #4740

@navytux, since there are a bunch of methods that are affected, maybe you could help us by writing up the tests for them? I added a few and it looks good, but most methods are still missing, and I think it'd be good to also test the None arguments separately and not just together as two Nones.