Support for str.split and str.join

[seibert]

As noted in #3674, support for str.split and str.join is pretty straightforward.

Remaining todos:

• enable support for maxsplit argument of str.split with default value of -1 (not currently possible with @overload_method, so we should fix that first)
• improve performance of split
• fast implementation of str.join just for lists (rather than generic iterables) that makes two passes over the list to preallocate the full size of the resulting string
• fast path string copy to use memcpy when character widths match
• figure out remaining performance spread between Numba and CPython
• update docs

[seibert]

OK, so currently join seems to be 40% slower than CPython, which is probably within the accuracy of my ability to benchmark things. split is still 6x slower, which is a very large improvement from where we started, but still concerning. The split loop does need to construct new strings and append to lists, which are places where Numba's slower atomic reference counting can hurt performance, but it isn't clear that explains all of the difference.

[seibert]

Basically, I'm trying to understand if the statement parts.append(a[last:idx]) is resulting in code that is very suboptimal.

Update: The speed issue seems to be mostly due to the performance of constructing a string slice.

[seibert]

After some additional experimentation and differential benchmarking, I'm now convinced that the speed difference is specifically the time required to allocate an empty string. The time required to fill the empty string with data seems to be trivial.

It isn't surprising to learn that the pymalloc allocator is much better for small allocations than the system allocator. Not sure what we can do about this, aside from experiment with enabling other memory allocators in Numba (which goes beyond the scope of this PR). I'll add a note to the string docs to warn people about the performance issues.

[seibert]

Perf difference is still not entirely understood, but I think now goes beyond the scope of this PR. Now just holding for default argument support in overload_method().

[ehsantn]

Some links below about string immutability issues we discussed. Looks like the aspects we discussed are the main considerations.

https://stackoverflow.com/questions/18042042/pythons-immutable-strings-and-their-slices
https://stackoverflow.com/questions/6742923/if-strings-are-immutable-in-net-then-why-does-substring-take-on-time
https://blogs.msdn.microsoft.com/ericlippert/2011/07/19/strings-immutability-and-persistence/

[seibert]

With #3704 merged now, maxsplit support has been enabled. This PR is ready for review now.

[seibert]

OK, I think I've addressed all of Stuart's comments, unless we want str.join(str) to have a faster implementation.

[stuartarchibald]

thanks for the fixes, there's just a couple more things to sort out.

[stuartarchibald]

Did flake8 not complain about this line ?!

[seibert]

When I enabled flake8, I copied over Dask's error suppressions (as that seemed a good style to copy):

ignore =
    E20,   # Extra space in brackets
    E231,E241,  # Multiple spaces around ","
    E26,   # Comments
    E731,  # Assigning lambda expression
    E741,  # Ambiguous variable names
    W503,  # line break before binary operator
    W504,  # line break after binary operator
max-line-length = 120

[stuartarchibald]

None is fine as a literal, but as an argument types.nonetype, this would fail. e.g.:

from numba import njit

def foo(x, y):
    return x.split(sep=y)

args= ('abacadae', None)
print('"%s"' % njit(foo)(*args))
print(foo(*args))

as a consequence of adding literals, both literal and as-arg typing is needed. This sort of pattern seems to be working:

sep is None or isinstance(sep, types.NoneType) or getattr(sep, 'value', False) is None:

[seibert]

fixed

[stuartarchibald]

The @overload safety net needs extending to @overload_method. This function declaration has a signature that does not match the typing signature. maxsplit is not implemented and sep is missing as a kwarg, this breaks as a result:

from numba import njit

def foo(x):
    return x.split(maxsplit=3)

args= ('\taa a a aa a aa a',)
print('"%s"' % njit(foo)(*args))

[seibert]

fixed

[stuartarchibald]

Perhaps numba.unsafe.bytes?

[seibert]

good catch. typo should be fixed now

[stuartarchibald]

These probably need including: https://github.com/python/cpython/blob/5105483acb3aca318304bed056dcfd7e188fe4b5/Objects/unicodetype_db.h#L5996-L6031

[seibert]

ok, should be fixed

[stuartarchibald]

Thanks for your persistence in fixing these last few issues. I think conditional on CI pass this can be merged.

[seibert]

CI is passing (except the known Windows Py27 issue).