Default handling of empty strings

[MurrayData]

I've been evaluating cuStrings/nvstrings in the latest Rapids container, and like it. It is very fast. I'm going to try it with some more involved string processing later.

I've tried it on UK Companies House corporate register, company names, and in this example the content of 10.8 million UK Tweets.

import nvstrings
filename = "/data/mesh_twitter_20181211/Data/mesh_twitter_20181211.csv"

%%time
a = nvstrings.from_csv(filename,3)

CPU times: user 1.69 s, sys: 1.82 s, total: 3.51 s
Wall time: 3.5 s

I've observed that an empty string in the source data is converted into a None type, and this causes errors in post analysis work for example summing records matching a criterion.

%%time
s = a.size()
c = a.count('\#')
h = c.count(0)
print("Number of tweets:",s,"Number of hashtags:",sum(c),"Tweets including a hashtag:",s-h,round(100.0*(s-h)/s,2),"%")

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<timed exec> in <module>

TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

I am having to use a Python list comprehension, or a filter, to check for None type before further processing in such cases. Filter is ok for map reduction type operations, but if order or volume need to be preserved for concatenation, then it would need a list comprehension.

%%time
s = a.size()
c = [x if x is not None else 0 for x in a.count('\#')]
h = c.count(0)
print("Number of tweets:",s,"Number of hashtags:",sum(c),"Tweets including a hashtag:",s-h,round(100.0*(s-h)/s,2),"%")

Number of tweets: 10836607 Number of hashtags: 10801624 Tweets including a hashtag: 4641176 42.83 %
CPU times: user 833 ms, sys: 169 ms, total: 1 s
Wall time: 998 ms

In strings, an empty string can actually be a value ''. It may be concatenated with other fields too.

Therefore my suggestion, in the from_csv function, would be good to have an option to treat empty strings as a value or None. I would recommend '' as the default,. Possibly a Boolean argument?

[davidwendt]

This is great feedback. So there is a little trick here using device memory to hold the results from contains() where the None value is turned identified as -1 as so can be ignored/counted:

>>> import nvstrings
>>> import numpy as np
>>> from numba import cuda
>>>
>>> a = nvstrings.to_device(["test #woot","strings are awesome","#hashme",None,""])
>>> a.count('\#')
[1, 0, 1, None, 0]

>>> arr = np.arange(a.size(),dtype=np.int32)
>>> d_arr = cuda.to_device(arr)
>>> a.count('\#',d_arr.device_ctypes_pointer.value)

>>> d_arr.copy_to_host()
array([ 1,  0,  1, -1,  0], dtype=int32)

>>> c = d_arr.copy_to_host().tolist()
>>> c.count(0)
2
>>> c.count(-1)
1

The -1 is a place holder identifying None strings for contains() in this case.

[davidwendt]

In the next release of nvstrings, we are adding a couple more parameters to the from_csv method including a set of flags. This will include a flag to treat nulls as empty strings. Also, the next release should include some documentation on from_csv

[davidwendt]

Parameter added in new 0.2.0 release.