Fix bitarray related bugs

[Zhicheng-Liu]

This pull request contains changes to fix the following issues:

1. Since bitarray 0.9.0, the numpy.where(bitarray) call does not return
   the correct non zero positions for bits. Instead, it returns the indices
   of non zero bytes. For example, for bitarray("0000000001000000"),
   the function call returns non zero positions [1], instead of [9], since
   bitarray release 0.9.0. This pull request fixes that issue by unpacking the
   bitarray first before getting non zero positions.

2. The bitarray constructor does not initialise values. For example, if you
   want to set the initial values in the constructed bitarray when you call
   bitarray(size) you will have to call setAll(0) to set all bits to 0. In our
   BloomFilter class we call the bitarray consturctor that way, but did not
   initialise the bitarray with zeros. This pull request fixes that issue.

3. Also fixes the Travis build.

Co-authored-by: leandro

[iqbal-lab]

Any chance of a code review @mbhall88 @martinghunt or @leoisl?

[mbhall88]

I think I found a much faster solution (thanks SO). Benchmarking done in jupyter notebook

from bitarray import bitarray
import numpy as np
from itertools import compress, count

arr = bitarray("0000000001000000")

def f1(arr):
    """Taken from https://stackoverflow.com/a/4111521/5299417"""
    gen = compress(count(), arr)
    return gen

>>> list(f1(arr))
[9]

%%timeit
f1(arr)

# 338 ns ± 7.86 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

## the above is slightly unfair as it returns a generator
## a version which evaluates the whole generator

def f1a(arr):
    """Taken from https://stackoverflow.com/a/4111521/5299417"""
    gen = compress(count(), arr)
    return list(gen)

>>> f1a(arr)
[9]

%%timeit f1a(arr)
# 672 ns ± 10.2 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

def f2(arr):
    return [index for index in range(len(arr)) if arr[index]]

>>> f2(arr)
[9]

%%timeit
f2(arr)

# 2.14 µs ± 58.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Note: I tried a bunch of others but nothing came close to the itertools magic.

And on a massive bit array

arr = bitarray('001101100110101001001010') * 2000000

%%timeit
f1(arr)
# 333 ns ± 8.49 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

%%timeit
f1a(arr)
# 1.6 s ± 13.5 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

%%timeit
f2(arr)
# 6.17 s ± 57 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

I guess I don't really know how the output of this function will be used but generators are great and will save lots of time and memory.

[Zhicheng-Liu]

Hi @mbhall88 , that's interesting and prompted me to do some profiling on this particular case with four different implementations:

1. [index for index in range(len(bitarray)) if bitarray[index]]

2. [index for index, bit in enumerate(bitarray) if bit]

3. numpy.where(numpy.unpackbits(bitarray))[0].tolist()

4. [itertools.compress(itertools.count(), bitarray)]

And I profiled them with these length categories of bitarrays:

• 1 - 800
• 800 - 4,000
• 4,000 - 16,000
• 16,000 - 80,000

The results are interesting:

1. impl 1 is slowest in most categories, except for the first categories.
2. impl 2 is fast and comparable to itertools one in lower categories, but slow down in longer categories.
3. impl 3 is slowest in the first category, but it is the fastest one in the last category.
4. impl 4 is consistently fast across all categories compared to others, but slower than numpy implementation in the last category.

So, after seeing this, I have decided to revert back to use numpy in the implementation, because it is only with larger bitarray that the performance could become a concern. I will use the implementation 2 with the builtin enumerate function in the test for readability. I did not use itertools a lot before, it took me a little while to figure what implementation 4 is doing.

This has been updated in b508e22.

Thank you very much for pointing this out. The itertools is really good, very useful in a lot of cases I believe. As you rightly pointed out, the generator is performant, time and space wise.

[leoisl]

Amazing work @Zhicheng-Liu ! The changes and tests are essential. I just have a minor comment.

[leoisl]

All fine for me!