Add Unit Tests for `mmotifs` and `aamp_mmotifs`

[seanlaw]

The mmotifs and aamp_mmotifs features were recently added but they need unit tests to accompany them:

• Add naive implementation for mmotifs and aamp_motifs in tests/naive.py
• Add pytest -x -W ignore::RuntimeWarning -W ignore::DeprecationWarning tests/test_motifs.py to the test_unit function in test.sh
• Add pytest -x -W ignore::RuntimeWarning -W ignore::DeprecationWarning tests/test_aamp_motifs.py to the test_unit function in test.sh
• Add unit tests for mmotifs in tests/test_mmotifs.py
• Add unit tests for aamp_motifs in tests/test_aamp_mmotifs.py
• Add mmotifs and aamp_motifs to stumpy/__init__.py
• Add mmotifs to docs/api.rst
• Add Example section to mmotifs() docstring:

    Examples
    --------
    >>> mp = stumpy.stump(np.array([584., -11., 23., 79., 1001., 0., -19.]), m=3)
    >>> stumpy.motifs(
    ...     np.array([584., -11., 23., 79., 1001., 0., -19.]),
    ...     mp[:, 0],
    ...     max_distance=2.0)
    (array([[0.        , 0.11633857]]), array([[0, 4]]))

[seanlaw]

@SaVoAMP had asked:

Unfortunately, I have never worked with unit tests, which means that I am a complete beginner. Are there any resources available since I have absolutely no clue how to start?

Unfortunately, there isn't a good resource since we do things somewhat uniquely in STUMPY but I can maybe provide a starting point for you to explore and then have you ask questions. Considering how quickly you figured out the Python decorator stuff, I think you'll be able to pick this up quickly.

[SaVoAMP]

All right, so how to implement functions naivly? 🤔
Can I then also start with the tutorial in parallel?

[seanlaw]

@SaVoAMP Normally, before a new feature/code is allowed to be merged, we usually require that all new features/code to be accompanied by proper unit tests. In your case, I made an exception so that we can make things a bit easier. Here's a rough list of things that you'll want to do:

1. Start fresh
2. With your PR merged, please go ahead and delete/remove your fork of STUMPY
3. Fork and clone the STUMPY repository again as you had done before and create a new "unit_test" branch

Hopefully, this should be a clean and familiar starting point where we can start writing our tests. However, unlike before, you may need to also install the additional packages necessary for unit testing. If you are already using conda then you can try executing the conda.sh script in the STUMPY root directory OR, if you are a pip user, you might have some luck using the pip.sh script (also in the root directory), which essentially converts the environment.yml file to a temprary requirements.dev.txt file and then installs it via pip. To check if your environment is correctly setup, execute:

./setup.sh && ./test.sh

Running the tests takes some time but is a good exercise. Please report back once you've attempted these steps.

[seanlaw]

Can I then also start with the tutorial in parallel?

Yes, please feel free to start!

All right, so how to implement functions naively? 🤔 Can I then also start with the tutorial in parallel?

In this case, given that your functions are already quite simple, I think we might be able to avoid adding a naive implementation (I'll remove that from the checklist) and only worry about testing that specific inputs would produce specific outputs. In other words, given a multi-dimensional time series and its corresponding multi-dimensional matrix profile, what should be the expected outputs?

As an oversimplified example here's what I would do:

Imagine that you had added a new function/feature in stumpy/some_func.py:

import numpy as np

def some_func(T, k=2):
    # Find the `k` smallest values in T
    T = np.sort(T)
    return T[:k]

You'd add this function to the top level of stumpy/__init__.py (see the second last line):

from .motifs import motifs, match  # noqa: F401
from .aamp_motifs import aamp_motifs, aamp_match  # noqa: F401
from .snippets import snippets  # noqa: F401
from .aampdist_snippets import aampdist_snippets  # noqa: F401
from .stimp import stimp, stimped  # noqa: F401
from .some_func import some_func  # noqa: F401
from numba import cuda

This will ensure that when you import stumpy into any project, you can access the function directly via stumpy.some_func:

import stumpy
import numpy as np

T = np.random.rand(10)
stumpy.some_func(T)

If you don't do this then you can only access the function via from stumpy.some_func import some_func. Now, we can write a test for this simple function:

1. Create a new file called tests/test_some_func.py
2. Add a new test in tests/test_some_func.py like:

import numpy as np
import numpy.testing as npt
from stumpy import some_func

test_data = [
    (np.array([[584, -11, 23, 79, 1001, 0, -19]], dtype=np.float64)),
]

@pytest.mark.parametrize("T", test_data)
def test_some_func(T):
    ref = np.array([-19.0, -11.0])
    cmp = some_func(T)
    npt.assert_almost_equal(ref, cmp)


@pytest.mark.parametrize("T", test_data)
def test_some_func_3_smallest(T):
    ref = np.array([-19.0, -11.0, 0.0])
    cmp = some_func(T)
    npt.assert_almost_equal(ref, cmp)

Notice that we hardcoded the inputs in test_data and we hardcoded the expected outputs in ref. Now, if I had a more complex function, I might have created a naive implementation in naive.py (again, you can skip this approach):

# naive.py

def some_func(T, k):
    T = T.copy()
    out = np.empty(k)
    for i in k:
        smallest_val = np.inf
        smallest_idx = -1
        for j, val in enumerate(T):
            if val < smallest_val:
                smallest_val = val
                smallest_idx = j
        T[smallest_idx] = np.inf
        out[i] = smallest_val

    return out

Now, this example is oversimplified and it is important to point out that we do not/should not need to write unit tests for numpy functions but sometimes when the STUMPY function is too simple/obvious, it might be necessary to replace numpy functions with Python for-loops.

Finally, to run the tests, you can do:

./setup.sh && test.sh

Of course, this will run ALL tests. However, this will take a long time. So, you can tell test.sh to run only your tests by changing the test_custom function test.sh as shown below:

test_custom()
{
    # export NUMBA_DISABLE_JIT=1
    # export NUMBA_ENABLE_CUDASIM=1
    # Test one or more user-defined functions repeatedly
    for VARIABLE in {1..10}
    do
        pytest -x -W ignore::RuntimeWarning -W ignore::DeprecationWarning tests/test_some_func
        check_errs $?
    done
    clean_up
    exit 0
}

Then, you can execute your custom test by adding a "custom" keyword:

./setup.sh && ./test.sh custom

Eventually, in order to run the full unit test suite (i.e., ./setup.sh && test.sh) and not only the custom tests, you'll want to add additional lines to the test_unit function in test.sh

pytest -x -W ignore::RuntimeWarning -W ignore::DeprecationWarning tests/test_motifs.py
check_errs $?
pytest -x -W ignore::RuntimeWarning -W ignore::DeprecationWarning tests/test_some_func.py
check_errs $?

Please let try it out and let me what problems you run into. I'm writing this from the top of my head and I hope to use these notes as something that we can/should add to our contributor guide.

[SaVoAMP]

Running the tests takes some time but is a good exercise. Please report back once you've attempted these steps.

I think I have got it! I've run the tests and got results as shown:

[seanlaw]

I think I have got it! I've run the tests and got results as shown:

Awesome! Hmm, the results look strange though. STUMPY usually ensures 100% code coverage and your results are showing only 99% coverage. It looks like line 227-238 of stumpy/aamp.py and lines 408-419 of stumpy/stump were not traversed during the tests. I would've expected mmotifs and aamp_mmotifs to not be covered but not those modules! I will need to investigate why mmotifs and aamp_motifs aren't triggering missing coverage.

Btw, did you have to add any additional steps that may have been omitted in my description above?

[seanlaw]

I will need to investigate why mmotifs and aamp_motifs aren't triggering missing coverage.

@SaVoAMP I fixed this. Can you please pull and merge the latest commit to your branch and rerun the tests? You should see:

tests/test_stamp.py ....................                                 [ 86%]
tests/test_stimp.py ...........                                          [ 87%]
tests/test_stomp.py ........................                             [ 89%]
tests/test_stump.py .......................                              [ 90%]
tests/test_stumped.py .................................................. [ 94%]
................................................                         [ 98%]
tests/test_stumpi.py ................                                    [100%]

======================= 1251 passed in 486.07s (0:08:06) =======================
Name                     Stmts   Miss  Cover   Missing
------------------------------------------------------
stumpy/aamp_mmotifs.py      51     51     0%   5-195
stumpy/mmotifs.py           51     51     0%   5-203
------------------------------------------------------
TOTAL                    10619    102    99%

72 files skipped due to complete coverage.
Cleaning Up

[SaVoAMP]

Btw, did you have to add any additional steps that may have been omitted in my description above?

Not really, I reforked the repo, added a branch, cloned it inside my linux vm, activated a virtual environment for python and ran pip.sh and ./setup.sh && test.sh as you described.

[SaVoAMP]

tests/test_stamp.py ....................                                 [ 86%]
tests/test_stimp.py ...........                                          [ 87%]
tests/test_stomp.py ........................                             [ 89%]
tests/test_stump.py .......................                              [ 90%]
tests/test_stumped.py .................................................. [ 94%]
................................................                         [ 98%]
tests/test_stumpi.py ................                                    [100%]

====================== 1254 passed in 2003.87s (0:33:23) =======================
Name                     Stmts   Miss  Cover   Missing
------------------------------------------------------
stumpy/aamp.py              80     10    88%   227-238
stumpy/aamp_mmotifs.py      51     51     0%   5-195
stumpy/mmotifs.py           51     51     0%   5-203
stumpy/stump.py            107     10    91%   408-419
------------------------------------------------------
TOTAL                    10637    122    99%

70 files skipped due to complete coverage.
Cleaning Up

Hmm strange. I still have got something different 🤔

[seanlaw]

Hmm strange. I still have got something different

So, I looked at those lines in stumpy.stump and stumpy.aamp and that only happens if you somehow are limited to a single thread. You had briefly mentioned:

cloned it inside my linux vm, activated a virtual environment for python

Can you tell me how many threads are made available to the linux vm? Is it a Windows subsystem? If the answer is 1 thread then that would explain why stumpy.stump and stumpy.aamp are coming up.