Refactor Non-normalized Methods

[seanlaw]

Instead of having aamp, we should allow users to set normalize=False and have this automatically get re-routed to its corresponding non-normalized method. This would greatly reduce confusion and API support. This can be accomplished by wrapping our normalized functions/classes with a decorator.

[seanlaw]

@hmcoservit Just a heads up that we've simplified the interface in v1.8.0 so you can now do:

mp = stumpy.stump(T, m=50, normalize=False)

and this will be identical to:

mp = stumpy.aamp(T, m=50)

Both are still supported but, in the background, when normalize=False is detected this automatically calls the non-normalized version instead. And all z-normalized functions have a normalize=False equivalent now (i.e., every z-normalized function now carries a normalize parameter).

This should simplify the API.

[hmcoservit]

Thanks for the heads up @seanlaw. That is great, thanks for informing me.

Sorry about the period of silence, I was caught with other priorities. Soon matrix profile will become top priority again, I will keep you posted ;)

[hmcoservit]

Hi @seanlaw, I am doing performance and stress tests for data ingestion using aampi. I noticed something, maybe you can point me to somewhere:

After running a first load:
matrix_profile_stream_out = stumpy.aampi(metric_df['value'].values, subsequence_length, egress=True)

And then updating every time a new point arrives:
pickle.load(...)
matrix_profile_stream_out.update(value_list[-1])

And then dumping using Pickle:
dumped_stream = pickle.dumps(matrix_profile_stream_out, protocol=2)

I noticed that the size of the object increases as new points are added. Initially I thought it is the problem is with Pickle, but even when I used the correct protocol version, it still increases.

I was going to ask you that if there is anything that can cause the size of stumpy aampi object to keep on increasing even when egress=True?

[seanlaw]

@hmcoservit I am not sure about pickle but I looked at the aampi code to see if _update_egress was appending values to self attributes but I didn't see anything that looked out of the ordinary:

https://github.com/TDAmeritrade/stumpy/blob/4ea9fb8368498eba2581aa50a8fc7d782025bb0c/stumpy/aampi.py#L152-L212

All self attributes appear to be updating using numpy indexing, which implies that arrays should not be growing. Can you try adding a single new point at a time and then plotting how much the pickled object grows as you add up to 10 thousand data points (one at a time)? Is pickle size growing monotonically?

[hmcoservit]

Thanks for your confirmation @seanlaw. Yes you are right, the problem seems to be with Pickle, when dumping and loading the objects. I have to investigate further, as to why after Pickle dump the size is increased every time.

The reason for using Pickle is for an experiment to use the Python object inside Java in Flink.

[hmcoservit]

Hi @seanlaw, back to this issue that we faced, just wanted to update you.

We managed to solve it by doing something like this:

model = stumpy.aampi(..., subsequence_length, egress=True)

# constraining arrays in model to a fixed size
model._P = np.array(model._P, dtype=float)
model._left_I = ...

By specifying the data type of the array (cells no longer stored as objects), we will not have variable sizes anymore.

[seanlaw]

Hi @seanlaw, back to this issue that we faced, just wanted to update you.

We managed to solve it by doing something like this:

model = stumpy.aampi(..., subsequence_length, egress=True)

# constraining arrays in model to a fixed size
model._P = np.array(model._P, dtype=float)
model._left_I = ...

By specifying the data type of the array (cells no longer stored as objects), we will not have variable sizes anymore.

Thanks @hmcoservit. I wonder if this could also be resolved directly in the aampi function by explicitly casting these lines to the proper types:

https://github.com/TDAmeritrade/stumpy/blob/10b26725b7d5a6b74ef416083860e48f168a46ce/stumpy/aampi.py#L99-L102

[hmcoservit]

@seanlaw To solve it internally perhaps the index values can be cast to int and the mp values to float?

[seanlaw]

@seanlaw To solve it internally perhaps the index values can be cast to int and the mp values to float?

Right. That's exactly what I was thinking:

mp = aamp(self._T, self._m) 
self._P = mp[:, 0].astype(float)
self._I = mp[:, 1].astype(int)
self._left_I = mp[:, 2].astype(int)

[hmcoservit]

@seanlaw seems good