Cannot get the right envelope of the raw signal when using sliding_window_amplitude()

[pascalschulthess]

I'm scripting pyBOAT and having trouble recreating the plot of the raw signal together with the envelope. Im using the provided scripting_demo.py

 --- Filtering ---

# calculate the trend with a 60s cutoff
trend = wAn.sinc_smooth(signal, T_c=60)

# calculate envelope of raw signal with a sliding window of 70s
envelope_raw = wAn.sliding_window_amplitude(signal, window_size=70)

# detrending here is then just a subtraction
detrended_signal = signal - trend

# calculate envelope of detrended signal with a sliding window of 70s
envelope_detr = wAn.sliding_window_amplitude(detrended_signal, window_size=70)

# normalize the amplitude with a sliding window of 70s
norm_signal = wAn.normalize_amplitude(detrended_signal, window_size=70)

When I now plot together detrended_signal and envelope_detr it looks like the result in UI. But not when I plot together signal, trend, and envelope_raw. Here the envelope is very different from what is shown in the UI.

[tensionhead]

Thx for reaching out @pascalschulthess! Before digging any deeper.. in the UI there are two y-axes in the dataviewer plot (one for original signal, and one for detrended signal, can make a huge difference if there are strong low-freq. trends present) Depending on what you plot together, the envelope should adhere to the current scale. Maybe this does not work as expected in the UI?

It is easily possible to post images here (just drag and drop into a comment), could you maybe post the two discrepant plots here? Also, which one you think is the correct one 😁 ? Thanks!

EDIT: we can also continue on chat here: https://gitter.im/pyBOATbase/support

[tensionhead]

Ok so to address this directly here, to begin with you posted a line reading wAn.sliding_window_amplitude(...), however assuming that wAn is an instance of the WAnalyzer, there is no such method in pyboat 0.9.8! Maybe you are running an older version?!

There is an wAn.get_envelope(...), and with this I produced this following plot:

Which I assume is what we expect?

The script to produce this plot I put in a gist
Hope this helps 🙂 ?!

[tensionhead]

Ok, so I think we finally got to the bottom of this, the problem was within the UI of course.

Note that in no case actual computations were inconsistent, both the scripting and the UI had the correct envelope all the time

It's just the UI had a plotting tweak when plotting the raw signal along with the trend and envelope that is now probably regrettable. There are basically two cases:

• 'real' low freq trends which don't 'oscillate' themselves, or where the period is much longer than the time-series.
• slower oscillatory components, which of course also get removed by the sinc filter

pyBOAT 0.9.8 (old)

The right plot is the situation @pascalschulthess ran into. So what is actually plotted here is envelope + trend, which makes it nicely fitting on top of the signal on the left (very low freq non-oscillatory trend). But looks awkward on the right, as now the envelope seemingly oscillates, however it's of course the 'trend' which captures the slower oscillations.

pyBOAT 0.9.9 (new)

Now with the new 0.9.9 there is really only the 'naked' envelope plotted alongside the raw signals, and the situation reverses. The effective envelope with a strong very low-freq trend present will not sit on top of the signal, as it is dominated by the (non-zero mean) trend. On the right, for the slower oscillation it now fits more or less exactly, as this 'trend' is actually 0-mean.

Conclusion

Even though I think for most use cases, non-parametic detrending and not removal of a 2nd slower oscillation, the plots will look poorer, this is still better as it is non-ambigious. I should have at least communicated somehow clearly that in the UI envelope + trend were plotted, but now the UI results fit directly to the scripting results.
Note that for the detrended signal + envelope plots nothing changes, and both old and new look fine! So I would recommend to always plot the envelope on the detrended signal, if detrending is anyways to be done!

Last but not least, thx to @pascalschulthess for finding and reporting this inconsistency between scripting interface and the UI!

[pascalschulthess]

Great, thanks for the collaboration in working this out.