Presumably shifted uncertainty result for FIRuncFilter for non-stationary uncertainty

[mgrub]

In cases of non-stationary uncertainty (king="diag"), the output of FIRuncFilter returns an uncertainty-result, that seems to be shifted.

Looking into the code, this is very likely caused by the equations to calculate UncCov: L.182 + L.186 + L.190 + L.192. Here, theta and Ulow are multiplied. However, for values in theta it holds theta[i+1] refers to an earlier point in time than theta[i] - while in Ulow it is reversed. Therefore, during the multiplication values are combined, that do not correspond in their time-order. To fix this, theta probably needs to be reversed, to achieve correct (time-ascending) order.

It seems like this misbehaviour is not covered by tests or examples so far. Therefore a suitable test/example should be introduced (Maybe against some MC-method). A comparison with the upcoming IIRuncFilter could also be an option.

Environment

• OS: Windows 10
• PyDynamic Version 1.6.0 (direct from master)

MWE showing comparison with MC-result

import matplotlib.pyplot as plt
import numpy as np

from PyDynamic.uncertainty.propagate_MonteCarlo import MC
from PyDynamic.uncertainty.propagate_filter import FIRuncFilter

n_signal = 100
n_filter = 50

x = np.append(np.arange(n_signal//2), np.zeros(n_signal//2))
ux = 0.1 * np.abs(x)

theta = np.array([0.5,0.5] + [0] * (n_filter - 2))
utheta = np.zeros((n_filter, n_filter))


# run filter
y, uy = FIRuncFilter(
    x, ux, theta, utheta, kind="diag"
)  # apply uncertain FIR filter (GUM formula)
yMC, uyMC = MC(
    x, ux, theta, [1.0], utheta, runs=1000
)  # apply uncertain FIR filter (Monte Carlo)


# compare FIR and MC results
fig, ax = plt.subplots(nrows=2)

ax[0].plot(x, label="x")
ax[0].plot(theta, label="theta")
ax[0].plot(y, label="y")
ax[0].plot(yMC, label="y (MC)")
ax[0].legend()

ax[1].plot(ux, label="ux")
ax[1].plot(np.diag(utheta), label="utheta")
ax[1].plot(uy, label="uy")
ax[1].plot(np.sqrt(np.diag(uyMC)), label="uy (MC)")
ax[1].legend()

plt.show()

Output of MWE
Current situation:

Expected result:

Note
The effect becomes especially obvious for long filter lengths and if the signal gets appended by many zeros.

[eichstaedtPTB]

Thanks for pointing this out. Could you extend your example above with an application of IIRuncFilter to see how that behaves in that situation?

[BjoernLudwigPTB]

One first thing while starting to code on that should be implementing a test, that reveals that erroneous behaviour.

[mgrub]

Thanks for pointing this out. Could you extend your example above with an application of IIRuncFilter to see how that behaves in that situation?

I will do that. However, I am not fully certain, what's the best way to achieve this. The revised IIRuncFilter currently only resides in the prepare_release_2.0.0 branch, which I don't want to pull in its entirety into a fix. Also,I can't get IIRuncFilter in the master-branch to work. So I could do the comparison in a q'n'd-fashion or we could decide to release the revised IIRuncFilter prior to PyDynamic 2.0.0.

One first thing while starting to code on that should be implementing a test, that reveals that erroneous behaviour.

Yes! However, as long as the IIRuncFilter is not available in the master branch, I can only think of an approximative check with a Monte-Carlo run. (Similar to the MWE proposed above.)

[mgrub]

Here are some screenshots with the revised IIRuncFilter added to the comparison.

Issue active: IIR and MC show good agreement, FIR differs

Issue resolved: IIR and FIR match exactly, MC shows good agreement to both

Note: FIRuncFilter is based on a method, that incorporates an additional term over the pure GUM-propagation, while IIRuncFilter sticks to the GUM-scheme. Therefore IIRuncFilter and FIRuncFilter can only show the exact same output if Utheta=np.zeros((n_filter,n_filter)). As soon as Utheta becomes a full covariance matrix, the observed shift of this bug-report becomes less noticable, as the contribution of other terms becomes more prominent.

[BjoernLudwigPTB]

Thanks for pointing this out. Could you extend your example above with an application of IIRuncFilter to see how that behaves in that situation?

I will do that. However, I am not fully certain, what's the best way to achieve this. The revised IIRuncFilter currently only resides in the prepare_release_2.0.0 branch, which I don't want to pull in its entirety into a fix. Also,I can't get IIRuncFilter in the master-branch to work. So I could do the comparison in a q'n'd-fashion or we could decide to release the revised IIRuncFilter prior to PyDynamic 2.0.0.

One first thing while starting to code on that should be implementing a test, that reveals that erroneous behaviour.

Yes! However, as long as the IIRuncFilter is not available in the master branch, I can only think of an approximative check with a Monte-Carlo run. (Similar to the MWE proposed above.)

Then we should simply push towards releasing as soon as possible. Or do we need a backport of this bug fix to 1.x anyway?

[mgrub]

Then we should simply push towards releasing as soon as possible. Or do we need a backport of this bug fix to 1.x anyway?

I think, now that I have implemented a fix in PR #184 , we should release that in 1.x. But we should rewrite some tests after the initial 2.0.0 release.