hpfilter: changing lambda parameter for certain data does not affect trend

[Henry-White]

Describe the bug

statsmodels 0.14.1

On certain data hpfilter doesn't work properly. I've changed lamb param on wide range (1600 - 20000), but it doesn't affected on trend line at all.

Code Sample,

Colab env with data

or
Tick time-series from HW adc1.zip

import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from statsmodels.graphics.tsaplots import plot_acf
import statsmodels.api as sm

# Assuming df is your DataFrame
df = pd.read_csv('/content/adc1.csv', sep = ';')
df.head()

Y= df['2'].copy()
df_norm = (Y-Y.min())/(Y.max () - Y.min ())
# Y= df['2'].astype( 'float', copy= True)

window_size = 14600
df['high_cycle'], df['high_smoothed'] = sm.tsa.filters.hpfilter(df_norm, lamb= window_size)

# Plotting
w=10
h=6
plt.figure(figsize=(w, h))

fig, ax1 = plt.subplots()
fig.set_figwidth(w)
fig.set_figheight(h)
ax2 = ax1.twinx()

ax1.plot(df_norm, label='Original norm', color='blue', linewidth = 1)
ax1.plot(df['high_smoothed'], linestyle='-', color='orange', linewidth = 1)
plt.xlabel('Tick')
plt.legend()
plt.show()

Output of import statsmodels.api as sm; sm.show_versions()

INSTALLED VERSIONS ------------------ Python: 3.10.12.final.0 OS: Linux 6.1.58+ #1 SMP PREEMPT_DYNAMIC Sat Nov 18 15:31:17 UTC 2023 x86_64 byteorder: little LC_ALL: en_US.UTF-8 LANG: en_US.UTF-8

statsmodels

Installed: 0.14.1 (/usr/local/lib/python3.10/dist-packages/statsmodels)

Required Dependencies

cython: 3.0.8 (/usr/local/lib/python3.10/dist-packages/Cython)
numpy: 1.25.2 (/usr/local/lib/python3.10/dist-packages/numpy)
scipy: 1.11.4 (/usr/local/lib/python3.10/dist-packages/scipy)
pandas: 1.5.3 (/usr/local/lib/python3.10/dist-packages/pandas)
dateutil: 2.8.2 (/usr/local/lib/python3.10/dist-packages/dateutil)
patsy: 0.5.6 (/usr/local/lib/python3.10/dist-packages/patsy)

Optional Dependencies

matplotlib: 3.7.1 (/usr/local/lib/python3.10/dist-packages/matplotlib)
backend: module://matplotlib_inline.backend_inline
cvxopt: 1.3.2 (/usr/local/lib/python3.10/dist-packages/cvxopt)
joblib: 1.3.2 (/usr/local/lib/python3.10/dist-packages/joblib)

Developer Tools

IPython: 7.34.0 (/usr/local/lib/python3.10/dist-packages/IPython)
jinja2: 3.1.3 (/usr/local/lib/python3.10/dist-packages/jinja2)
sphinx: 5.0.2 (/usr/local/lib/python3.10/dist-packages/sphinx)
pygments: 2.16.1 (/usr/local/lib/python3.10/dist-packages/pygments)
pytest: 7.4.4 (/usr/local/lib/python3.10/dist-packages/pytest)
virtualenv: Not installed

[bashtage]

It does change. The small variation is simply a function of your data.

In [30]: _,a=sm.tsa.filters.hpfilter(df_norm, lamb= 900)

In [31]: _,b=sm.tsa.filters.hpfilter(df_norm, lamb= 14400)

In [32]: (b-a).abs().max()
Out[32]: 0.07377144333098037

In [33]: (b-a).iloc[1000:].abs().max()
Out[33]: 0.07377144333098037

[Henry-White]

The variations are quite significant. But in the short term you will not see them due to the high data density.
As you can see, there is no difference between lambda=1600 and 14600 (оr even more - 100000600 for example):

[bashtage]

All that the pictures show is that once you get the smoothing parameter large enough, there isn't much difference. But there is a numerical difference, and the HP filter code implementation is correct. What do you expect to happen?