Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

is sm.tsa.filters.arfilter an AR filter? #1478

Closed
jseabold opened this issue Mar 14, 2014 · 7 comments

Comments

Projects
None yet
2 participants
@jseabold
Copy link
Member

commented Mar 14, 2014

In the terminology R uses in their filter function [1], what we have as arfilter is just the convolution method of their filter, which is just a centered moving average. Their recursive method is autoregressive and we would do the same using scipy.signal.lfilter. I was going to add a method argument to arfilter for convolution vs. recursive, but I thought to maybe just rename it to linear_filter first? Thoughts?

[1] http://stat.ethz.ch/R-manual/R-patched/library/stats/html/filter.html

@jseabold

This comment has been minimized.

Copy link
Member Author

commented Mar 14, 2014

Or arfilter -> convolution_filter and arfilter uses signal.lfilter internally.

@josef-pkt

This comment has been minimized.

Copy link
Member

commented Mar 14, 2014

In terms of general terminology.

I had big problems keeping apart which is which, especially also with the signal processing terminology in scipy.signal (num, denom), and I didn't read R at the time.

I tried to translate things into ARMA terminology, ar_filter, ma_filter for things that signal.lfilter can do, ARMA terminology is more mnemonic for me. (assuming I got it correct)

edit: I'm not sure about the following after 30 minutes:
I think the term arfilter is wrong, a moving average, convolution filter should be an mafilter.

assume y is observed

y_t for ar(L) y_t = e_t
versus
y_t = ma(L) e_t

ma(L) would be a FIR filter (finite impulse response) in scipy.signal
ar or arma is a IIR filter

I thought I had an arma_filter wrapper around lfilter, but cannot find anything like that.

ArmaFft in the sandbox has a frequency domain arma filter method.

one of the things I got confused about in signal processing terminology is transfer function filter, where x is the input and y is the output


linear_filter sounds fine,

related: for using lfilter we should have the correct choice of starting values, so we get identical results to recursion with fixed initial conditions.

@jseabold

This comment has been minimized.

Copy link
Member Author

commented Mar 14, 2014

The one-sided convolution filter maps to MA using lfilter. These should give the same, but it doesn't look like lfilter properly handles NA values in this case. Looks like a bug to me.

signal.convolve(dta.co2.values, [.75, .25], mode='valid')
signal.lfilter([0, .75, .25], [1.], dta.co2.values)

I made a quick mapping between convolve, lfilter and R's filter methods, optionally using initial conditions. For the one-sided convolution, I might just do a one-linear using pd.rolling_apply since it handles NAs correctly.

@jseabold

This comment has been minimized.

Copy link
Member Author

commented Mar 14, 2014

Sorry, no those shouldn't give the same. The first line should be

filter(dta$co2, c(.75, .25), sides=1)

in R.

@josef-pkt

This comment has been minimized.

Copy link
Member

commented Mar 14, 2014

lfilter should have leading ones in both polynomials, and the sign is polynomial not regression

signal.lfilter([1, .75, .25], [1.], dta.co2.values)

@jseabold

This comment has been minimized.

Copy link
Member Author

commented Mar 14, 2014

If you want a one-sided convolution filter, then the 0 lag for MA is 0. I.e., don't include the current term if you're doing a moving average that should only include past terms.

@josef-pkt

This comment has been minimized.

Copy link
Member

commented Mar 14, 2014

ok, right for shifted to use only past information,
the ar part needs (or needed) to have a strictly positive 1st value, IIRC. I haven't looked at these things in a long time.

jseabold added a commit to jseabold/statsmodels that referenced this issue Mar 15, 2014

jseabold added a commit to jseabold/statsmodels that referenced this issue Mar 23, 2014

jseabold added a commit to jseabold/statsmodels that referenced this issue Mar 28, 2014

@jseabold jseabold closed this in de46fa9 Apr 4, 2014

PierreBdR pushed a commit to PierreBdR/statsmodels that referenced this issue Sep 2, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.