is sm.tsa.filters.arfilter an AR filter? #1478

Closed
jseabold opened this Issue Mar 14, 2014 · 7 comments

Projects

None yet

2 participants

@jseabold
Member

In the terminology R uses in their filter function [1], what we have as arfilter is just the convolution method of their filter, which is just a centered moving average. Their recursive method is autoregressive and we would do the same using scipy.signal.lfilter. I was going to add a method argument to arfilter for convolution vs. recursive, but I thought to maybe just rename it to linear_filter first? Thoughts?

[1] http://stat.ethz.ch/R-manual/R-patched/library/stats/html/filter.html

@jseabold
Member

Or arfilter -> convolution_filter and arfilter uses signal.lfilter internally.

@josef-pkt
Member

In terms of general terminology.

I had big problems keeping apart which is which, especially also with the signal processing terminology in scipy.signal (num, denom), and I didn't read R at the time.

I tried to translate things into ARMA terminology, ar_filter, ma_filter for things that signal.lfilter can do, ARMA terminology is more mnemonic for me. (assuming I got it correct)

edit: I'm not sure about the following after 30 minutes:
I think the term arfilter is wrong, a moving average, convolution filter should be an mafilter.

assume y is observed

y_t for ar(L) y_t = e_t
versus
y_t = ma(L) e_t

ma(L) would be a FIR filter (finite impulse response) in scipy.signal
ar or arma is a IIR filter

I thought I had an arma_filter wrapper around lfilter, but cannot find anything like that.

ArmaFft in the sandbox has a frequency domain arma filter method.

one of the things I got confused about in signal processing terminology is transfer function filter, where x is the input and y is the output


linear_filter sounds fine,

related: for using lfilter we should have the correct choice of starting values, so we get identical results to recursion with fixed initial conditions.

@jseabold
Member

The one-sided convolution filter maps to MA using lfilter. These should give the same, but it doesn't look like lfilter properly handles NA values in this case. Looks like a bug to me.

signal.convolve(dta.co2.values, [.75, .25], mode='valid')
signal.lfilter([0, .75, .25], [1.], dta.co2.values)

I made a quick mapping between convolve, lfilter and R's filter methods, optionally using initial conditions. For the one-sided convolution, I might just do a one-linear using pd.rolling_apply since it handles NAs correctly.

@jseabold
Member

Sorry, no those shouldn't give the same. The first line should be

filter(dta$co2, c(.75, .25), sides=1)

in R.

@josef-pkt
Member

lfilter should have leading ones in both polynomials, and the sign is polynomial not regression

signal.lfilter([1, .75, .25], [1.], dta.co2.values)

@jseabold
Member

If you want a one-sided convolution filter, then the 0 lag for MA is 0. I.e., don't include the current term if you're doing a moving average that should only include past terms.

@josef-pkt
Member

ok, right for shifted to use only past information,
the ar part needs (or needed) to have a strictly positive 1st value, IIRC. I haven't looked at these things in a long time.

@jseabold jseabold added a commit to jseabold/statsmodels that referenced this issue Mar 15, 2014
@jseabold jseabold REF: Refactor arfilter. Correct name and split. Closes #1478. 1a1434e
@jseabold jseabold added a commit to jseabold/statsmodels that referenced this issue Mar 23, 2014
@jseabold jseabold REF: Refactor arfilter. Correct name and split. Closes #1478. 006a1e7
@jseabold jseabold added a commit to jseabold/statsmodels that referenced this issue Mar 28, 2014
@jseabold jseabold REF: Refactor arfilter. Correct name and split. Closes #1478. 86d5c69
@jseabold jseabold closed this in de46fa9 Apr 4, 2014
@PierreBdR PierreBdR pushed a commit to PierreBdR/statsmodels that referenced this issue Sep 2, 2014
@jseabold jseabold REF: Refactor arfilter. Correct name and split. Closes #1478. 2801165
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment