What is the proper basis to be selected for this data example?

[hovinh]

Hi scikit-fda team,

I have a time-series dataset with 50 traces and approximate 330 data points per trace.

The mean trace looks like this:

I would like to find a good basis to represent them, so I try out with BSpline, Fourier, and Monomial and the number of basis varying from 5 to 350. Among them, I see that BSpline and Fourier have comparable result at n_basis=200, whereas Monomial has no acceptable parameter at all.
Question:

• Given acceptable n_basis is more than half of data point in discrete form, do you think this is a good choice of basis, or my dataset is in ill-posed representation that should have further transformation in prior?
• Do you have any rule of thump on the number of n_basis? I only know that BSpline is good for most curves, unless periodic then Fourier is prefered.

BSplines

Fourier

Thank you,
Vinh

P.S.: really appreciate your work on this wonderful package :D

[vnmabus]

As this is not an issue of the package, but instead a usage question, I am moving it to Discussions.

Your data is very atypical for Functional Data Analysis. Usually data in FDA is continuous and may take a continuum of values, while your data seems to take only two values (as far as I can see, the jump from one to the other is instantaneous, between measures). My first approach would be to use multivariate tools encoding each signal as a sequence of ones and zeros. Alternatively if you know that the data has always the same number of peaks you can "compress" your data as a vector of positions and durations of the peaks.

If you really like to use functional tools, for example for alignment, I would use the discretized representation or maybe a wavelet basis, such as the haar basis. This basis is not currently implemented, but you can add custom bases easily by just subclassing the Basis class and implementing the _evaluate method (see for example how it is done in Monomial).

[hovinh]

Thank you for your answer. Could you please help with these follow-up questions?

• My data is not binary and there is a slight curve at the peak. So I guess "a continuum of values" implies a gradual change in value, not so sudden as in my case?
• What intrigues me is, BSpline is unable to fit the curve, given that the space between adjacent knots are covered by an independent polynomial curve, which I think it's not difficult to dealt with in my data, say, each curve is either linear (to fit straight line) or quadratic function (to fit parabolic curve)?
• The Haar basis is a really good suggestion I think. I will give it a try.

Cheers,
Vinh

[hovinh]

Also, if I develop a basis selection procedure as below, could you please review if it's done properly?

basis_set = [BSpline, Fourier, Monomial]
n_basis_set = [5, 10, 20, 50]
opt_recon_error = Inf
opt_basis = None

for basis in basis_set:
    for n_basis in n_basis_set:
        basis_data = data.to_basis(basis, n_basis)
        recon_data = basis_data.to_grid()
        recon_error = abs(basis_data - recon_data)

        if (recon_error < opt_recon_error):
            opt_recon_error = recon_error 
            opt_basis = (basis, n_basis)

return opt_basis

[vnmabus]

* My data is not binary and there is a slight curve at the peak. So I guess "a continuum of values" implies a gradual change in value, not so sudden as in my case?

I thought your data was binary. Even if it really isn't, maybe treating it like binary could obtain better results. FDA can still be applied, but it is better suited for continuous data, although a small number of discontinuities is acceptable.

* What intrigues me is, `BSpline` is unable to fit the curve, given that the space between adjacent knots are covered by an independent polynomial curve, which I think it's not difficult to dealt with in my data, say, each curve is either linear (to fit straight line) or quadratic function (to fit parabolic curve)?

AFAIK, all the basis elements in a B-spline basis have the same degree, so they can't fit a line properly (order 2 basis splines are NOT a subset of order 3 basis splines). You could try putting more knots in certain areas to allow for more variation, but I doubt that is a good idea unless you have expert information.

* The Haar basis is a really good suggestion I think. I will give it a try.

Try that and tell me what happens! If you have any doubt in the basis creation process please tell me.

[vnmabus]

Also, if I develop a basis selection procedure as below, could you please review if it's done properly?

basis_set = [BSpline, Fourier, Monomial]
n_basis_set = [5, 10, 20, 50]
opt_recon_error = Inf
opt_basis = None

for basis in basis_set:
    for n_basis in n_basis_set:
        basis_data = data.to_basis(basis, n_basis)
        recon_data = basis_data.to_grid()
        recon_error = abs(basis_data - recon_data)

        if (recon_error < opt_recon_error):
            opt_recon_error = recon_error 
            opt_basis = (basis, n_basis)

return opt_basis

I think you are comparing pears with oranges here. You probably should compare data and recon_data. You can try using a functional metric in misc.metrics, such as L1 or L2, in order to take all the points into account.

Also, in order to convert to basis you can use the class BasisSmoother. The advantage is that you can choose to penalize the result of applying a linear operator over your data. In your case, maybe it would be useful to penalize the first derivative, as your data is flat almost everywhere.

In addition, BasisSmoother is a scikit-learn-like transformer, so with a little bit of effort you could apply the validation methods in scikit-learn, or maybe the methods in preprocessing.smoothing.validation.

[hovinh]

Hi vnmabus,

all the basis elements in a B-spline basis have the same degree, so they can't fit a line properly (order 2 basis splines are NOT a subset of order 3 basis splines).

This comment is useful. As I tried B-spline (order = 2), this implies a straight line in each interval defined by 2 adjacent knots, and varying the number of basis, I think I have attained a curve that fits nicely with my discretized data (the figure below shows that they are perfectly overlapped). As I think Haar basis cannot fit better, so I didn't implement it.