Proper way to create a FDataGrid object of curves with various lengths?

[hovinh]

Hi scikit-fda team,

I have checked the doc and it seems the underlying assumption when creating FDataGrid object is that data_matrix must have the same shape. I have a data set, whose half of them having approx 50 points and the other half approx 70 points. In order to apply group-wise alignment, I would like to create a single FDataGrid object. How could I do that?

Attempt 1: create data_matrix with different length throws error, even when I don't clarify grid_points.
Code:

import skfda

grid_points = [[0, 0.2, 0.5, 0.9, 1], [0, 0.2, 0.5, 0.9]]
data_matrix = np.array([
    [0, 0.2, 0.5, 0.9, 1],     # First observation
    [0, 0.04, 0.25, 0.81],  # Second observation
])

fd = skfda.FDataGrid(
    data_matrix=data_matrix,
    grid_points=grid_points,
)

Output:

 /opt/conda/envs/fda_py37/lib/python3.7/site-packages/ipykernel_launcher.py:5: VisibleDeprecationWarning:Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-52-720e51573dae> in <module>
      8 fd = skfda.FDataGrid(
      9     data_matrix=data_matrix,
---> 10     grid_points=grid_points,
     11 )
     12 

/opt/conda/envs/fda_py37/lib/python3.7/site-packages/skfda/representation/grid.py in __init__(self, data_matrix, grid_points, sample_points, domain_range, dataset_label, dataset_name, argument_names, coordinate_names, sample_names, axes_labels, extrapolation, interpolation)
    159             grid_points = sample_points
    160 
--> 161         self.data_matrix = _int_to_real(np.atleast_2d(data_matrix))
    162 
    163         if grid_points is None:

/opt/conda/envs/fda_py37/lib/python3.7/site-packages/skfda/_utils/_utils.py in _int_to_real(array)
    654 def _int_to_real(array: np.ndarray) -> np.ndarray:
    655     """Convert integer arrays to floating point."""
--> 656     return array + 0.0
    657 
    658 

TypeError: can only concatenate list (not "float") to list

Attempt 2: Create a FDataGrid for each trace, then l need a function to merge them. Do you have such thing?

Attempt 3: Resample each trace to a median length. This is doable but I'm afraid certain information could be lost that potentially affect the warping result.

Could you please advise on what is the proper way to do it?

Thank you,
Vinh

[vnmabus]

What do you mean by different lengths? As far as I know, that could mean two different things:

• "Partial" curves where some fragments of the curve are missing. This could probably be represented using NaN for missing values, but you should expect most methods to fail, as that case (currently) has not been taken into consideration.
• Curves in the same interval but sampled at different points, such as longitudinal data. We should create a new FData subclass to handle this kind of data, but currently we are not prioritizing that. A possible workaround is to express your data in a common grid (by interpolation) or project it to a common basis. Once you have both with the exact same representation you can use skfda.concatenate to join them. Issue Allow BasisSmoother to accept irregularly sampled data #325 discuss a possible future way to implement that as part of a pipeline.

Which one is your case?

[vnmabus]

Landmark registration has nothing to do with (Fisher Rao) elastic registration. It is only "elastic" in the sense that it does not only use rigid transformations, but allows time deformations too. It construct a warping by placing the corresponding landmark points at the same position in system time and interpolating the values between them.

[hovinh]

Thank you :))

[hovinh]

Hi,

I would like to resume this conversation. I have tried out pairwise alignment and notice that the input trace can have different length from the template. After transformation, the trace takes the original length, not template length. Not sure if this is a bug or not?

Example 1:

import matplotlib.colors as clr
import matplotlib.pyplot as plt
import numpy as np
import skfda

from skfda.preprocessing.registration import ElasticRegistration, invert_warping
from skfda.datasets import make_multimodal_samples

# Samples with modes in 1/3 and 2/3
fd = make_multimodal_samples(n_samples=2, modes_location=[1 / 3, 2 / 3],
                             random_state=1, start=0, mode_std=.01)

fig = fd.plot()
fig.axes[0].legend(['$f$', '$g$'])
plt.show()

def crop_fd_data(fd, size):
    data_matrix = fd.data_matrix[:, :size, :]
    grid_points = fd.grid_points[0][:size]
    return skfda.FDataGrid(
        data_matrix=data_matrix,
        grid_points=grid_points
    )

f = crop_fd_data(fd[0], 60) 
g = fd[1]
elastic_registration = ElasticRegistration(template=g)


# Aligns f to g
f_align = elastic_registration.fit_transform(f)

fig = f.plot()
fig = g.plot(fig=fig)
f_align.plot(fig=fig, color='r', linestyle='--')

# Legend
fig.axes[0].legend(['$f$', '$g$', '$f \\circ \\gamma $'])
plt.show()

Output:

Example 2:

def crop_fd_data(fd, size):
    data_matrix = fd.data_matrix[:, :size, :]
    grid_points = fd.grid_points[0][:size]
    return skfda.FDataGrid(
        data_matrix=data_matrix,
        grid_points=grid_points
    )

f = fd[0] 
g = crop_fd_data(fd[1], 60)
elastic_registration = ElasticRegistration(template=g)


# Aligns f to g
f_align = elastic_registration.fit_transform(f)

fig = f.plot()
fig = g.plot(fig=fig)
f_align.plot(fig=fig, color='r', linestyle='--')

# Legend
fig.axes[0].legend(['$f$', '$g$', '$f \\circ \\gamma $'])
plt.show()

Output:

[vnmabus]

I don't think that it ever makes sense to pass a template with different grid points than the curves. I think we should check for that in fit and predict, to prevent confusion.

[jdtuck]

I agree with that. It goes against the theory I the package. Partial curve matching is much harder and more complex problem.