Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement a new NaN handling strategy for pspm_prepdata #35

Closed
eozd opened this issue Oct 3, 2019 · 2 comments · Fixed by #435
Closed

Implement a new NaN handling strategy for pspm_prepdata #35

eozd opened this issue Oct 3, 2019 · 2 comments · Fixed by #435
Assignees
Labels
New Feature New feature

Comments

@eozd
Copy link
Contributor

eozd commented Oct 3, 2019

Feature Description

As described in #34, pspm_prepdata cannot gracefully handle signals containing NaNs. We need to implement a new filtering strategy that can account for this possibility.

@eozd eozd added New Feature New feature TODO labels Oct 3, 2019
@eozd eozd added this to the v4.3.0 milestone Oct 3, 2019
@eozd
Copy link
Contributor Author

eozd commented Oct 3, 2019

Below is a list of possible ideas to implement this feature:

  1. Perform linear (cubic, spline) interpolation for NaN segments. Then, filter the data. Finally, return the samples from the filtered signal that correspond to valid samples in the input signal.
  2. Split the signal on NaN segments, and filter the resulting signal segments separately. Then, construct the output signal using these filtered signal segments and the original NaN segments.
  3. Combine the above two approaches: if NaN segments are very long (to be defined), go for option 2; otherwise, go for option 1.

@eozd
Copy link
Contributor Author

eozd commented Oct 8, 2019

After the latest discussions we decided on the following strategy when the input signal contains NaNs:

  1. Issue a warning about NaN values and the strategy that will be applied to solve it.
  2. We determine a cutoff number of samples either dynamically or statically. (to be decided)
  3. By default, the function applies spline interpolation to fill the missing values before filtering. However, if there is a NaN segment whose length is greater than the cutoff value, a separate warning will be issued.
  4. User can choose to split the signal on NaN segments to create some number of signal segments that contain no NaNs. Then, each signal segment is filtered separately. In this case, if the length of any signal segment is less than the cutoff value, a separate warning will be issued.
  5. If downsampling is enabled, both filtered signal(s) and NaN segments are downsampled.
  6. After applying either step 2 or step 3, the function constructs the output signal by combining filtered signal(s) with corresponding NaN values.

@irojkov-ph irojkov-ph modified the milestones: v4.3.0, v4.4.0 Apr 1, 2020
@irojkov-ph irojkov-ph modified the milestones: v4.4.0, v5.1.0 Sep 9, 2020
@teddychao teddychao removed this from the v5.1.0 milestone Jun 7, 2021
@teddychao teddychao removed the TODO label Jun 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
New Feature New feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants