Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add imputation for missing values #3

Open
bashtage opened this issue Jul 6, 2020 · 5 comments
Open

Add imputation for missing values #3

bashtage opened this issue Jul 6, 2020 · 5 comments
Assignees
Milestone

Comments

@bashtage
Copy link
Contributor

bashtage commented Jul 6, 2020

Has missing value handling been implemented? If so, is there any way to configure what to do? I would like to examine moving linearmodels to formulaic since I would really like an extensible formula parser, which I couldn't do with patsy.

@matthewwardrop
Copy link
Owner

Hi @bashtage! Can you clarify what you mean by "missing values"? Do you mean imputation in the original data set? Or missing values in a second dataset that you are massaging to look like the first? Or just any case where the are nulls in the data?

@bashtage
Copy link
Contributor Author

bashtage commented Jul 7, 2020

patsy has an input called NA_action that lets you tell it what to do with NAs. I was trying to use formulas in linearmodels but noticed that I am strict with dropping and prefer to raise. I didn't see any obvious way to achieve the same result in formulaic.

@matthewwardrop
Copy link
Owner

Ah! After thinking about this a little, here is how I'd like to implement this.

I will add a new stateful transform called impute which allow you to do things like: a + impute(b, mean). It will keep track of its state, and passing future dataframes into the formula will remember which value to impute when data is missing. If after the columns are all evaluated (but before encoding) there are any null values, the materialiser will either: ignore the nulls, drop the nulls or raise, depending on the argument passed to the materialiser during its construction.

In full, then, you would something like:

from formulaic import model_matrix
df = ...
model_matrix('a + impute(b, mean)', df, na_action='drop'|'raise'|'ignore')

In this example, the na_action could only be triggered based on a (since b would have null values imputed).

Does that sound good?

@bashtage
Copy link
Contributor Author

bashtage commented Jul 7, 2020

That sounds like an excellent solution.

As a wish list item, it would be great if impute could have an interface so that it would be possible for end users to supply their own imputers. For example, statsmodels as MICE which could be wrapped if there was an interface to do sophisticated imputing, or someone might want to use a PCA-based computer, or one that uses some non-model data such as a regression projection.

@matthewwardrop
Copy link
Owner

I was imagining that the mean passed above is a python function that exists in the namespace (it could have been np.mean, etc). So that should cover all local imputations (imputations local to a column), or non-local imputations where the function already has the relevant context (via a closure or some such). Is that sufficient?

If not, what would a non-local imputation look like (as I imagine would be the case for PCA) expressed in a formula?

@matthewwardrop matthewwardrop changed the title Handling missing values Add imputation for missing values Oct 17, 2021
@matthewwardrop matthewwardrop self-assigned this Oct 17, 2021
@matthewwardrop matthewwardrop added this to the 0.4.x milestone Oct 17, 2021
@matthewwardrop matthewwardrop modified the milestones: 0.3.x, 0.4.x Jun 20, 2022
@matthewwardrop matthewwardrop modified the milestones: 0.4.x, 1.0.x Dec 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants