-
Notifications
You must be signed in to change notification settings - Fork 2.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ENH: Replace patsy with formulaic #6858
Comments
I don't use these features much, but I think it's great that there's a new project that might push formulas forward a bit more. |
It would be great to have an alternative to patsy especially one that avoids |
@josef-pkt This Stack Overflow Post describes pickling as
What is the purpose of pickling a formula in the case of statsmodels? It seems similar to how custom Javscript objects can simply be represented in JSON. Is that a fair comparison? |
Pickling is the simplest way to store python objects in a file or to transmit python objects between threads or processes. The main use case for statsmodels users is to store a model so it can later be used for prediction. This works with arrays and dataframes but not when patsy formulas are used. As a workaround we are rebuilding the design matrix, exog, from the formula. Prediction, when formulas are used, needs to go through patsy to create the design matrix for the prediction exog from data in the original format. |
Why can't you simply write a formula out as a string? Is a formula in this sense something along the lines of @josef-pkt What did you mean by this:
Why are Wilkinson formulas preferred? Is the alternative to use features as |
It's not the formula itself that cannot be pickled, It's all the information associate with the formula information about how the design matrix was built, and how new design matrices with the same structure can be built.
I never really looked at the details. The main point is that formulas are a short hand notation for defining a model. All other options require the user to specify a lot more explicitly. Statsmodels uses a more limited set of formula features compared to R, e.g. our Mixed Models don't put the random effects specification inside the formula as R does, and we don't use multi-part formulas. |
So are patsy/formulaic libraries that translate formulas into instructions for building design matrices?
Is this a choice made by statsmodels or is it due to limitations of patsy? |
Yes, and build the design matrix, and provide information about the design matrix that we can use for example for hypothesis tests.
Mostly our choice. Beyond the basic formulas, formulas start to become cryptic. Being more explicit in those cases, makes it easier to understand if you are not a formula expert. Partially it's also that we have more control over parts that don't go through patsy. Especially because we don't have a maintainer anymore that also contributes to patsy (or whose work on patsy is getting merged). |
Gotcha. @josef-pkt @bashtage You both have been very helpful to me, so thank you guys. |
In linearmodels I use some custom specifications that go beyond Patsy. I manually parse these. It would be nice to ultimately have a more integrated approach. For example, in instrumental variable models, which are estimated using 2 regressions, I use a formula like
where the In panel models I use keywords like
where |
@bashtage Why is linearmodels seperate from statsmodels? Is linearmodels more specifically for finance? |
statsmodels base structure is too rigid to reliably accommodate data structures that do not resemble a single series. |
What is a data structure that doesn't resemble a single series in this case? Grouped data? Time series? Multiple time series? |
I never do this, I always use post-mortem debugger, mainly with pytest |
I think we should add an experimental A prime candidate where it would go beyond patsy, would be sparse support. Based on the popularity by users, it would be important to support it in |
It would be easier to add |
Has adding support for formulaic been considered, recently? |
considered yes, but no work to support it has been done yet. It's still on the wishlist. |
Thank you, Josef, for the information!
From: Josef Perktold ***@***.***>
Sent: Dienstag, 3. Januar 2023 03:16
To: statsmodels/statsmodels ***@***.***>
Cc: Rostock, Claas ***@***.***>; Comment ***@***.***>
Subject: Re: [statsmodels/statsmodels] ENH: Replace patsy with formulaic (#6858)
considered yes, but no work to support it has been done yet. It's still on the wishlist.
-
Reply to this email directly, view it on GitHub<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fstatsmodels%2Fstatsmodels%2Fissues%2F6858%23issuecomment-1369329236&data=05%7C01%7Cclaas.rostock%40dnv.com%7C5ec71d8248564b25fb8408daed307885%7Cadf10e2bb6e941d6be2fc12bb566019c%7C0%7C0%7C638083089681722573%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=bIxlgSa0uIfX0jV1Jw2u2DwtriQAhaB8%2B8IfgQuebBk%3D&reserved=0>, or unsubscribe<https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FALT6QOC4ASZN5HLQXDAHOZTWQODWLANCNFSM4OR2INJQ&data=05%7C01%7Cclaas.rostock%40dnv.com%7C5ec71d8248564b25fb8408daed307885%7Cadf10e2bb6e941d6be2fc12bb566019c%7C0%7C0%7C638083089681722573%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=ksQOaOwwRsQEB55qYwQbkchfsrI5sNchVn%2FRr6Kfu8s%3D&reserved=0>.
You are receiving this because you commented.Message ID: ***@***.******@***.***>>
DNV GL SE, Registered Office Hamburg No. HRB 115442 * Chairman of the Supervisory Board: Remi Eriksen * Executive Board: Knut ?rbeck-Nilssen * Torsten Schramm * Bj?rn-Olaf Borth
The latest edition of the General Terms and Conditions of DNV GL SE is applicable.
**************************************************************************************
This e-mail and any attachments thereto may contain confidential information and/or information protected by intellectual property rights for the exclusive attention of the intended addressees named above. If you have received this transmission in error, please immediately notify the sender by return e-mail and delete this message and its attachments. Unauthorized use, copying or further full or partial distribution of this e-mail or its contents is prohibited.
**************************************************************************************
|
I'm looking at formulaic again, but it's difficult to see what is supported and how to get the information for post-estimation. @bashtage One of the main open problems we have is to get the formula information to construct margins (get_margeff) for interaction effects (discrete-continuous, continuous-continuous) and multicolumn transformation (like polynomials). The two current post-estimation usages of patsy's formula info (beside predict) is anova_lm (which I'm not very familiar with) and wald tests, those based on terms and the pairwise tests for categorical ( |
Adding here just in case. patsy issue 191 Please ignore if not relevant. Relates to statsmodels running on tests with Python 11. |
Hi @josef-pkt ! I'm the author of Formulaic, and am just now wrapping up the final bits before v1.0.0 in Formulaic. What was mostly missing hitherto was documentation, which is now mostly complete. I'd be more than happy to help port statsmodels over to formulaic, including adding any missing features; and can shoulder most of the load if that is desired. Re: re-use of the spec used for model matrices, please refer to the docs here: https://matthewwardrop.github.io/formulaic/guides/model_specs/#reusing-model-specifications . There's also a brief migration guide here: https://matthewwardrop.github.io/formulaic/migration/ . The ordering differences mentioned here are about to be mitigated and/or completely disappear. Also, thanks for posting the patsy issue @karrmagadgeteer2 . I'll take a look and get it patched. [Update: This is already fixed in 0.5.3, which I released a while back; please upgrade!] |
According to patsy's github https://github.com/pydata/patsy, it is no longer under active development. They are recommending people use formulaic. |
I think for v0.15.0 we should have formulaic in place. Whether it is mandatory or we provide a period with both patsy and formulaic needs consideration. |
The new documentation of formulaic looks very helpful. pull requests would be very welcome AFAIR, the main current usage of formula are
I think we will not be able to drop patsy anytime soon, both for backwards compatibility and for missing features in formulaic like splines. for the future:
The derivative of the formula transform from original data to design/model matrix would be very useful, but difficult to get. (*) e.g. #7609 |
Thanks for your reply, and for calling out the various interesting pieces. I'll set about putting up a PR that adds optional support for I've left a few notes about various things inline below. When we get closer / during the PR I may have other questions/comments, but we can deal with that then.
Formulaic also implements linear constraint matrices, though this has so far evaded the documentation. This was initially requested by
This is customizable in formulaic, you can optionally drop nan rows (the default), raise exceptions, or ignore and propagate them. Will have to see if those modes are sufficient.
Appending a
Is the set of required functionality from patsy unit tested? Formulaic is highly compatible with
This is technically straightforward if a bit of a chore, but we could definitely add some helper methods to simplify things.
This should be easier in Formulaic since the formula is a first class entity that is designed to be mutated. But... we shall see if it caters to your use-cases.
This is about to be added. Turned out to be pretty straightforward.
Interesting. I also added to formulaic symbolic differentiation of model matrices using sympy. I also intended to add finite differencing support... but since then I have become less convinced of the approach and utility (internally at work we marginalise over the delta of model matrices explicitly, which works fine if being slightly less computationally efficient than computing the categorical delta during factor evaluation). I'd be curious to see if either or both of these would be helpful to
Definitely curious about this one. I have in mind to add support for transforms that extended terms and/or formulae with metadata that can be used during modelling... but I will dig into this use-cases a bit more before over-promising! It's definitely a possibility! |
comment to just one item
The problem for this in patsy is that if there are categorical variables, then -1 adds the implicit intercept to the categorical encoding, e.g. does not drop any reference level. |
Ah... yes, Formulaic will do that too by default. Will see what can be done when we get to that chunk of code :). |
one more issue that I had forgotten
I guess it will be a lot of work to fully switch to formulaic. And then we can start using features that formulaic has but patsy does not, like supporting sparse design matrices. |
Just putting it out here in case it is useful: we run into a similar issue in a slightly different context. Even when an intercept is included in the model, we did not want to create a dense all-ones column for it, as the downstream library handles intercepts internally. Our solution was to create an option that adds an empty intercept term. Then full-rankness resolution works as if the intercept was there, but the model matrix does not contain one. |
At this point, Patsy seems to be broken with Python 3.12. |
Hi all! Circling back around to this now that Formulaic 1.0.0 has been published. Hopefully I can put up an initial PR soon. |
Formulaic is a rewrite of patsy that has been designed for more extensibility. It already has features that patsy doesn't such as sparse array support. Should consider moving to this when it is a bit more mature.
https://github.com/matthewwardrop/formulaic
The text was updated successfully, but these errors were encountered: