-
-
Notifications
You must be signed in to change notification settings - Fork 308
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generate pyarrow schema from pandera schema #689
Comments
@cristianmatache Any chance you made any headway on this? |
@justinlboyer not really, i recently changed jobs so i currently have a lot on my plate. Happy to guide you though, if you would be up for implementing it. |
@justinlboyer , did you ever take a look at this? This would be useful, though I'm assuming it would be limited in its implementation, i.e. using If a basic implementation is satisfactory (i.e. not able to handle complex types like the list example above), I'd be up for collaborating on this. |
@the-matt-morris I did not, we don't need it much anymore, but I'm happy to help out, feel free to ping me. |
hey @the-matt-morris the basic implementation would be a first good step! (i.e. support for primitive/scalar data types) This related to #260, support for things like |
@cosmicBboy , cool! Well I can take a stab at a PR on this...thinking would be a |
I'd consider this part of the My recommendation would be to implement a
Seems about right! |
Is this PR close to being merged? This is an excellent feature I would be keen to leverage! |
hi @louis-vines all current PRs are being blocked by #913, which involves a signifant re-write of the pandera internals. Once that's merged (hopefully within the next 2 weeks) we'll circle back to incorporate all the recent PRs, including this one. |
Excited for #913 ! Even once that is merged, I will need to go back and make a few updates to the PR anyways. I'd like to try out |
I see #913 is now merged (🥳). Any news on this one? Anything I could do to help? |
Also checking in on the status of this please. |
Checking in on the status. How can we further this along? |
Also excited for this feature! |
➕ Super stoked for this! |
Is your feature request related to a problem? Please describe.
Need to maintain the schema twice, once for the pandas dataframe and again for the pyarrow table. An example where we need both is writing partitioned parquet datasets.
Describe the solution you'd like
Generate pyarrow schema from pandera schema.
I plan to implement this over the Christmas holidays.
The text was updated successfully, but these errors were encountered: