Add support for pydantic 2.0, polars 0.20.10 and remove duckdb support

[brendancooley]

builds upon #4, all existing tests (excepting tests/test_duckdb/*) passing after upgrade to pydantic==2.4.2, polars==0.19.11

Closes #11
Closes #28
Closes #26
...possibly closes others

Summary

• (breaking change) recover legacy ValidationError from pydantic v1 to allow for multiple error reporting, collect data frame validation errors in this object, rename as DataFrameValidationError to distinguish from errors raised by pydantic directly.
  • users will need to update exception catching on data frame validation
• subclass pydantic.fields.FieldInfo to accommodate patito-specific field attributes (constraints, derived_from, dtype, unique)
• update patito.Model._schema_properties to append patito-specific field attributes to the field property dictionaries to pydantic.BaseModel.model_json_schema
  • avoids serialization errors raised when passing patito-specific fields via json_schema_extra
• (breaking change?) separation of concepts of required and nullable. nullable now refers to columns that allow pl.Null as a valid entry. required (pydantic-side) refers to columns that do not have a default value, and therefore must be passed to the BaseModel constructor.
• fix polars.collect bug (see fix: update LDF.collect() for polars==0.19.8 #24)

TODO

• refactor directory structure for patito.pydantic

patito
+-- src
|  +-- pydantic
|  |  +-- main.py (`BaseModel`)
|  |  +-- fields.py (`FieldInfo`, `Field`)

• updates where necessary to documentation and readme

And I'm sure there are other issues introduced by these changes, and bugs that the existing test suite does not yet catch. But hopefully this serves as a template for discussion and new test collection and can help move the ball forward on getting this package onto pydantic2. Feedback very welcome.

[ion-elgreco]

I'll build this one locally and see if it will fix the other bugs Patito 0.50 still has

[ion-elgreco]

They were also working on v2 here: https://github.com/JakobGM/patito/tree/jakobgm%2Fpatito-v2

[brendancooley]

I'll build this one locally and see if it will fix the other bugs Patito 0.50 still has

Anxious to hear how it goes

[ion-elgreco]

@brendancooley nice work! The FieldInfo class nicely fixes the issue with properties being lost after any class method, I solved a bit a different with v1 #20. Perhaps you can grab my test from there and include it in your pr. I already checked the test also passes with your changes.

One thing that still will be broken is the examples creation, you need to include this fix #20 and then build on top of it since now a type which is Optional[str] will have the parameter minimum behind a list in the anyOf key.

I will run it tomorrow on our production repo to see if anything breaks : ) (possibly in the examples creation it will)

[brendancooley]

@brendancooley nice work! The FieldInfo class nicely fixes the issue with properties being lost after any class method, I solved a bit a different with v1 #20. Perhaps you can grab my test from there and include it in your pr. I already checked the test also passes with your changes.

One thing that still will be broken is the examples creation, you need to include this fix #20 and then build on top of it since now a type which is Optional[str] will have the parameter minimum behind a list in the anyOf key.

I will run it tomorrow on our production repo to see if anything breaks : ) (possibly in the examples creation it will)

happy to include the test. did you mean to reference a different issue/pr in your second comment? not clear on how #20 helps with example string generation but perhaps I am missing something.

[ion-elgreco]

@brendancooley nice work! The FieldInfo class nicely fixes the issue with properties being lost after any class method, I solved a bit a different with v1 #20. Perhaps you can grab my test from there and include it in your pr. I already checked the test also passes with your changes.

One thing that still will be broken is the examples creation, you need to include this fix #20 and then build on top of it since now a type which is Optional[str] will have the parameter minimum behind a list in the anyOf key.

I will run it tomorrow on our production repo to see if anything breaks : ) (possibly in the examples creation it will)

happy to include the test. did you mean to reference a different issue/pr in your second comment? not clear on how #20 helps with example string generation but perhaps I am missing something.

#20 is a fix regarding, creating proper examples with numerical values. I meant just taking that fix directly into pydantic v2 won't work. I saw that grabbing the minimum field is now nested if you have a field that is optional

[ion-elgreco]

@brendancooley tried it on our repo and tests, I am also getting this now static type hint error:

Expression of type "FieldInfo" cannot be assigned to declared type "int"
  "FieldInfo" is incompatible with "int"

on this code:

class Test(pt.Model):
    test: int= pt.Field(constraints=pl.struct("a", "b").is_unique())

[ion-elgreco]

Also nullable columns are not working, mre:

class Test(pt.Model):
    foo: str|None = pt.Field(dtype=pl.Utf8)
    
print(Test.nullable_columns)
set()

Result should be: {'foo'}

[brendancooley]

Thanks @ion-elgreco, I'll add these tests to the PR and start hacking at them tomorrow. Have not looked much into structs but it would be nice if we had more tests for those.

[ion-elgreco]

Thanks @ion-elgreco, I'll add these tests to the PR and start hacking at them tomorrow. Have not looked much into structs but it would be nice if we had more tests for those.

This example where I am using a struct is just a simple trick to do multiple column uniqueness as a constraint.

One thing that is missing though in Patito but probably out of scope for the migration is proper struct support as a dtype

[brendancooley]

Also nullable columns are not working, mre:

class Test(pt.Model):
    foo: str|None = pt.Field(dtype=pl.Utf8)
    
print(Test.nullable_columns)
set()

Result should be: {'foo'}

@ion-elgreco should be addressed by e2bf0d7

[brendancooley]

017c59b also adds some sanity checking that the annotated types match the allowable polars types specified in Field.dtype

[brendancooley]

@brendancooley tried it on our repo and tests, I am also getting this now static type hint error:

Expression of type "FieldInfo" cannot be assigned to declared type "int"
  "FieldInfo" is incompatible with "int"

on this code:

class Test(pt.Model):
    test: int= pt.Field(constraints=pl.struct("a", "b").is_unique())

yes I can replicate these in vscode if I enable

{
  "python.analysis.typeCheckingMode": "basic"
}

in settings.json. Looking into it.

[brendancooley]

@brendancooley tried it on our repo and tests, I am also getting this now static type hint error:

Expression of type "FieldInfo" cannot be assigned to declared type "int"
  "FieldInfo" is incompatible with "int"

on this code:

class Test(pt.Model):
    test: int= pt.Field(constraints=pl.struct("a", "b").is_unique())

This should be handled by 161300b

[brendancooley]

@brendancooley nice work! The FieldInfo class nicely fixes the issue with properties being lost after any class method, I solved a bit a different with v1 #20. Perhaps you can grab my test from there and include it in your pr. I already checked the test also passes with your changes.
One thing that still will be broken is the examples creation, you need to include this fix #20 and then build on top of it since now a type which is Optional[str] will have the parameter minimum behind a list in the anyOf key.
I will run it tomorrow on our production repo to see if anything breaks : ) (possibly in the examples creation it will)

happy to include the test. did you mean to reference a different issue/pr in your second comment? not clear on how #20 helps with example string generation but perhaps I am missing something.

#20 is a fix regarding, creating proper examples with numerical values. I meant just taking that fix directly into pydantic v2 won't work. I saw that grabbing the minimum field is now nested if you have a field that is optional

Yes this makes sense. Let me know if 9e132bc fixes the issue @ion-elgreco.

[ion-elgreco]

@brendancooley 🚀🚀, I'm going to try it out now

[ion-elgreco]

@brendancooley datetime doesn't work if the type doesn't contain a TZ with .examples():

This will fix it:

if dtype.time_zone is not None:
    tzinfo = ZoneInfo(dtype.time_zone)
else:
    tzinfo = None
return datetime(
    year=1970, month=1, day=1, tzinfo=tzinfo
)

[ion-elgreco]

Adding dtype in pt.Field(dtype=) breaks validation for datetimes:

class Test(pt.Model):
    date_value: datetime = pt.Field(dtype=pl.Datetime)

Test.validate(pl.DataFrame({"date_value": [datetime(2020,1,1)]}))

DataFrameValidationError: 1 validation error for Test
date_value
  Polars dtype Datetime(time_unit='us', time_zone=None) does not match model field type. (type=type_error.columndtype)

While this works:

class Test(pt.Model):
    date_value: datetime

Test.validate(pl.DataFrame({"date_value": [datetime(2020,1,1)]}))

[ion-elgreco]

GE constraint not respected while creating examples:

class Test(pt.Model):
    value: int = pt.Field(ge=0)
    
print(Test.examples())
shape: (1, 1)
┌───────┐
│ value │
│ ---   │
│ i64   │
╞═══════╡
│ -1    │
└───────┘

This should return 0 or larger :) 

Good to mention this only happens when ge=0, otherwise it works fine.

LOL, python evalautes 0.0 or None as None, while None or 0.0 is 0.0

@brendancooley this will fix it:

minimum = properties.get('minimum')
exclusive_minimum = properties.get("exclusiveMinimum")
maximum = properties.get('maximum')
exclusive_maximum = properties.get("exclusiveMaximum")            

lower = minimum if minimum is not None else exclusive_minimum
upper = maximum if maximum is not None else exclusive_maximum

[ion-elgreco]

So, besides these small bugs everything works as expected!

[brendancooley]

awesome @ion-elgreco, should be good to go on all of these now

[ion-elgreco]

@brendancooley awesome! Then I think it's in a state to release now☺️

[ion-elgreco]

Left a couple small comments, really nice work @brendancooley!

Hope we can get this merged soon @thomasaarholt :)

[brendancooley]

Left a couple small comments, really nice work @brendancooley!

Hope we can get this merged soon @thomasaarholt :)

Useful for knowing where to flesh out the docs -- thanks!

[thomasaarholt]

Given that the tests pass, I am keen to get this merged now, and then we can work on smaller parts separately.

Very well done to everyone involved 😊

[ion-elgreco]

Awesome! @thomasaarholt can't wait for the release :D

[thomasaarholt]

Doing it right now

[thomasaarholt]

Released now on github, should be on pypi shortly!

[thomasaarholt]

I see that the pypi release action failed, because everything has to pass. 😬 I'll see if I can tone down the requirements this evening.

[thomasaarholt]

We are live now with 0.6.1! https://pypi.org/project/patito/

[ion-elgreco]

Woohoo 🎉🎉

[brendancooley]

Awesome, thanks @thomasaarholt. Excited to do more with this, glad that we have a stable foundation to build on top of.