Update to pydantic v2

[dragosmc]

Is your feature request related to a problem? Please describe.
Pydantic v2 has been out for a few months now and it brings new cool features as well as big performance improvements due to its use of Rust for validation. Currently Ludwig requires Pydantic < 2.0.

The conflict is caused by:
    The user requested pydantic==2.4.2
    ludwig 0.8.4 depends on pydantic<2.0

Are there any plans to update to the new version?

I might be able to help with this.

[justinxzhao]

Hi @dragosmc,

We had to pin pydantic to <2.0 in #3537 because it was incompatible with transformers.

If we could get your help to check if transformers is compatible, and then releasing the pin on Ludwig, that would be great!

[dragosmc]

I see. In that case I guess we'll wait for them to update (or one of their dependencies) as they are still forcing pydantic<2 at the moment. Thanks.

[dragosmc]

Hi, I'm reopening this as transformers have updated their code to use pydantic2 huggingface/transformers#27933 and it feels like ludwig could remove the version restriction.