Plan to support pydantic 2.x versions

[sjsinghai]

When can we expect support for pydantic 2.x versions?

[cmgrote]

Hi @sjsinghai — this is something we'd investigated a couple months ago, but ran into severe performance issues with at that time (things that were sub-second suddenly took minutes). We'd also like to make the move, but not at the cost of slowing down the SDK by the amount we observed...

Curious: what particular features are you most looking forward to adopting or are reliant on with pydantic v2.x?

Perhaps even if we're still unable to move to pydantic 2.x we can look at some other way of unlocking those needs.

[sjsinghai]

That is a bit surpirsing given that the main thrust of pydantic 2.x is speed up offered by the improved backend.

In fact it is because of the promised speed-up that we are investigation move of our stack to 2.x.

Have you considered using the pydantic.v1 that ships with 2.x e.g. as documented here?
https://docs.pydantic.dev/latest/migration/#continue-using-pydantic-v1-features.

This should

• unblock others from using 2.x
• should have minimal impact to your existing performance.

[cmgrote]

Here's the thread over on pydantic's repo — seems at least one other adopter has faced similar challenges: pydantic/pydantic#7263

[tdebroc]

Agree we need that as well. I sent a mail to Atlan support. The migration is pretty smooth to do, we did in our repo

[cmgrote]

We're looking at this again — we definitely want to get there, too!

[cmgrote]

We're making more progress on this now than we had previously, but still some work to do.

However, to give an update on our direction:

• We will be using the pydantic.v1 V1 API (or "compatibility layer", as I call it) available in Pyandatic 2.x.
• In this way we will remove the dependency on the Pydantic 1.x libraries.
• This should allow folks to use "pure" Pydantic 2.x if-desired in their own code, without any conflicting libraries.

A bit more context:

We had hoped to move to "pure" Pydantic V2 ourselves, but the performance hit is simply unacceptable. We have proven this by:

1. Starting with a subset of our (complex) data model (~30 out of 100+ classes).
2. Modeling out the true inheritance and inter-relationships within that subset.
3. Using the latest released version of Pydantic 2.

First we tested with importing straight from pydantic (so V2), and creating instances of all the various classes in that subset of the model. We updated our code to use the model_rebuild() method only where necessary for the inter-relationships to resolve and be able to successfully instantiate all the classes. (Rationale: running the model_rebuild() against every class made the imports alone take significantly longer.)

Result: 35 seconds on average to go through the imports, model rebuilds, and class instantiations. ❌

Next, we took exactly the same code but replaced the import straight from pydantic with an import from pydantic.v1, to use the V1 APIs (compatibility layer). We also replaced the model_rebuild() calls with update_forward_refs() calls, and then re-ran the same code to instantiate all the classes.

Result: 0.8 seconds on average to go through the imports, forward ref updates, and class instantiations. ✅

Given this is only a subset of our model, we expect the performance degradation of "pure" V2 to be even worse with the full complexity of our model (likely growing from 35x slower to close to 100x slower or more), and thus will stick to the V1 API for the foreseeable future, though we do intend to update to v2 of the library itself.

[rajivramanathan]

Yes. That sounds like a right approach without having to swallow v2 in one shot. And will help us a lot in the short term @cmgrote.