Question re package naming (i.e. pydantic1 / pydantic2)

[tlambert03]

First off, thanks a million for all of your hard work on pydantic v1 and the exciting changes in v2. I'm thrilled by the prospects and have been diving in to begin migrating all of my pydantic-using packages. Congrats on all the progress so far 🏆

As I see just how big of an API change it is, i find myself wondering how it's going to affect environment-solving going forward, and I'm curious whether there's already been a discussion about having a differently named package for v1 and v2. I can definitely understand that you might not want to name the new version pydantic2, but seeing as it is mostly a different API, have you considered releasing a new pydantic1 package on pypi (something that would not receive any updates after the final pydantic v1 release)? (apologies if I've missed a pre-existing issue on this)

My primary concern is that there will be some packages that use pydantic for whom it's simply not worth the time to update, and they will likely leave pydantic<2 in their install requires and be done with it (at least for the foreseable future). Their package works fine for their needs and they're not looking to update. This of course works fine for applications who aren't likely to appear in an environment with other packages (e.g. django or fastAPI apps, etc...). But for libraries that try to coexist in python environment (where we obviously have a flat single-version package structure unlike node) I do fear a little bit the upcoming version clashes.

I do realize that the pydantic1 strategy still won't work for packages that don't change their install requirements to pydantic1 in their project metadata and go through and update all of their imports... but it still seems like a possible escape hatch that will enable coexistence of packages that might otherwise break each other.

Please don't get me wrong: it's super clear you've been very transparent and diligent about all the breaking API changes. Now is definitely the right time, and you're giving everyone plenty of time! Curious to hear your thoughts or prior discussion on this. thanks!

[tlambert03]

maybe to put this another way. Do you imagine that it will be possible for packages that use pydantic <v2 and packages that use ≥v2 to co-exist in the same environment? Or that that will be more or less impossible? And if the latter, do you have thoughts on how to "help" the ecosystem deal with this incompatibility? (such as a differently named package?)

[caniko]

This has been discussed before: #4649

[tlambert03]

Thank you for your response. I'm not sure I agree that's the same issue really. That issue was really more about a single large project incrementally updating itself. My point is about compatibility across multiple unrelated libraries with different authors: As soon as python_library_A updates itself and adds pydantic>=2.0 to its pip install requirements, it will break python_library_B (that uses pydantic<2.0) in the same environment.

To summarize points that were made there, you said:

It is impossible to install two separate versions of the same package in one environment; maybe possible with some dirty tricks (make a 2nd package with the old version). This seems more like a deployment issue, and not pydantic-related 🤔

I do, however, see this is a pydantic issue. Obviously, pydantic is a very widely used package these days. And while a major use case might be applications/projects that don't need to co-exist in an environment with other projects, another very valid use case is python libraries that use pydantic. As soon as pydantic v2 is released as a non-pre-release on pypi under the name pydantic, any environment that has an updated package (with pydantic>=2.0) and an un-updated package (with pydantic<2.0) will immediately break (I'm referring to the kind of concerns that were clarified nicely in "Should You Use Upper Bound Version Constraints? ). It's not a deployment issue per se (though I can see that an individual application could deal with this on the deployment side), it's an issue that every python package needs to consider when releasing changes that change the entire API.

I'm hoping that pydantic has a solution "ready" for libraries (not just applications) before releasing pydantic v2 under the name pydantic on PyPI. Releasing either the current version as pydantic1 or the new version as pydantic2 would solve this issue (as this is the best we can do in python to allow two separate versions of the same package in one environment). But just releasing pydantic v2 would leave a lot of communities with a massive headache

[jni]

There just is no way around it.

There is, that is precisely the argument: if pydantic were to release differently-named packages for each major version, packages could coexist that use either of those differently named packages. napari could continue to use pydantic1 while new plugins popped up using pydantic2, and our community could migrate our codebase gradually, without having a compatibility break.

Without this "escape hatch", popular upstream packages are indeed forced to support either 1 or 2, and thus limited to the subset of pydantic that is the same between 1 and 2, just like with Python 2->3.

[tlambert03]

This! 👆

[caniko]

Let's not use fallacies, every popular package will eventually be on V2, otherwise it goes against the definition of popular.

You could use the "escape hatch" argument for every major release in every ecosystem, and you would be fine for making it. You are free to make this request or fork and create pydantic_v1 on PyPi yourself. Nobody is stopping you.

[tlambert03]

Nobody is stopping you.

@samuelcolvin politely requested that we don't do that. so, let's not :)

[caniko]

Just don't name it so that it is confused with pydantic.

You can name it oldantic_v1, making sure the name is OK with Samuel is probably the best course forward.

[samuelcolvin]

The idea of pydantic_next is that you can get started with pydantic V2 while still using v1 else where in your codebase, and critically while packages like fastapi are still using pydantic v1.

This wouldn't work if we released a pydantic v1 package.

[tlambert03]

Thanks all for your comments. I do still feel like my point has been missed. Im not worried about my packages or my migration plan or strategy. I know I can use pinning strategies, and such and migrate when I desire (as long as no other package shows up in an environment with one of my libraries with an incompatible pin). As a developer on projects (like napari/napari) that have many plugins and complicated environment solving, my concern was more that pydantic will soon be causing unsolvable environments quite a lot. And that if it notices that possibility, it could do something to ameliorate it.

Anyway, I respect that you've got lots of stuff going on at the moment. Thanks for the conversation.

[tlambert03]

If you genuinely don't understand or agree with the problem, that is ok... but would you have any particular objections to us releasing our own pydantic_old on PyPI and conda-forge?

[caniko]

@tlambert03 please be nice, you are escalating. I deleted it because I figured I didn't have time to argue, bedtime ☺️

You have pydantic v1, the other package is locked to v2: You need to upgrade to v2.

You have v1 upstream has v1 or v2: You can use poetry to lock to v1.

There are multiple packages that use v1 and/or v2. There is no solution even if there was pydantic_old because those packages use a locked version of pydantic. Your best bet is to fork, and make an upgrade PR towards V2 for every V1 package if you can't wait. There just is no way around it.

@tlambert03 the scenario you are talking about is from the perspective of an enduser. You should focus on your side of things. I outlined the options available. You still don't use pydantic_next the right way in your scenario IMHO. I really don't see anyone using pydantic_next this way.

[tlambert03]

Sorry you felt I was escalating. I do appreciate the discussion, just kinda felt like we were going in circles a bit since poetry is a solution that an application with a one time deployment might use, but doesn't solve the library case. Hope my example clarified the case I'm talking about :)

[tlambert03]

You should focus on your side of things. I outlined the options available.

This is where we disagree I guess. Before starting this thread I spent a few days trying to make a couple packages support both pydantic v1 and v2. Seeing that it would be extremely hard if at all possible, I realized that there's very little I can do on my side of things to prevent myself from being broken from the scenario I describe above (and I'm afraid the options you outline above are insufficient as well, let me know if you disagree).

It's easy enough to say "don't think about it from the end user situation", but I feel that the pydantic_old scenario really does provide at least a partial solution to the problem that I haven't seen in any of your proposals. Many packages have done that sort of thing (and that's the reason it is done)

As always, thanks for your input!

[Czaki]

You have v1 upstream has v1 or v2: You can use poetry to lock to v1.

It is not a problem for ane experienced python user to lock to pydantic<2. One could use poetry, the * second one could use .env with proper variables. The problem happens when your package users are scientists/data scientists/data analysist etc and your app/library is extensible by users using plugins.

Who do not have deep Python knowledge and will open issues or request support in another way.

[tlambert03]

one last thing... I did find this comment from @samuelcolvin: #4649 (reply in thread)

Instead of pydantic_next, it might be better to flip it around and have pydantic_old to be explicit to encourage newer pydantic to be the default.

which got a number of +1s ... so it does sound like there had been at least a little internal support for this idea.

[samuelcolvin]

I'm very opposed to this on reflection, for reasons explained below #5402 (reply in thread) - this package would never be upgraded thus users would be more tempted to stick with using it indefinitely - long after it's no longer supported.

As mentioned below, I think it's a bad idea to release any other packages to PyPI, instead we should have instructions for installing pydantic_next from git.

[tlambert03]

thanks for your input! understood, and response below: #5402 (reply in thread)

[samuelcolvin]

The way this was solved with python 2 vs 3, and the way I think @tiangolo is intending to solve this with fastapi is to support both pydantic v1 and v2 for a while.

This might seem like lots of work, but actually it's probably as simple as copying your code and having two modules - one for V1 and one for V2.

[Czaki]

And if versions 1 and 2 live in different namespaces it will be simpler to write such a backward compatibility layer.

[samuelcolvin]

Honestly this whole conversation, and the amount of confusion, makes me want to avoid releasing any other packages.

I think releasing with pydantic1 or pydantic2 (or both) might lead to people thinking those are the correct ways to install pydantic, and with pydantic1/pydantic_old there's a temptation that since newer versions are never released, that they're still fine to use, long after we stop supporting them.

For this reason I think it would be a Bad idea for the community to release any other pydantic packages.

I think even we (the pydantic organisation) probably shouldn't add any other pckages to PyPI, instead we should suggest users install pydantic_next via git to make it as clear as possible that it's a temporary hack, not a package/solution you should use long term.

[tlambert03]

For this reason I think it would be a Bad idea for the community to release any other pydantic packages.

Thanks for weighing in @samuelcolvin ... I did very much want to hear your opinion on this, so I appreciate it. I'm absolutely dedicated to finding a solution for our ecosytem that you and the pydantic team are also happy with, and so wouldn't put out anything you weren't at least ok with on some level. Hear you loud and clear here 👍

We'll definitely continuing to work on this and think about it. I'll continue working on some code to see if I can create a solution for our stuff that I feel is robust enough to the kinds of problems we've been describing. Haven't quite heard anything that I feel like is a really satisfying approach, but haven't given up :)

thanks again

[caniko]

Signing of this thread, it has been a tremendous brain-exercise trying to emphasize the pain and suffering for those that are affected by this issue. I really wish there was a golden bullet for this issue, but I don't think there is... Not something that makes everybody happy!

I guess the silver-lining is that everyone here has a lot of opinions, and expertise that can be put to better use in their IDEs where they actually do the work to move over to the more safe and efficient version of Pydantic, which is V2. I know it is dissatisfying to be told you have to work a little extra for something you didn't perhaps ask for, but Pydantic is mostly used in FastAPI (correct me if I am wrong). We come second, I personally can't wait to use computed_fields.

Kind regards

[samuelcolvin]

Pydantic is mostly used in FastAPI (correct me if I am wrong)

That's not correct, only about 25% of pydantic's use is with fastapi - that's what the downloads suggest.

By my calculation, roughly another 25% of usage is with other libraries - then 50% of usage is direct, with no library.