Pydantic v2 migration issue

[joostlek]

The problem

Pydantic released v2 of their library. Some libraries want to utilize the new features of Pydantic v2 so HA should update to Pydantic v2. To not burden all libraries to rework, Pydantic is providing v1 in a separate package of v2. This has been implemented in several libraries already made by some HA community members (https://github.com/AngellusMortis/pyunifiprotect/pull/297). This way the library supports both v1 and v2, this way of working is preferred to not burden the v2 upgrade with a lot of libraries to update at once.

After installing all dependencies and using pipdeptree we see that the current libraries use pydantic.

pydantic==1.10.12
├── aiolivisi==0.0.19 [requires: pydantic]
├── aionotion==2023.5.5 [requires: pydantic>=1.10.7,<2.0.0]
├── aioopenexchangerates==0.4.0 [requires: pydantic>=1.9,<2.0]
├── aiopurpleair==2022.12.1 [requires: pydantic>=1.10.2,<2.0.0]
├── aiowaqi==0.2.1 [requires: pydantic>=1.10.8]
├── demetriek==0.4.0 [requires: pydantic>=1.9.0,<2.0.0]
├── elgato==4.0.1 [requires: pydantic>=1.8.0,<2.0.0]
├── gcal-sync==4.1.4 [requires: pydantic>=1.9.0,<2.0a]
├── google-nest-sdm==2.2.5 [requires: pydantic>=1.10.4]
├── ical==5.0.1 [requires: pydantic>=1.9.1]
│   ├── gcal-sync==4.1.4 [requires: ical>=4.2.5]
│   └── pyrainbird==4.0.0 [requires: ical>=4.2.9]
├── inflect==6.0.4 [requires: pydantic>=1.9.1]
│   ├── jaraco.itertools==6.2.1 [requires: inflect]
│   │   └── jaraco.abode==3.3.0 [requires: jaraco.itertools]
│   └── jaraco.text==3.11.1 [requires: inflect]
│       ├── jaraco.collections==4.2.0 [requires: jaraco.text]
│       │   ├── jaraco.abode==3.3.0 [requires: jaraco.collections]
│       │   ├── jaraco.email==3.1.0 [requires: jaraco.collections]
│       │   │   └── jaraco.net==9.3.1 [requires: jaraco.email]
│       │   │       └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
│       │   └── jaraco.net==9.3.1 [requires: jaraco.collections]
│       │       └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
│       ├── jaraco.email==3.1.0 [requires: jaraco.text>=1.3]
│       │   └── jaraco.net==9.3.1 [requires: jaraco.email]
│       │       └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
│       └── jaraco.net==9.3.1 [requires: jaraco.text]
│           └── jaraco.abode==3.3.0 [requires: jaraco.net>=9]
├── intellifire4py==2.2.2 [requires: pydantic]
├── lacrosse-view==1.0.1 [requires: pydantic>=1.9.0]
├── open-meteo==0.2.1 [requires: pydantic>=1.8.0,<2.0.0]
├── peco==0.0.29 [requires: pydantic>=1.9.0]
├── pvo==1.0.0 [requires: pydantic>=1.8.0,<2.0.0]
├── pyaussiebb==0.0.15 [requires: pydantic>=1.9.0,<2.0.0]
├── pyrainbird==4.0.0 [requires: pydantic>=1.10.4]
├── python-bsblan==0.5.11 [requires: pydantic>=1.9.0]
├── python-kasa==0.5.3 [requires: pydantic>=1,<2]
├── python-opensky==0.2.0 [requires: pydantic>=1.10.8]
├── pytraccar==1.0.0 [requires: pydantic>=1,<2]
├── pyunifiprotect==4.10.6 [requires: pydantic!=1.9.1]
├── radios==0.1.1 [requires: pydantic>=1.9,<2.0]
├── sfrbox-api==0.0.6 [requires: pydantic>=1.10.2]
├── systembridgeconnector==3.4.9 [requires: pydantic>=1.9.0]
├── tailscale==0.2.0 [requires: pydantic>=1.8.0,<2.0.0]
├── vehicle==1.0.1 [requires: pydantic>=1.8.0,<2.0.0]
├── withings-api==2.4.0 [requires: pydantic>=1.7.2,<2.0.0]
├── xbox-webapi==2.0.11 [requires: pydantic]
├── yolink-api==0.3.0 [requires: pydantic>=1.9.0]
├── youtubeaio==1.1.5 [requires: pydantic>=1.10.8]
└── zwave-js-server-python==0.50.1 [requires: pydantic>=1.10.0]

This issue will be the progress tracker of this migration.

❌ aiolivisi

Library: [aiolivisi](https://github.com/StefanIacobLivisi/aiolivisi) Integrations using this library: [Livisi](https://github.com/home-assistant/core/tree/dev/homeassistant/components/livisi) Code owners: @StefanIacobLivisi @planbnet

• Supports Pydantic v2 (PR: Support for pydantic v2 using v1 shims StefanIacobLivisi/aiolivisi#12)
• Is updated in HA (PR: # )

Remarks:

✔️ aionotion

Library: aionotion
Integrations using this library: Notion
Code owners: @bachya

• Supports Pydantic v2 (PR: Migrate to Pydantic 2.0 standards bachya/aionotion#251)
• Is updated in HA (PR: Update aionotion to 2024.02.2 #111388 )

Remarks:
There seems to be no version that supports both v1 and v2, so this library should be directly bumped with the v2 update.

❌ aioopenexchangerates

--
Library: aioopenexchangerates
Integrations using this library: OpenExchangeRates
Code owners: @MartinHjelmare

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: # )

Remarks:

❌ aiopurpleair

--
Library: aiopurpleair
Integrations using this library: PurpleAir
Code owners: @bachya

• Supports Pydantic v2 (PR: Migrate to Pydantic 2.0 standards bachya/aiopurpleair#195)
• Is updated in HA (PR: # )

Remarks:
There seems to be no version that supports both v1 and v2, so this library should be directly bumped with the v2 update.

✔️ aiowaqi

--
Library: aiowaqi
Integrations using this library: WAQI
Code owners: @joostlek

• Supports Pydantic v2 (PR: Initial commit)
• Is updated in HA (PR: Migrate WAQI to aiowaqi library #98000)

Remarks:

❌ demetriek

--
Library: demetriek
Integrations using this library: lametric
Code owners: @robbiet480 @frenck @bachya

• Supports Pydantic v2 (PR: Support for pydantic v2 using v1 shims frenck/python-demetriek#531)
• Is updated in HA (PR: # )

Remarks:

✔️ elgato

--
Library: elgato
Integrations using this library: elgato
Code owners: @frenck

• Supports Pydantic v2 (PR: ⬆️ Update dependency pydantic to v2 frenck/python-elgato#833)
• Is updated in HA (PR: Update elgato to 5.0.0 #102405 )

Remarks:

✔️ gcal-sync

--
Library: gcal-sync
Integrations using this library: Google Calendar
Code owners: @allenporter

• Supports Pydantic v2 (PR: Support for pydantic v2 using v1 shims allenporter/gcal_sync#275)
• Is updated in HA (PR: Bump gcal_sync to 5.0.0 #102010)

Remarks:
Supports shims. Needs a release so we can bump the dependency beforehand. Latest version also has ical 5.0.1, and that has pydantic v2 support since 5.0.0.

✔️ google-nest-sdm

--
Library: google-nest-sdm
Integrations using this library: Nest
Code owners: @allenporter

• Supports Pydantic v2 (PR: pydantic v2 with shim for v1 allenporter/python-google-nest-sdm#670)
• Is updated in HA (PR: Update google-nest-sdm to 3.0.2 #99175)

Remarks:

✔️ ical

--
Library: ical
Integrations using this library: Local calendar
Code owners: @allenporter

• Supports Pydantic v2 (PR: Support pydantic v2 with v1 shims allenporter/ical#207)
• Is updated in HA (PR: Bump ical to 5.0.1 #98998)

Remarks:

❌ inflect

--
Library: inflect
Integrations using this library: Abode
Code owners: @shred86

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: # )

Remarks:
This seems to be a deeper problem. I need to dive in to this to know the status.

❌ intellifire4py

--
Library: intellifire4py
Integrations using this library: IntelliFire
Code owners: @jeeftor

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: # )

Remarks:

❌ lacrosse-view

--
Library: lacrosse-view
Integrations using this library: LaCrosse View
Code owners: @IceBotYT

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: # )

Remarks:

✔️ open-meteo

--
Library: open-meteo
Integrations using this library: Open Meteo
Code owners: @frenck

• Supports Pydantic v2 (PR: Bump pydantic from 1.10.9 to 2.0 frenck/python-open-meteo#353)
• Is updated in HA (PR: Update open-meteo to v0.3.1 #103613)

Remarks:
To verify: There has been no code change to support it, but in the lock file is 2.0.3 so I am expecting this works.
If this is as expected, it still needs a release and to be bumped.

❌ peco

--
Library: peco
Integrations using this library: PECO Outage Counter
Code owners: @IceBotYT

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: # )

Remarks:
There has been no explicit PR to fix support for v2. This might work but needs to be verified.

✔️ pvo

--
Library: pvo
Integrations using this library: PVOutput
Code owners: @frenck

• Supports Pydantic v2 (PR: ⬆️ Update dependency pydantic to v2 frenck/python-pvoutput#394)
• Is updated in HA (PR: Update pvo to 2.0.0 #102398 )

Remarks:

❌ pyaussiebb

--
Library: pyaussiebb
Integrations using this library: Aussie Broadband
Code owners: @nickw444 @Bre77

• Supports Pydantic v2 (commit)
• Is updated in HA (PR: Patch service validation in Aussie Broadband #99077)

Remarks:
Library does not support v1 shims. Should be bumped with the v2 bump.
cc @yaleman

✔️ pyrainbird

--
Library: pyrainbird
Integrations using this library: Rain Bird
Code owners: @konikvranik @allenporter

• Supports Pydantic v2 (PR: Support pydantic v2 using v1 shims allenporter/pyrainbird#217)
• Is updated in HA (PR: Bump pyrainbird to 3.0.0 #96610)

Remarks:
Still needs to update ical to =>5.0.0 as ical relies on pydantic as well.

❌ python-bsblan

--
Library: python-bsblan
Integrations using this library: BSB-Lan
Code owners: @liudger

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: # )

Remarks:

✔️ python-kasa

--
Library: python-kasa
Integrations using this library: TP-Link Kasa Smart
Code owners: @rytilahti @TheGardenMonkey

• Supports Pydantic v2 (PR: Add support for pydantic v2 using v1 shims python-kasa/python-kasa#504)
• Is updated in HA (PR: Bump python-kasa to 0.5.4 for tplink #103038)

Remarks:

✔️ python-opensky

--
Library: python-opensky
Integrations using this library: OpenSky Network
Code owners: @joostlek

• Supports Pydantic v2 (PR: Use pydantic shims joostlek/python-opensky#85)
• Is updated in HA (PR: Bump python-opensky to 0.2.0 #97687)

Remarks:

✔️ pytraccar

--
Library: pytraccar
Integrations using this library: Traccar
Code owners: @ludeeus

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: Bump pytraccar from 1.0.0 to 2.0.0 #103318)

Remarks:

✔️ pyunifiprotect

--
Library: pyunifiprotect
Integrations using this library: asd
Code owners: @AngellusMortis @bdraco

• Supports Pydantic v2 (PR: https://github.com/AngellusMortis/pyunifiprotect/pull/297)
• Is updated in HA (PR: Bump unifiprotect to 4.10.5 #96486)

Remarks:

✔️ radios

--
Library: radios
Integrations using this library: Radio Browser
Code owners: @frenck

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: Update radios to v0.2.0 #103614)

Remarks:

❌ sfrbox-api

--
Library: sfrbox-api
Integrations using this library: SFR Box
Code owners: @epenet

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: # )

Remarks:

❌ systembridgeconnector

--
Library: systembridgeconnector
Integrations using this library: System Bridge
Code owners: @timmo001

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: # )

Remarks:

✔️ tailscale

--
Library: tailscale
Integrations using this library: asd
Code owners: @frenck

• Supports Pydantic v2 (PR: Bump pydantic from 1.10.9 to 2.0.2 frenck/python-tailscale#370)
• Is updated in HA (PR: # )

Remarks:
To verify: There has been no code change to support it, but in the lock file is 2.0.3 so I am expecting this works.
If this is as expected, it still needs a release and to be bumped.

✔️ vehicle

--
Library: vehicle
Integrations using this library: RDW
Code owners: @frenck @joostlek

• Supports Pydantic v2 (PR: Restore pydantic v1 compatibility frenck/python-vehicle#441)
• Is updated in HA (PR: Bump vehicle to 2.0.0 #102379 )

Remarks:

✔️ withings-api

--
Library: withings-api
Integrations using this library: Withings
Code owners: @vangorra

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: Use aiowithings #101819)

Remarks:
Is now/will be replaced with a new lib without pydantic

❌ xbox-webapi

--
Library: xbox-webapi
Integrations using this library: Xbox
Code owners: @hunterjm

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: # )

Remarks:

✔️ yolink-api

--
Library: yolink-api
Integrations using this library: YoLink
Code owners: @matrixd2

• Supports Pydantic v2 (PR: YoSmart-Inc/yolink-api@836e61b)
• Is updated in HA (PR: Bump yolink-api to 0.3.4 #105124)

Remarks:

✔️ youtubeaio

--
Library: youtubeaio
Integrations using this library: YouTube
Code owners: @joostlek

• Supports Pydantic v2 (PR: )
• Is updated in HA (PR: # )

Remarks:

✔️ zwave-js-server-python

--
Library: zwave-js-server-python
Integrations using this library: Z-Wave
Code owners: @home-assistant/z-wave

• Supports Pydantic v2 (PR: Bump pydantic to 2.0 with backwards compatibility home-assistant-libs/zwave-js-server-python#703)
• Is updated in HA (PR: Bump zwave-js-server-python to 0.50.1 #94760)

Remarks:

--

For the core members, feel free to update where needed. For others, let us know if something has updated in a comment and we will update it.

cc @cdce8p

[frenck]

As also reported on Discord. But replying here as well, as I was tagged. I have no intention of supporting v2 of Pydantic in my libraries.

Instead, I might be re-evaluating the use of Pydantic overall.

../Frenck

[raman325]

@frenck if you do find a good alternative please do share it here. Others may want to choose the same option. I have v2 semi working but since all the internals have changed and are IMO more complicated I haven't been interested much in investing time into it

[bachya]

A good reminder: one can retain the V1 standards while upgrading to the V2 library. This is an excellent way to unstick this PR without committing yourself to a complete upgrade (assuming they don't get rid of this path at some point).

https://docs.pydantic.dev/latest/migration/#continue-using-pydantic-v1-features

[frenck]

As far as I am aware, it doesn't support everything backward. As said, I have no plans of supporting V2 at this point, and I am not willing to spend any second on it until I have explored other options.

../Frenck

[frenck]

pydantic has been fully removed from tailscale and elgato (and thus no longer taking part in this issue).

[frenck]

pydantic has been fully removed from vehicle (RDW integration) and thus no longer taking part in this issue.

[joostlek]

I've been checking all the boxes :)

[frenck]

Right, but those don't take part in the migration process. It simply aren't using Pydantic at all anymore.

[frenck]

pydantic has been fully removed from pvo (PVOutput integration) and thus no longer taking part in this issue.

[frenck]

pydantic has been fully removed from open-meteo and thus no longer taking part in this issue.
PR @ Core: #103613

[frenck]

pydantic has been fully removed from radios (Radio Browser integration) and thus no longer taking part in this issue.
PR @ Core: #103614

[frenck]

Man, what a liberating feeling! I should have learned about other options before using Pydantic in the first place. It would have saved me from many headaches now and in the past.

[raman325]

@frenck Are you using mashumaro as a replacement? I ask because I don't see a way to use it to validate a dict based on a TypedDict which is what I am looking for, so may have to keep looking :/

[joostlek]

Do you have a repo which we can look at?

[AlexanderLanin]

Sure, although I'm not sure it's worth looking: https://github.com/AlexanderLanin/homeassistant-sungrow/blob/main/custom_components/__init__.py

So far it found incorrect assignment to dataclasses and incorrect parameter types in functions. Both in addition to running mypy.

See https://beartype.readthedocs.io/en/latest/ for more info.

[kquinsland]

See https://beartype.readthedocs.io/en/latest/ for more info.

That looks pretty promising. Thank you for sharing!
I - and I think others - get benefit from the ser/de + coercion aspects of Pydantic which beartype doesn't do :(.

[jeeftor]

I think this worked for my codebase:

try:
    from pydantic.v1 import Field, validator  # type: ignore # noqa F401 # pragma: no cover
    from pydantic.v1 import BaseModel  # type: ignore # pragma: no cover
except ImportError:
    from pydantic import Field  # type: ignore # pragma: no cover
    from pydantic import BaseModel  # type: ignore # pragma: no cover

[timmo001]

Pydantic will be removed from systembridgeconnector in #107957

The dependent package systembridgemodels now only uses standard dataclasses.

[bachya]

aionotion now uses mashumaro and can be removed from the list.

[rytilahti]

fyi, pydantic/pydantic#9042 will allow cleaning up the compat code a bit when it ships.

[jaraco]

FYI, the latest release of inflect no longer depends on pydantic.

[KyeRussell]

fyi, pydantic/pydantic#9042 will allow cleaning up the compat code a bit when it ships.

This has now shipped, and is available in Pydantic 1.10.15.

[joostlek]

I want to spend more time on this when I am done with my thesis, I definetly want to speed the whole process up then

[bdraco]

xbox seems to support v2 now so it might be able to be bumped with v2 as well

OpenXbox/xbox-webapi-python@307a36d

[KyeRussell]

It does not work. They also have no plans on fixing since v1 is out of support. They are working on v3 now and we still have not even gotten to v2.

Unsure what you are referring to. Are you referring to this comment?

Indeed, looks like things weren't fully implemented there. PR welcome with a fix! This is relatively low priority though, given that it's on the V1 end of things.

There is by all accounts an active effort to fix / improve the v1 shim. The comment is just saying that it's low priority, and quite rightfully.

[IceBotYT]

PECO should be compatible once #117165 gets merged

[IceBotYT]

PECO now supports Pydantic v2 and is now merged into HA 👍

[Kludex]

If helped is needed to migrate to V2, I can help.

[drc38]

If helped is needed to migrate to V2, I can help.

See MartinHjelmare/aioopenexchangerates#96 as @MartinHjelmare was wanting help to convert to mashumaro