195 migrate to pydantic v2 #199

djs0109 · 2023-07-13T11:54:19Z

@tstorek I met the first difficulty while doing the migration. Previously, you defined the QueryStatement and QueryStatement in simple_ql.py, and now they cause the issues: pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'filip.utils.simple_ql.QueryString'>. Do you have an Idea about how to fix that?

tstorek · 2023-07-13T13:37:04Z

@djs0109 I have to look into it.

tstorek · 2023-07-13T14:53:29Z

@djs0109 did you run the migration tool on the tests as well? Otherwise I will add it very quickly

djs0109 · 2023-07-13T16:01:11Z

Yes, I run it already. These parts are not automatically migrated. I guess the reason is that they are not inherited from the pydantic.BaseModel

tstorek · 2023-07-13T16:04:01Z

@djs0109 I will need a little more time. I will go ahead but will require a little for it. The trouble is probably in pydantic-settings but pydantiv itself.

tstorek · 2023-07-14T06:37:34Z

@djs0109 Hi, i looked into it bit. I think that regex expressions are still an issue. There is an open issue in bump-pydantic pydantic/bump-pydantic#80. Lets see, what will happen within the next days :)

For #196

filip/models/ngsi_v2/base.py

tests/models/test_ngsi_v2_subscriptions.py

… 195-Migrate-to-pydantic-v2

djs0109 · 2023-09-29T14:22:59Z

The migration is mostly done. All tests pass also in CI-runner. @tstorek do you have time to review it?

tstorek

Hi, thanks for that big effort. I found some stuff. Please double check on them

tstorek · 2023-10-16T15:03:51Z

filip/clients/ngsi_v2/cb.py

@@ -385,9 +386,11 @@ def get_entity_list(self,
                                      params=params,
                                      headers=headers)
            if AttrsFormat.NORMALIZED in response_format:
-                return parse_obj_as(List[ContextEntity], items)
+                ta = TypeAdapter(List[ContextEntity])


Please follow the notation in the migration guide: https://docs.pydantic.dev/2.0/migration/#introduction-of-typeadapter

I change the variable name from ta to adapter just as in the documentation.

filip/clients/ngsi_v2/cb.py

tstorek · 2023-10-16T15:16:22Z

filip/clients/ngsi_v2/iota.py

@@ -92,7 +93,7 @@ def post_groups(self,

        url = urljoin(self.base_url, 'iot/services')
        headers = self.headers
-        data = {'services': [group.dict(exclude={'service', 'subservice'},
+        data = {'services': [group.model_dump(exclude={'service', 'subservice'},
                                        exclude_none=True,


adapt indentation

tstorek · 2023-10-16T15:23:45Z

filip/config.py

-        env_file_encoding = 'utf-8'
-        case_sensitive = False
+                               validation_alias=AliasChoices('QUANTUMLEAP_URL', 'QL_URL'))
+    model_config = SettingsConfigDict(env_file='.env.filip', env_file_encoding='utf-8',


consider to move to the top of the class. To make it easier to access

Can you describe more about this point? I could not get your suggestion

Model config should be directly below the class doc-string to have a better overview about whats going on

It is already fixed

tstorek · 2023-10-16T15:25:59Z

filip/models/ngsi_v2/base.py

@@ -25,7 +27,8 @@ class Http(BaseModel):
                    "also be supported."
    )

-    @validator('url', allow_reuse=True)
+    @field_validator('url')
+    @classmethod
    def check_url(cls, value):
        return validate_http_url(url=value)


Does this stil work with the new Url-pattern?

Maybe use the new validation pattern for reusing the validator

What new URL-pattern do you mean?

Check on here: This seems far easier:) https://docs.pydantic.dev/latest/api/networks/

~~If I replace L23 with url: HttpUrl or url: AnyHttpUrl, the http data can not be passed into the data model with the syntax Subscription(**sub_dict)~~
That is a good hint 😄

Hence, to change or not?

It is already updated, please check the latest commit

filip/models/ngsi_v2/base.py

filip/semantics/semantics_models.py

djs0109

@tstorek Thanks for the reviewing effort! I went through your comments. Please check again, if it looks ok for you

tstorek

@djs0109 There are jsut minor things. I prefer to have the model config as first entry a pydantic model. That's what I orginally meant with my comment

tstorek · 2023-11-14T09:22:40Z

filip/models/ngsi_v2/base.py

@@ -25,7 +27,8 @@ class Http(BaseModel):
                    "also be supported."
    )

-    @validator('url', allow_reuse=True)
+    @field_validator('url')
+    @classmethod
    def check_url(cls, value):
        return validate_http_url(url=value)


Check on here: This seems far easier:) https://docs.pydantic.dev/latest/api/networks/

filip/semantics/semantics_models.py

djs0109 · 2023-11-16T19:38:56Z

@tstorek sure! I have now adjusted the location of model_config in each class

.gitlab-ci.yml

tstorek

only minor things. Max 1h of work

tstorek · 2023-11-28T10:37:40Z

filip/config.py

-        env_file_encoding = 'utf-8'
-        case_sensitive = False
+                               validation_alias=AliasChoices('QUANTUMLEAP_URL', 'QL_URL'))
+    model_config = SettingsConfigDict(env_file='.env.filip', env_file_encoding='utf-8',


Model config should be directly below the class doc-string to have a better overview about whats going on

tstorek · 2023-11-28T10:40:40Z

filip/models/ngsi_v2/base.py

@@ -25,7 +27,8 @@ class Http(BaseModel):
                    "also be supported."
    )

-    @validator('url', allow_reuse=True)
+    @field_validator('url')
+    @classmethod
    def check_url(cls, value):
        return validate_http_url(url=value)


Hence, to change or not?

filip/models/ngsi_v2/context.py

tstorek · 2023-11-28T10:44:46Z

filip/models/ngsi_v2/iot.py

        default=None,
        description="Endpoint where the device is going to receive commands, "
                    "if any."
    )
+    @field_validator('endpoint')
+    @classmethod
+    def validate_endpoint(cls, value):


https://docs.pydantic.dev/latest/api/networks/

filip/models/ngsi_v2/subscriptions.py

filip/models/ngsi_v2/timeseries.py

tstorek · 2023-11-28T10:48:02Z

filip/utils/validators.py

@@ -22,10 +25,10 @@ def validate_http_url(url: AnyHttpUrl) -> str:
    Returns:
        validated url
    """
-    return url
+    return str(url) if url else url


https://docs.pydantic.dev/latest/api/networks/ this doesn't work?

tests/clients/test_mqtt_client.py

filip/models/ngsi_v2/base.py

tstorek · 2023-11-28T12:57:05Z

@djs0109 i'm sorry somehow the latest commit did not show up

tstorek

Whoop Whoop. Thanks a lot! Let#s go for it. Don#t forget the Changelog ;)

djs0109 added 3 commits July 13, 2023 13:42

chore: change AnyMqttUrl

b01fe7c

chore: pydantic migration based on scripts

7a4a3fc

chore: migration of QueryString

720888c

tstorek mentioned this pull request Jul 14, 2023

Migrate to pydantic v2 #195

Closed

tstorek and others added 7 commits July 14, 2023 11:48

chore: added some migration to tests

330c816

For #196

chore: fixed some more issues

ff9b17a

For #196

chore: fixed open TODO[pydantic]

2d87ca0

For #196

chore: fixed some deprecations

6e127d7

For #196

fix: replace json() with model_dump_json()

c1a3235

chore: migrate function calls in test_units

4b43c09

fix: pydantic-V2 regex validation

cd9693d

tstorek mentioned this pull request Jul 24, 2023

Can the installation of pandas/numpy be optimized/made optional #194

Open

djs0109 added 4 commits July 24, 2023 15:35

fix: log level in env template must be uppercase

ec15855

fix: IOT_JSON should not exist in .env

878e4b6

chore: further migration

bc6edc2

chore: rename filip.types

54361fd

tstorek reviewed Jul 24, 2023

View reviewed changes

filip/models/ngsi_v2/base.py Outdated Show resolved Hide resolved

djs0109 added 3 commits July 28, 2023 11:24

chore: update signature of post validator

78eec86

chore: migration of TypeAdapter, Url, and some small things

b43d3fb

chore: add test for model_dump_json

112d66e

djs0109 commented Jul 28, 2023

View reviewed changes

tests/models/test_ngsi_v2_subscriptions.py Show resolved Hide resolved

djs0109 added 4 commits July 31, 2023 17:02

chore: make validation check optional

a477820

chore: rework mqtt client test

3c2c302

chore: migration changes for Url

3b77341

chore: migration changes for Url

7236a42

djs0109 and others added 3 commits September 28, 2023 14:10

test CI

59697ab

chore: update deprecated functions

9769e94

Merge remote-tracking branch 'origin/195-Migrate-to-pydantic-v2' into…

165dac1

… 195-Migrate-to-pydantic-v2

djs0109 requested a review from tstorek September 29, 2023 14:18

chore: update change log

96a9796

tstorek requested changes Oct 17, 2023

View reviewed changes

djs0109 added 2 commits October 30, 2023 12:12

chore: revision after review

1e8a1c9

chore: revision after review

9f34c9b

djs0109 commented Oct 31, 2023

View reviewed changes

chore: revision after review

0b271de

tstorek requested changes Nov 14, 2023

View reviewed changes

djs0109 added 2 commits November 16, 2023 20:36

chore: use new syntax for HTTP model

af583fb

chore: move model_config to the top of each class

70ab616

FelixStege reviewed Nov 24, 2023

View reviewed changes

.gitlab-ci.yml Show resolved Hide resolved

djs0109 added 3 commits November 24, 2023 21:16

fix: github pages

6525921

chore: add hints for pydantic v1 users

13537d0

chore: change back to gitlab pages

da5bbd1

FelixStege approved these changes Nov 28, 2023

View reviewed changes

tstorek requested changes Nov 28, 2023

View reviewed changes

tstorek reviewed Nov 28, 2023

View reviewed changes

filip/models/ngsi_v2/base.py Outdated Show resolved Hide resolved

djs0109 added 3 commits November 28, 2023 16:05

chore: use AnyMqttUrl in TestMQTTClient

b881bf1

chore: update the usage of network types

01d1848

chore: exclude python 3.12 in CI

b340730

tstorek self-requested a review November 29, 2023 15:21

tstorek approved these changes Nov 29, 2023

View reviewed changes

djs0109 merged commit f5f1d9f into master Nov 29, 2023
1 check passed

tstorek mentioned this pull request Jan 15, 2024

Pydantic >2.0.* seems to result in problems #196

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

195 migrate to pydantic v2 #199

195 migrate to pydantic v2 #199

djs0109 commented Jul 13, 2023 •

edited

tstorek commented Jul 13, 2023

tstorek commented Jul 13, 2023

djs0109 commented Jul 13, 2023

tstorek commented Jul 13, 2023

tstorek commented Jul 14, 2023

djs0109 commented Sep 29, 2023

tstorek left a comment

tstorek Oct 16, 2023

djs0109 Oct 30, 2023

tstorek Oct 16, 2023

tstorek Oct 16, 2023

djs0109 Oct 30, 2023

tstorek Nov 28, 2023

djs0109 Nov 28, 2023

tstorek Oct 16, 2023

tstorek Oct 16, 2023

djs0109 Oct 30, 2023

tstorek Nov 14, 2023

djs0109 Nov 16, 2023 •

edited

tstorek Nov 28, 2023

djs0109 Nov 28, 2023

djs0109 left a comment

tstorek left a comment

tstorek Nov 14, 2023

djs0109 commented Nov 16, 2023

tstorek left a comment

tstorek Nov 28, 2023

tstorek Nov 28, 2023

tstorek Nov 28, 2023

tstorek Nov 28, 2023

tstorek commented Nov 28, 2023

tstorek left a comment •

edited

195 migrate to pydantic v2 #199

195 migrate to pydantic v2 #199

Conversation

djs0109 commented Jul 13, 2023 • edited

tstorek commented Jul 13, 2023

tstorek commented Jul 13, 2023

djs0109 commented Jul 13, 2023

tstorek commented Jul 13, 2023

tstorek commented Jul 14, 2023

djs0109 commented Sep 29, 2023

tstorek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djs0109 Nov 16, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djs0109 left a comment

Choose a reason for hiding this comment

tstorek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

djs0109 commented Nov 16, 2023

tstorek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tstorek commented Nov 28, 2023

tstorek left a comment • edited

Choose a reason for hiding this comment

djs0109 commented Jul 13, 2023 •

edited

djs0109 Nov 16, 2023 •

edited

tstorek left a comment •

edited