Add auto generate conversion table to doc #5324

hramezani · 2023-03-31T13:33:34Z

Change Summary

Add auto-generate conversion table

This is the initial work to generate the conversion table automatically.
I've replaced the conversion table in v2 blog with {{ conversion_table }}. The conversion table rows are not complete yet and I will update the PR later.
Meanwhile, I am going to add missed tests as well.

I've built the docs locally and the new build_conversion_table works fine and generates the conversion table as expected.

Related issue number

Fixes #4680

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)
My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

Selected Reviewer: @adriangb

samuelcolvin

otherwise looking good.

I guess we need to add more types before we see how this really works. In particular, do we need more information to describe generic types?

samuelcolvin · 2023-03-31T14:30:02Z

docs/blog/pydantic-v2.md

-| `is_instance` | -           | both   | JSON         | never valid                                                                 |
-| `callable`    | `Any`       | both   | python       | `callable()` check returns `True`                                           |
-| `callable`    | -           | both   | JSON         | never valid                                                                 |
+{{ conversion_table }}


Probably best to leave the one in the blog post unchanged.

Instead a new page to usage with the conversion table.

We'll need to add a big heading to this blog post that it's superseded.

Probably best to leave the one in the blog post unchanged.

Reverted

Instead a new page to usage with the conversion table.

A new page was added as usage/conversion_table.md

We'll need to add a big heading to this blog post that it's superseded.

I've made the existing sentence bold.

samuelcolvin · 2023-03-31T14:31:08Z

docs/plugins/conversion_table.py

+class Row:
+    field_type: type[Any]
+    input_type: type[Any]
+    mode: Literal['lax', 'strict', 'both']


Suggested change

mode: Literal['lax', 'strict', 'both']

mode: Literal['lax', 'strict']

"strict" should always imply "lax" as well, if there are exceptions, we can add specific comments.

You are right. both removed

samuelcolvin · 2023-03-31T14:32:13Z

docs/plugins/conversion_table.py

+        'lax',
+        'python',
+        condition='assumes UTF-8, error on unicode decoding error',
+        valid_examples=[b's'],


Suggested change

valid_examples=[b's'],

valid_examples=[b'this is bytes'],

probably good to change the others to something more readable too.

All changed

samuelcolvin · 2023-03-31T14:33:00Z

docs/plugins/conversion_table.py

+    field_type: type[Any]
+    input_type: type[Any]
+    mode: Literal['lax', 'strict', 'both']
+    input_format: Literal['python', 'JSON', 'python, JSON']


Suggested change

input_format: Literal['python', 'JSON', 'python, JSON']

input_format: Literal['python', 'JSON', 'python & JSON']

samuelcolvin · 2023-03-31T14:33:51Z

docs/plugins/conversion_table.py

+        'both',
+        'python, JSON',
+        condition='max abs value 2^64 - i64 is used internally',
+        core_schema=core_schema.IntSchema,


can you add an invalid example.

I've added an invalid example. Also, I've separated the strict and lax mode for int type in table because it raises ValidationError for number > limit in strict mode but it will limit the input to the max in lax mode.
I've added descriptions for these in the condition field but I am not sure about the wording.

I've added tests for both above-mentioned cases.

samuelcolvin · 2023-03-31T14:34:29Z

docs/plugins/conversion_table.py

+    core_schema: type[CoreSchema] = None
+
+
+table_infos: list[Row] = [


Suggested change

table_infos: list[Row] = [

table: list[Row] = [

samuelcolvin · 2023-03-31T14:35:53Z

docs/plugins/main.py

-    'Defined in',
-]
-
-
 def md2html(s: str) -> str:


we might want other things, can we use the proper markdown -> html library from mkdocs?

Not very familiar with mkdocs and it's the related library.
@tpdorsey Do you have an idea?

if this isn't easy, keep it as-is, at least for now.

samuelcolvin · 2023-04-01T10:02:46Z

please update.

koxudaxi · 2023-04-03T14:23:12Z

docs/plugins/conversion_table.py

+    core_schema: type[CoreSchema] = None
+
+
+table: list[Row] = [


@samuelcolvin
The row list will be distributed as JSON lines or another format? in the wheel.
Does the list affect the behavior of the strict argument? Or does the list explain the behavior pydantic-core?
Also, Does the user custom type use the Row class?

if including the JSON in the wheel/sdist would help you, we definitely can.

Yes, the strictargument is relevant here, that logic is mostly implemented in pydantic-core, some is implemented in types in pydantic.

There's currently no way for user custom types to provide Row information.

if including the JSON in the wheel/sdist would help you, we definitely can.

Thank you. I will pick up the JSON from the package.

Yes, the strictargument is relevant here, that logic is mostly implemented in pydantic-core, some is implemented in types in pydantic.

I see.

There's currently no way for user custom types to provide Row information.

How can I get the expected type of user custom types?
I don't need the answer now. But, we should be clear sometime.

tpdorsey · 2023-04-04T15:57:05Z

docs/plugins/conversion_table.py

+    Row(
+        date,
+        date,
+        'stric',


tpdorsey · 2023-04-04T15:57:16Z

docs/plugins/conversion_table.py

+    Row(
+        None,
+        None,
+        'stric',


tpdorsey · 2023-04-04T15:59:15Z

docs/plugins/conversion_table.py

+        datetime,
+        'lax',
+        'python',
+        condition='must be exact date, eg. no H, M, S, f',


Suggested change

condition='must be exact date, eg. no H, M, S, f',

condition='must be exact date, eg. no `H`, `M`, `S`, `f`',

tpdorsey · 2023-04-04T15:59:56Z

docs/plugins/conversion_table.py

+        'lax',
+        'python & JSON',
+        condition=(
+            'interpreted as seconds or ms from epoch, '


Suggested change

'interpreted as seconds or ms from epoch, '

'Interpreted as seconds or ms from epoch. '

tpdorsey · 2023-04-04T16:00:15Z

docs/plugins/conversion_table.py

+        'python & JSON',
+        condition=(
+            'interpreted as seconds or ms from epoch, '
+            'see [speedate](https://docs.rs/speedate/latest/speedate/), must be exact date'


Suggested change

'see [speedate](https://docs.rs/speedate/latest/speedate/), must be exact date'

'See [speedate](https://docs.rs/speedate/latest/speedate/). Must be exact date.'

Suggest applying same change to equivalent conditions.

tpdorsey · 2023-04-04T16:01:21Z

docs/plugins/conversion_table.py

+        str,
+        'strict',
+        'JSON',
+        condition='format YYYY-MM-DDTHH:MM:SS.f etc. see [speedate](https://docs.rs/speedate/latest/speedate/)',


Suggested change

condition='format YYYY-MM-DDTHH:MM:SS.f etc. see [speedate](https://docs.rs/speedate/latest/speedate/)',

condition='Format: `YYYY-MM-DDTHH:MM:SS.f`. See [speedate](https://docs.rs/speedate/latest/speedate/).',

Again, apply same edit to similar conditions.

tpdorsey · 2023-04-04T16:02:23Z

docs/plugins/conversion_table.py

+        Any,
+        'strict',
+        'python',
+        condition='`isinstance()` check returns True',


Suggested change

condition='`isinstance()` check returns True',

condition='`isinstance()` check returns `True`.',

hramezani · 2023-04-05T08:25:54Z

Thanks @tpdorsey, All addressed. please review

samuelcolvin reviewed Mar 31, 2023

View reviewed changes

pydantic-hooky bot added the awaiting author revision awaiting changes from the PR author label Apr 1, 2023

pydantic-hooky bot assigned hramezani Apr 1, 2023

samuelcolvin added the documentation label Apr 1, 2023

hramezani force-pushed the conversion_table branch 5 times, most recently from a89e3fd to f6e9ab6 Compare April 3, 2023 11:37

samuelcolvin mentioned this pull request Apr 3, 2023

Support Pydantic V2 koxudaxi/pydantic-pycharm-plugin#694

Open

10 tasks

koxudaxi reviewed Apr 3, 2023

View reviewed changes

hramezani force-pushed the conversion_table branch 3 times, most recently from 5bc1742 to 74fa64c Compare April 4, 2023 08:59

tpdorsey reviewed Apr 4, 2023

View reviewed changes

hramezani added 13 commits April 5, 2023 11:06

Add auto generate conversion table to doc

fd08e8f

Address comments

74042ef

Rewrite doc tables with markdown

c8dff37

Add more types and move some tests

51a2adb

Add None, date, and datetime type

fe962c1

Add time and timedelta type

a59333c

Add dict type

e5fa532

Add pydantic-core to docs requirements

1796eb0

Add TypedDict

6b2d78e

Add list type

2ac025e

Add more types

2bbd24d

Add missed types

363b5e1

Address comments

1df088e

hramezani force-pushed the conversion_table branch from 0f99980 to 1df088e Compare April 5, 2023 08:21

pydantic-hooky bot added ready for review and removed awaiting author revision awaiting changes from the PR author labels Apr 5, 2023

pydantic-hooky bot assigned adriangb and unassigned hramezani Apr 5, 2023

hramezani assigned tpdorsey and unassigned adriangb Apr 5, 2023

tpdorsey approved these changes Apr 5, 2023

View reviewed changes

tpdorsey merged commit 62faa35 into pydantic:main Apr 5, 2023

samuelcolvin mentioned this pull request Apr 25, 2023

Enhance pydantic types, add info about actual expected type of a value. #1055

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add auto generate conversion table to doc #5324

Add auto generate conversion table to doc #5324

hramezani commented Mar 31, 2023 •

edited by pydantic-hooky bot

Loading

samuelcolvin left a comment •

edited

Loading

samuelcolvin Mar 31, 2023

hramezani Apr 1, 2023

samuelcolvin Mar 31, 2023

hramezani Apr 1, 2023

samuelcolvin Mar 31, 2023

hramezani Apr 1, 2023

samuelcolvin Mar 31, 2023

hramezani Apr 1, 2023

samuelcolvin Mar 31, 2023

hramezani Apr 1, 2023

samuelcolvin Mar 31, 2023

hramezani Apr 1, 2023

samuelcolvin Mar 31, 2023

hramezani Apr 1, 2023

samuelcolvin Apr 1, 2023

samuelcolvin commented Apr 1, 2023

koxudaxi Apr 3, 2023

samuelcolvin Apr 3, 2023

koxudaxi Apr 3, 2023

tpdorsey Apr 4, 2023

tpdorsey Apr 4, 2023

tpdorsey Apr 4, 2023

tpdorsey Apr 4, 2023

tpdorsey Apr 4, 2023

tpdorsey Apr 4, 2023

tpdorsey Apr 4, 2023

tpdorsey Apr 4, 2023

tpdorsey Apr 4, 2023

hramezani commented Apr 5, 2023

	mode: Literal['lax', 'strict', 'both']
	mode: Literal['lax', 'strict']

	input_format: Literal['python', 'JSON', 'python, JSON']
	input_format: Literal['python', 'JSON', 'python & JSON']

		core_schema: type[CoreSchema] = None


		table_infos: list[Row] = [

	condition='must be exact date, eg. no H, M, S, f',
	condition='must be exact date, eg. no `H`, `M`, `S`, `f`',

	'interpreted as seconds or ms from epoch, '
	'Interpreted as seconds or ms from epoch. '

	'see [speedate](https://docs.rs/speedate/latest/speedate/), must be exact date'
	'See [speedate](https://docs.rs/speedate/latest/speedate/). Must be exact date.'

	condition='format YYYY-MM-DDTHH:MM:SS.f etc. see [speedate](https://docs.rs/speedate/latest/speedate/)',
	condition='Format: `YYYY-MM-DDTHH:MM:SS.f`. See [speedate](https://docs.rs/speedate/latest/speedate/).',

	condition='`isinstance()` check returns True',
	condition='`isinstance()` check returns `True`.',

Add auto generate conversion table to doc #5324

Add auto generate conversion table to doc #5324

Conversation

hramezani commented Mar 31, 2023 • edited by pydantic-hooky bot Loading

Change Summary

Related issue number

Checklist

samuelcolvin left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samuelcolvin commented Apr 1, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hramezani commented Apr 5, 2023

hramezani commented Mar 31, 2023 •

edited by pydantic-hooky bot

Loading

samuelcolvin left a comment •

edited

Loading