Implement support for declaring infinite generators #1152

tiangolo · 2020-01-06T20:04:44Z

Change Summary

This implements support for adding typing declarations for infinite generators.

The current implementation allows annotating a generator with Sequence[subType], but it consumes the generator to validate the internal data, so an infinite generator would just hang on model creation, while the model consumes the generator.

The recommended way to annotate a generator that doesn't support len nor __getitem__ (e.g. an infinite generator) is with Iterable[subType]: https://mypy.readthedocs.io/en/latest/cheat_sheet_py3.html#standard-duck-types

This implementation intentionally doesn't consume the generator, assuming it could be infinite or expensive, so it doesn't validate the iterated values. But it allows supporting these infinite generators in pydantic models.

The added docs give an explanation/warning about these generators not being validated and how they are there mainly to support infinite generators, but having Sequence as the (probably) preferred way to annotate consumable generators.

The use case for this is that pydantic will be a requirement for https://github.com/explosion/spaCy and https://github.com/explosion/thinc (already on the develop branches). pydantic is used throughout the libraries, and for some configurations, the libraries take possibly infinite or expensive generators. For things like Machine Learning dataset loading, etc. So it's not viable to have the generators consumed at model creation.

Related issue number

Checklist

Unit tests for the changes exist
Tests pass on CI and coverage remains at 100%
Documentation reflects the changes where applicable
changes/<pull request or issue id>-<github username>.md file added describing change
(see changes/README.md for details)

codecov · 2020-01-06T20:13:39Z

Codecov Report

Merging #1152 into master will not change coverage.
The diff coverage is 100%.

@@          Coverage Diff           @@
##           master   #1152   +/-   ##
======================================
  Coverage     100%    100%           
======================================
  Files          20      20           
  Lines        3445    3479   +34     
  Branches      665     674    +9     
======================================
+ Hits         3445    3479   +34

Impacted Files	Coverage Δ
pydantic/errors.py	`100% <100%> (ø)`	⬆️
pydantic/fields.py	`100% <100%> (ø)`	⬆️
pydantic/schema.py	`100% <100%> (ø)`	⬆️
pydantic/generics.py	`100% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e169bd6...dbe25cc. Read the comment docs.

tiangolo · 2020-01-10T10:11:55Z

@honnibal had the idea that we could do some validation for the first item without consuming the iterator, with something like value = next(iterator); return (value, itertools.chain([value], iterator).

I could document how to do that in a validator if it sounds okay.

…tion

tiangolo · 2020-01-10T12:57:46Z

I went ahead and added docs for validating the first value: https://5e18740d4edea1000b2bd21d--pydantic-docs.netlify.com/usage/types/#infinite-generators-with-validation-for-first-value

That made me discover that there was an extra detail needed for it to work.

So I'm now storing the param type as a sub-field, with its automatic type and shape, to allow using it for normal validation.

It's documented and there's an extra test for that.

samuelcolvin

otherwise this is looking great.

samuelcolvin · 2020-01-10T19:54:11Z

docs/examples/types_infinite_generator_validate_first.py

+for i in m.infinite:
+    print(i)
+    if i == 10:
+        break
+


I don't think you need this again, it was useful in the first example, but hopefully they've got the idea now.

samuelcolvin · 2020-01-10T19:55:37Z

docs/usage/types.md

@@ -157,6 +160,36 @@ with custom properties and validation.
 ```
 _(This script is complete, it should run "as is")_

+### Infinite Generators
+
+If you have a generator you can use `Sequence` as described above. In that case, the generator will be consumed and its values will be validated with the sub-type of `Sequence` (e.g. `int` in `Sequence[int]`).


Suggested change

If you have a generator you can use `Sequence` as described above. In that case, the generator will be consumed and its values will be validated with the sub-type of `Sequence` (e.g. `int` in `Sequence[int]`).

If you have a generator you can use `Sequence` as described above. In that case, the generator will be consumed and stored on the model as a list and its values will be validated with the sub-type of `Sequence` (e.g. `int` in `Sequence[int]`).

(I think?)

Also can you manually wrap lines in this file. (I think all other line are?)

samuelcolvin · 2020-01-10T19:57:00Z

docs/usage/types.md

+
+    pydantic can't validate the values automatically for you because it would require consuming the infinite generator.
+
+#### Infinite Generators with Validation for First Value


Suggested change

#### Infinite Generators with Validation for First Value

#### Validating the for first value

samuelcolvin · 2020-01-10T19:57:17Z

docs/usage/types.md

+
+#### Infinite Generators with Validation for First Value
+
+You can create a [Validator](validators.md) to validate the first value in an infinite generator and still not consume it entirely.


Suggested change

You can create a [Validator](validators.md) to validate the first value in an infinite generator and still not consume it entirely.

You can create a [validator](validators.md) to validate the first value in an infinite generator and still not consume it entirely.

samuelcolvin · 2020-01-10T19:57:56Z

pydantic/fields.py

@@ -416,6 +420,12 @@ def _type_analysis(self) -> None:  # noqa: C901 (ignore complexity)
            self.key_field = self._create_sub_type(self.type_.__args__[0], 'key_' + self.name, for_keys=True)
            self.type_ = self.type_.__args__[1]
            self.shape = SHAPE_MAPPING
+        # Equality check as almost everything inherit form Iterable, including str


Suggested change

# Equality check as almost everything inherit form Iterable, including str

# Equality check as almost everything inherits form Iterable, including str

samuelcolvin · 2020-01-10T20:01:23Z

pydantic/fields.py

@@ -1,10 +1,12 @@
 import warnings
+from collections import abc


Suggested change

from collections import abc

from collections.abc import Iterable as CollectionsIterable

?

I was confused for a minute, confusing this with Lib/abc.py.

samuelcolvin · 2020-01-10T20:02:42Z

pydantic/fields.py

@@ -416,6 +420,12 @@ def _type_analysis(self) -> None:  # noqa: C901 (ignore complexity)
            self.key_field = self._create_sub_type(self.type_.__args__[0], 'key_' + self.name, for_keys=True)
            self.type_ = self.type_.__args__[1]
            self.shape = SHAPE_MAPPING
+        # Equality check as almost everything inherit form Iterable, including str
+        # check for typing.Iterable and abc.Iterable, as it could receive one even when declared with the other
+        elif origin == Iterable or origin == abc.Iterable:


Suggested change

elif origin == Iterable or origin == abc.Iterable:

elif origin in {Iterable, CollectionsIterable}:

or is this slower in cython?

No idea but this looks cleaner. Let's try it.

samuelcolvin · 2020-01-10T20:03:24Z

pydantic/fields.py

+        This intentionally doesn't validate values to allow infinite generators.
+        """
+
+        e: Optional[Exception] = None


do we need to define this?

samuelcolvin · 2020-01-10T20:04:36Z

pydantic/fields.py

+            iterable = iter(v)
+        except TypeError:
+            e = errors_.IterableError()
+            return v, ErrorWrapper(e, loc)


Suggested change

return v, ErrorWrapper(e, loc)

return v, ErrorWrapper(errors_.IterableError(), loc)

surely?

tiangolo · 2020-01-11T15:07:48Z

Thanks for the thorough code review @samuelcolvin ! I just implemented the changes.

* ✨ Implement support for infinite generators with Iterable * ✅ Add tests for infinite generators * 🎨 Fix format * 📝 Add docs for infinite generators * 📝 Add changes file * ✨ Store sub_field with original type parameter to allow custom validation * 📝 Add example for validating first value in infinite generators * 🔥 Remove unused import in example * ✅ Add test for infinite generator with first-value validation * ♻️ Update fields with code review * 📝 Update example from code review * 📝 Update docs and format from code review

tiangolo added 5 commits January 6, 2020 20:29

✨ Implement support for infinite generators with Iterable

233ce07

✅ Add tests for infinite generators

e2de010

🎨 Fix format

5291456

📝 Add docs for infinite generators

1829d57

📝 Add changes file

b08ec2e

tiangolo mentioned this pull request Jan 10, 2020

Add support for generics with __get_validators__ #1159

Merged

4 tasks

tiangolo added 4 commits January 10, 2020 13:52

✨ Store sub_field with original type parameter to allow custom valida…

aadfe1c

…tion

📝 Add example for validating first value in infinite generators

4d54014

🔥 Remove unused import in example

286e14d

✅ Add test for infinite generator with first-value validation

ef146e7

samuelcolvin reviewed Jan 10, 2020

View reviewed changes

tiangolo added 3 commits January 11, 2020 16:04

♻️ Update fields with code review

c7db26c

📝 Update example from code review

fa75fd1

📝 Update docs and format from code review

dbe25cc

samuelcolvin merged commit 496551c into pydantic:master Jan 13, 2020

tiangolo deleted the infinite-generator branch January 13, 2020 16:15

sasano8 mentioned this pull request Oct 12, 2020

How can I distinguish between iterable and iterator? #1995

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement support for declaring infinite generators #1152

Implement support for declaring infinite generators #1152

tiangolo commented Jan 6, 2020 •

edited

codecov bot commented Jan 6, 2020 •

edited

tiangolo commented Jan 10, 2020

tiangolo commented Jan 10, 2020

samuelcolvin left a comment

samuelcolvin Jan 10, 2020

samuelcolvin Jan 10, 2020

samuelcolvin Jan 10, 2020

samuelcolvin Jan 10, 2020

samuelcolvin Jan 10, 2020

samuelcolvin Jan 10, 2020

samuelcolvin Jan 10, 2020

tiangolo Jan 11, 2020

samuelcolvin Jan 10, 2020

samuelcolvin Jan 10, 2020

tiangolo commented Jan 11, 2020

	If you have a generator you can use `Sequence` as described above. In that case, the generator will be consumed and its values will be validated with the sub-type of `Sequence` (e.g. `int` in `Sequence[int]`).
	If you have a generator you can use `Sequence` as described above. In that case, the generator will be consumed and stored on the model as a list and its values will be validated with the sub-type of `Sequence` (e.g. `int` in `Sequence[int]`).


		pydantic can't validate the values automatically for you because it would require consuming the infinite generator.

		#### Infinite Generators with Validation for First Value

	#### Infinite Generators with Validation for First Value
	#### Validating the for first value


		#### Infinite Generators with Validation for First Value

		You can create a [Validator](validators.md) to validate the first value in an infinite generator and still not consume it entirely.

	# Equality check as almost everything inherit form Iterable, including str
	# Equality check as almost everything inherits form Iterable, including str

	from collections import abc
	from collections.abc import Iterable as CollectionsIterable

	elif origin == Iterable or origin == abc.Iterable:
	elif origin in {Iterable, CollectionsIterable}:

	return v, ErrorWrapper(e, loc)
	return v, ErrorWrapper(errors_.IterableError(), loc)

Implement support for declaring infinite generators #1152

Implement support for declaring infinite generators #1152

Conversation

tiangolo commented Jan 6, 2020 • edited

Change Summary

Related issue number

Checklist

codecov bot commented Jan 6, 2020 • edited

Codecov Report

tiangolo commented Jan 10, 2020

tiangolo commented Jan 10, 2020

samuelcolvin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tiangolo commented Jan 11, 2020

tiangolo commented Jan 6, 2020 •

edited

codecov bot commented Jan 6, 2020 •

edited