Replace Pydantic with custom validation logic, restore v0.1.5 config and registry algorithm#73
Merged
Conversation
Remove pydantic and pydantic-core dependencies entirely. Add confection/_validation.py with Schema base class, dynamic schema creation, and type validation supporting unions, generics, forward references, Literal, generators, and constrained types (StrictBool, PositiveInt, StrictFloat). Key changes: - New _validation.py module replaces all Pydantic functionality - _registry.py uses custom Schema/create_schema/Field/ValidationError - util.py Generator class uses metaclass for isinstance checks - Removed test_pydantic_generators.py (Pydantic-specific tests) - Updated all test imports from pydantic to confection._validation - Removed pydantic from setup.cfg install_requires
Downstream libraries (spaCy, thinc) pass pydantic BaseModel subclasses to registry.resolve(schema=...) and registry.fill(schema=...). The shim transparently converts these to our Schema interface: - Extracts fields from pydantic v1 (__fields__) or v2 (model_fields) - Extracts config from v1 (inner class Config) or v2 (model_config dict) - Handles v1 Extra enum for extra='forbid'/'allow' - Delegates model_validate to the original pydantic class so that pydantic validators, strict types, and constraints keep working - Recursively converts nested BaseModel field annotations - Caches conversions for performance Also fixes validate_type to accept constrained subtypes (e.g. pydantic's StrictInt which inherits from int) by checking issubclass(annotation, type(value)) as a fallback.
- resolve_type_hints: catch (NameError, AttributeError, TypeError) instead of bare Exception - _is_pydantic_model: catch (ImportError, ModuleNotFoundError) on import, use else clause for issubclass so import errors and logic errors are never conflated - _pydantic_model_validate: resolve the concrete pydantic ValidationError class(es) up front via _get_pydantic_validation_error() so the except clause catches only pydantic validation errors, not arbitrary exceptions
Start from confection v0.1.5 (last pydantic v1 release) and replace pydantic with our validation module, keeping the registry code (_fill, _make, etc.) intact: - Replace pydantic imports with validation.py (Schema, Field, etc.) - Replace srsly with stdlib json - BaseModel -> Schema, __fields__ -> model_fields, parse_obj -> model_validate - copy_model_field -> _override_field_to_any - create_model -> create_schema - Add ensure_schema at resolve/fill/_fill entry points - Drop pydantic and srsly from install_requires 102/108 stock tests passing. Remaining 5 failures need test updates for __fields__ -> model_fields and pydantic-specific assertions.
- test_make_promise_schema: __fields__ -> model_fields - test_resolve_schema_coerced: expect raw values (no pydantic coercion) - test_validation_custom_types: fix pydantic v1 validator calling — skip 3-arg validators needing config objects, use field shim for 2-arg - test_config_reserved_aliases: reorder validator hooks before constrained-subtype fallback so StrictBool/PositiveInt validators run - test_config_fill_without_resolve: reset model_validate to Schema's own after _override_field_to_any mutates fields - test_frozen_struct_deepcopy: unfreeze frozen dict/list defaults so _update_from_parsed can write to them 111/111 stock v0.1.5 tests passing.
- _override_field_to_any: create a new Schema subclass instead of mutating the cached wrapper, so ensure_schema cache isn't corrupted - Add _contains_promise: detect nested promises in dict values - Override fields to Any when value contains nested promises and resolve=False (matches stock confection's behavior) - Fix pydantic v1 validator calling: skip 3-arg validators needing config objects, use field shim for 2-arg constraint validators - Reorder validator hooks before constrained-subtype fallback 111/111 confection tests, 21/21 spaCy serialize_config tests passing.
- test_pydantic_shim: 16/16 pass — tests ensure_schema backward compat - test_config_values: 89/102 pass — Hypothesis property tests for config parsing. 13 failures are json serialization edge cases from srsly→json migration (whitespace handling, negative numeric strings), not pydantic related. Total: 202 passed across all test files.
Cherry-pick the config parsing fixes from dbc8484 and a4e2d58: - Simplify before_read: just warn about single quotes, no unquoting (the unquoting caused double-parsing of string values like '0') - Add _coerce_for_string_context: coerce values in compound string interpolation (bare refs keep JSON type, compounds join as strings) - Rewrite before_get: detect bare refs (len(L)==1) vs compound - Fix _get_section_ref: guard against value == SECTION_PREFIX, use replace(prefix, '', 1) to avoid stripping too much - Remove isdigit() hack in try_dump_json (no longer needed without the unquoting in before_read) - Remove JSON_EXCEPTIONS constant (no longer used) 215/215 tests passing, 0 failures.
- Replace black, isort, flake8 with ruff in CI, requirements, config - Remove mypy config from setup.cfg - Add ruff config to pyproject.toml - Fix lint issues: unused imports, unused vars, trailing whitespace, type comparison (== -> is), import sorting - CI: single 'ruff check' + 'ruff format --check' step
pydantic v1 compat layer is broken on Python 3.14. Skip test_config.py and test_pydantic_shim.py entirely on 3.14+. Fall back to our own StrictBool in tests/util.py when pydantic is unavailable.
When validate_type encounters a TypeVar with a bound, validate against the bound. For constrained TypeVars, try each constraint. This catches type mismatches like passing ndarray where SeqT (bound to Union[Ragged, Padded, List[...]]) is expected.
- FieldInfo.annotation: type as Any instead of None - alias_gen: add callable() check for type narrowing - sys.modules.get: guard against None module name - NewType __supertype__: suppress reportFunctionMemberAccess - Extra.value: suppress reportAttributeAccessIssue (guarded by hasattr) - pydantic ValidationError.errors(): suppress (typed as Exception) - SimpleFrozenDict.update: suppress override warning 7 remaining errors are pre-existing ConfigParser internals from stock.
Replace in-function try/except pydantic imports with module-level imports that set _PydanticV1BaseModel, _PydanticV2BaseModel, etc. to None when pydantic is not installed. Cleaner and avoids repeated import attempts on every call.
3 remaining errors are pre-existing stock ConfigParser internals.
44 tests covering all type validation branches: - None, Annotated, Union, Optional, pipe syntax - Literal, NewType, TypeVar (unbound, bound, constrained) - Plain types: bool, int, float, str, Path - Generic types: Callable, List, Dict, Tuple (bare, fixed, variable), Set, FrozenSet, Sequence, Iterable, Mapping, Iterator, Type[X] - Schema-as-dict validation - Pydantic hooks (__get_pydantic_core_schema__, __get_validators__) - Generator passthrough - _validate_schema: extra forbid/allow, missing required, aliases - ensure_schema passthrough - model_dump, Schema.from_function validation.py coverage: 66% -> 87%
Ruff splits combined imports into separate statements. Each from line needs its own pyright: ignore[reportMissingImports] comment since pydantic is not installed during CI lint.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pydantic v2 is stricter than v1 and I've been unable to replicate the behaviours we had in the config.
I had also attempted to refactor the library to make it easier to port, but this seems to have also introduced regressions.
To finally resolve this I'm dropping support for Pydantic in favour of a relatively small amount of custom validation logic. This should finally unblock spaCy and Thinc.