Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remove benchmarks completely #3973

Merged
merged 2 commits into from Apr 4, 2022
Merged

Remove benchmarks completely #3973

merged 2 commits into from Apr 4, 2022

Conversation

samuelcolvin
Copy link
Member

Benchmarks have just started failing, see #3972, so it seems like a good time to remove them.

See #3881, #1810, #1939 (comment) #1425 and many other issues for an explanation of why I'm removing benchmarks.

TL;DR - they're wrong, they're a burden to maintain, they're controversial, they don't really belong in pydantic - they've outlived their usefulness.

In future it would be great if:

  • we had a way of benchmarking pydantic between versions to improve performance and avoid performance slipping - I don't have time to do this now, I might in future
  • it might also be cool if someone build a comparison of different data validation libraries, a bit like the framework benchmarks. But this would be a big piece of work and is way outside the bounds of pydantic

@samuelcolvin samuelcolvin mentioned this pull request Apr 4, 2022
Closed
@samuelcolvin samuelcolvin merged commit 8997cc5 into master Apr 4, 2022
@samuelcolvin samuelcolvin deleted the remove-benchmarks branch April 4, 2022 12:29
@Kludex
Copy link
Member

Kludex commented Apr 4, 2022

we had a way of benchmarking pydantic between versions to improve performance and avoid performance slipping - I don't have time to do this now, I might in future

What do you have in mind here? I know that h11 has a script that runs a microbenchmark, and people are able to add in the PR, see: python-hyper/h11#119. Are you looking for something like that?

aleksul pushed a commit to aleksul/pydantic that referenced this pull request May 13, 2022
@dependabot @aleksul
Fetch upstream
c49868a 
source /home/aleksul/projects/pydantic/venv/bin/activatebuild(deps): bump actions/setup-python from 2 to 3
(pydantic#3868)

Bumps [actions/setup-python](https://github.com/actions/setup-python) from 2 to 3.
- [Release notes](https://github.com/actions/setup-python/releases)
- [Commits](actions/setup-python@v2...v3)

---
updated-dependencies:
- dependency-name: actions/setup-python
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump actions/checkout from 2 to 3 (pydantic#3869)

Bumps [actions/checkout](https://github.com/actions/checkout) from 2 to 3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](actions/checkout@v2...v3)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump mkdocs-material from 8.1.3 to 8.2.3 (pydantic#3865)

Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 8.1.3 to 8.2.3.
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG)
- [Commits](squidfunk/mkdocs-material@8.1.3...8.2.3)

---
updated-dependencies:
- dependency-name: mkdocs-material
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump coverage from 6.2 to 6.3.2 (pydantic#3839)

Bumps [coverage](https://github.com/nedbat/coveragepy) from 6.2 to 6.3.2.
- [Release notes](https://github.com/nedbat/coveragepy/releases)
- [Changelog](https://github.com/nedbat/coveragepy/blob/master/CHANGES.rst)
- [Commits](nedbat/coveragepy@6.2...6.3.2)

---
updated-dependencies:
- dependency-name: coverage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Fixed a typo in decimal_encoder's doc. (pydantic#3820)

build(deps): bump pre-commit from 2.16.0 to 2.17.0 (pydantic#3731)

Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 2.16.0 to 2.17.0.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/master/CHANGELOG.md)
- [Commits](pre-commit/pre-commit@v2.16.0...v2.17.0)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump mypy from 0.930 to 0.931 (pydantic#3656)

Bumps [mypy](https://github.com/python/mypy) from 0.930 to 0.931.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](python/mypy@v0.930...v0.931)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

docs: fix typo in settings management page (pydantic#3781)

build(deps): bump black from 21.12b0 to 22.3.0 (pydantic#3950)

* build(deps): bump black from 21.12b0 to 22.3.0

Bumps [black](https://github.com/psf/black) from 21.12b0 to 22.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/commits/22.3.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* apply new black styles, fix docs

* try upgrading pip before fastapi tests

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Samuel Colvin <s@muelcolvin.com>

build(deps): bump python-dotenv from 0.19.2 to 0.20.0 (pydantic#3963)

Bumps [python-dotenv](https://github.com/theskumar/python-dotenv) from 0.19.2 to 0.20.0.
- [Release notes](https://github.com/theskumar/python-dotenv/releases)
- [Changelog](https://github.com/theskumar/python-dotenv/blob/master/CHANGELOG.md)
- [Commits](theskumar/python-dotenv@v0.19.2...v0.20.0)

---
updated-dependencies:
- dependency-name: python-dotenv
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump hypothesis from 6.31.6 to 6.41.0 (pydantic#3964)

Bumps [hypothesis](https://github.com/HypothesisWorks/hypothesis) from 6.31.6 to 6.41.0.
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](HypothesisWorks/hypothesis@hypothesis-python-6.31.6...hypothesis-python-6.41.0)

---
updated-dependencies:
- dependency-name: hypothesis
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump pytest from 6.2.5 to 7.1.1 (pydantic#3926)

Bumps [pytest](https://github.com/pytest-dev/pytest) from 6.2.5 to 7.1.1.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@6.2.5...7.1.1)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump twine from 3.7.1 to 4.0.0 (pydantic#3965)

Bumps [twine](https://github.com/pypa/twine) from 3.7.1 to 4.0.0.
- [Release notes](https://github.com/pypa/twine/releases)
- [Changelog](https://github.com/pypa/twine/blob/main/docs/changelog.rst)
- [Commits](pypa/twine@3.7.1...4.0.0)

---
updated-dependencies:
- dependency-name: twine
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump cython from 0.29.26 to 0.29.28 (pydantic#3871)

Bumps [cython](https://github.com/cython/cython) from 0.29.26 to 0.29.28.
- [Release notes](https://github.com/cython/cython/releases)
- [Changelog](https://github.com/cython/cython/blob/master/CHANGES.rst)
- [Commits](cython/cython@0.29.26...0.29.28)

---
updated-dependencies:
- dependency-name: cython
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump typing-extensions from 4.0.1 to 4.1.1 (pydantic#3874)

Bumps [typing-extensions](https://github.com/python/typing) from 4.0.1 to 4.1.1.
- [Release notes](https://github.com/python/typing/releases)
- [Changelog](https://github.com/python/typing/blob/master/typing_extensions/CHANGELOG)
- [Commits](python/typing@4.0.1...4.1.1)

---
updated-dependencies:
- dependency-name: typing-extensions
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump pytest-mock from 3.6.1 to 3.7.0 (pydantic#3967)

Bumps [pytest-mock](https://github.com/pytest-dev/pytest-mock) from 3.6.1 to 3.7.0.
- [Release notes](https://github.com/pytest-dev/pytest-mock/releases)
- [Changelog](https://github.com/pytest-dev/pytest-mock/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest-mock@v3.6.1...v3.7.0)

---
updated-dependencies:
- dependency-name: pytest-mock
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Combine dependabot prs (pydantic#3969)

* allow combining of dependabot PRs

* add combine-dependabot.yml

CentOS 7: `read_text(encoding='utf-8')` (pydantic#3625)

With CentOS 7 Python 3.6, running install from source with pip failed:

    sudo podman run -ti --rm centos:7
    yum -y update
    yum -y install epel-release
    yum -y install git python3 python3-devel python3-pip python3-setuptools python3-wheel
    git clone https://github.com/samuelcolvin/pydantic.git
    cd pydantic
    pip3 install .

With following error message:

    [root@c99d0585636c pydantic]# pip3 install .
    Processing /pydantic
        Complete output from command python setup.py egg_info:
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-91v_ixvz-build/setup.py", line 62, in <module>
            history = (THIS_DIR / 'HISTORY.md').read_text()
          File "/usr/lib64/python3.6/pathlib.py", line 1197, in read_text
            return f.read()
          File "/usr/lib64/python3.6/encodings/ascii.py", line 26, in decode
            return codecs.ascii_decode(input, self.errors)[0]
        UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 14648: ordinal not in range(128)

        ----------------------------------------
    Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-91v_ixvz-build/

This PR add the required `read_text(encoding='utf-8')` for `setup.py`.

Signed-off-by: Wong Hoi Sing Edison <hswong3i@pantarei-design.com>

fix: clarify that discriminated unions do not support singletons (pydantic#3639)

Add Robusta.dev to list of Pydantic users (pydantic#3715)

* add robusta.dev to pydantic users

* update robusta.dev description and fix typo

Prevent subclasses of bytes being converted to bytes (pydantic#3707)

* adding a test

* fix and add change description

build(deps): bump mypy from 0.931 to 0.942 (pydantic#3968)

* build(deps): bump mypy from 0.931 to 0.942

Bumps [mypy](https://github.com/python/mypy) from 0.931 to 0.942.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](python/mypy@v0.931...v0.942)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump mypy from 0.931 to 0.942

Bumps [mypy](https://github.com/python/mypy) from 0.931 to 0.942.
- [Release notes](https://github.com/python/mypy/releases)
- [Commits](python/mypy@v0.931...v0.942)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* fix mypy

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Samuel Colvin <s@muelcolvin.com>

Fix pytest crashes with hypothesis and pydantic (pydantic#3727)

Pytest (sometimes?) crashes when it is invoked with `-vv` and pydantic and hypthesis are installed.

This is because `_registered(typ)` modifies `_DEFINED_TYPES` while it is being iterated:

```
INTERNALERROR>   File ".../lib/python3.9/site-packages/pydantic/_hypothesis_plugin.py", line 361, in <module>
INTERNALERROR>     for typ in pydantic.types._DEFINED_TYPES:
INTERNALERROR>   File ".../lib/python3.9/_weakrefset.py", line 65, in __iter__
INTERNALERROR>     for itemref in self.data:
INTERNALERROR> RuntimeError: Set changed size during iteration
```

Remove incorrect comment about lazy evaluation of setting sources (pydantic#3806)

* Remove incorrect comment about lazy evaluation of setting sources

It looks like the current implementation always evaluates every source (https://github.com/samuelcolvin/pydantic/blob/9d631a3429a66f30742c1a52c94ac18ec6ba848d/pydantic/env_settings.py#L73) before coalescing them into a single dictionary to pass to `BaseModel`. So the comment about lazy evaluation is incorrect and should be removed.

* Add changelog

[no ci] correct name of change file

Remove benchmarks completely (pydantic#3973)

* removing benchmarks completely

* [no ci] add change

fix: `Config.copy_on_model_validation` does a deep copy and not a shallow one (pydantic#3642)

* fix: `Config.copy_on_model_validation` does a deep copy and not a shallow one

closes pydantic#3641

* fix: typo

* use python 3.10 to run fastapi tests

* fix fastapi test call

Co-authored-by: Samuel Colvin <s@muelcolvin.com>

build(deps): bump typing-extensions from 4.1.1 to 4.2.0 (pydantic#4040)

Bumps [typing-extensions](https://github.com/python/typing) from 4.1.1 to 4.2.0.
- [Release notes](https://github.com/python/typing/releases)
- [Changelog](https://github.com/python/typing/blob/master/typing_extensions/CHANGELOG.md)
- [Commits](python/typing@4.1.1...4.2.0)

---
updated-dependencies:
- dependency-name: typing-extensions
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump mkdocs-material from 8.2.8 to 8.2.12 (pydantic#4038)

Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 8.2.8 to 8.2.12.
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG)
- [Commits](squidfunk/mkdocs-material@8.2.8...8.2.12)

---
updated-dependencies:
- dependency-name: mkdocs-material
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump pytest from 7.1.1 to 7.1.2 (pydantic#4037)

Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.1.1 to 7.1.2.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](pytest-dev/pytest@7.1.1...7.1.2)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump actions/upload-artifact from 2 to 3 (pydantic#4035)

Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 2 to 3.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](actions/upload-artifact@v2...v3)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump actions/download-artifact from 2 to 3 (pydantic#4034)

Bumps [actions/download-artifact](https://github.com/actions/download-artifact) from 2 to 3.
- [Release notes](https://github.com/actions/download-artifact/releases)
- [Commits](actions/download-artifact@v2...v3)

---
updated-dependencies:
- dependency-name: actions/download-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump email-validator from 1.1.3 to 1.2.1 (pydantic#4060)

Bumps [email-validator](https://github.com/JoshData/python-email-validator) from 1.1.3 to 1.2.1.
- [Release notes](https://github.com/JoshData/python-email-validator/releases)
- [Commits](JoshData/python-email-validator@v1.1.3...v1.2.1)

---
updated-dependencies:
- dependency-name: email-validator
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump hypothesis from 6.41.0 to 6.46.3 (pydantic#4059)

Bumps [hypothesis](https://github.com/HypothesisWorks/hypothesis) from 6.41.0 to 6.46.3.
- [Release notes](https://github.com/HypothesisWorks/hypothesis/releases)
- [Commits](HypothesisWorks/hypothesis@hypothesis-python-6.41.0...hypothesis-python-6.46.3)

---
updated-dependencies:
- dependency-name: hypothesis
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

build(deps): bump pre-commit from 2.17.0 to 2.19.0 (pydantic#4061)

Bumps [pre-commit](https://github.com/pre-commit/pre-commit) from 2.17.0 to 2.19.0.
- [Release notes](https://github.com/pre-commit/pre-commit/releases)
- [Changelog](https://github.com/pre-commit/pre-commit/blob/main/CHANGELOG.md)
- [Commits](pre-commit/pre-commit@v2.17.0...v2.19.0)

---
updated-dependencies:
- dependency-name: pre-commit
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

test pyright with pydantic (pydantic#3972)

* test pyright with pydantic

* rename file to avoid pytest running it

* try another name 😴

* add docs about BaseSettings and Field

* add change

Fix regression in handling of nested dataclasses in `get_flat_models_from_field` (pydantic#3819)

* add test for nested python dataclass schema generation

* fix handling of dataclasses in `get_flat_models_from_field`

* add change note

Fix issue with self-referencing dataclass (pydantic#3713)

* Fix issue with self-referencing dataclass

* Fix mypy issue

guard against ClassVar in fields (pydantic#4064)

* guard against ClassVar in fields, fix pydantic#3679

* fix linting

* skipif for test_class_var_forward_ref

Fix issue with in-place modification of FieldInfo (pydantic#4067)

* Fix info with in-place modification of field info

* add changes

* add test for 3714

* Update changes/4067-adriangb.md

Co-authored-by: Samuel Colvin <samcolvin@gmail.com>

Co-authored-by: Samuel Colvin <samcolvin@gmail.com>

build(deps): bump mkdocs-material from 8.2.8 to 8.2.14 (pydantic#4063)

Bumps [mkdocs-material](https://github.com/squidfunk/mkdocs-material) from 8.2.8 to 8.2.14.
- [Release notes](https://github.com/squidfunk/mkdocs-material/releases)
- [Changelog](https://github.com/squidfunk/mkdocs-material/blob/master/CHANGELOG)
- [Commits](squidfunk/mkdocs-material@8.2.8...8.2.14)

---
updated-dependencies:
- dependency-name: mkdocs-material
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
samuelcolvin added a commit that referenced this pull request May 14, 2022
@samuelcolvin
Remove benchmarks completely (#3973)
f4f4ab3 
* removing benchmarks completely

* [no ci] add change
ntaylorwss pushed a commit to nicejobinc/pydantic that referenced this pull request Jun 24, 2022
@samuelcolvin
Remove benchmarks completely (pydantic#3973)
c0f3965 
* removing benchmarks completely

* [no ci] add change
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@samuelcolvin @Kludex