Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

benchmark for cattrs #513

Merged
merged 11 commits into from Nov 8, 2019
Merged

Conversation

@sebastianmika
Copy link
Contributor

sebastianmika commented May 9, 2019

Change Summary

Added a benchmark for attr (http://www.attrs.org/en/stable/index.html). Run them on my machine (Mac) in a fresh venv with python 3.7.3

Checklist

  • Unit tests for the changes exist NA
  • Tests pass on CI and coverage remains at 100%
  • Documentation reflects the changes where applicable
  • HISTORY.rst has been updated
    • if this is the first change since a release, please add a new section
    • include the issue number or this pull request number #<number>
    • include your github username @<whomever>
@codecov

This comment has been minimized.

Copy link

codecov bot commented May 9, 2019

Codecov Report

Merging #513 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master   #513   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files          19     19           
  Lines        3237   3237           
  Branches      642    642           
=====================================
  Hits         3237   3237

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a34118f...aaf313b. Read the comment docs.

@samuelcolvin

This comment has been minimized.

Copy link
Owner

samuelcolvin commented May 9, 2019

Thanks, I'm keen to include attr as it's definitely a reasonable alternative to pydantic in some circumstances, however I'm not sure this is currently a fair comparison.

Hand written checks are always going to be much faster, but they're not the same thing:

  1. You have to write your own validators while all the other libraries bring their own.
  2. The checks are very different, for example pydantic and the other libraries will check for lots of different date formats where your dtconv method simply checks against a single date format.

IMHO this isn't a mistake in your PR, but rather that attr generally has a different use case from the other libraries tested here:

  • DRF is solely used for deserialising data submitted to an HTTP API (and serialising it, but that's not tested here)
  • marshmallow and trafaret are more general purpose, but are mostly used in the same API case I think
  • pydantic can be used for that but (often through dataclasses) is also used to formalise internal interfaces
  • attr is (as far as I've seen) mostly used for internal interfaces and functions/classes exported from libraries

Perhaps best to add another benchmark for simple type checks were only attr and pydantic are benchmarked and leave attr out of the "web API style" benchmark.

What do you think?

Also I have a dataset I've always used for generating benchmarks. It was generated with the same random case generator and it's not perfect, but I've always used it in the past so for consistency I think best to carry on using it. Just put cases.json into benchmarks/cases.json.

@sebastianmika

This comment has been minimized.

Copy link
Contributor Author

sebastianmika commented May 9, 2019

Thanks, I'm keen to include attr as it's definitely a reasonable alternative to pydantic in some circumstances, however I'm not sure this is currently a fair comparison.

Hand written checks are always going to be much faster, but they're not the same thing:

I agree, I just came across via FastAPI and I was testing whether using it with pydantic would resolve one of our bottlenecks. Since it didn't I ran this test directly and thought it might be worthwhile to commit.

I originally tried to use cattr - but there is a known bug in deserializing None strings to "None" which I didn't want to fix here - and it spoils the comparability.

Perhaps best to add another benchmark for simple type checks were only attr and pydantic are benchmarked and leave attr out of the "web API style" benchmark.

What do you think?

I surely agree. Would you then add another section in the docs? Shall I?

Also I have a dataset I've always used for generating benchmarks. It was generated with the same random case generator and it's not perfect, but I've always used it in the past so for consistency I think best to carry on using it. Just put cases.json into benchmarks/cases.json.

@samuelcolvin

This comment has been minimized.

Copy link
Owner

samuelcolvin commented May 9, 2019

I surely agree. Would you then add another section in the docs? Shall I?

If you're keen that would be great. Yes, another sub-session just below the current benchmarks.

* add env "ATTRS=1" to benchmark runner to only compare again the two
  and save results in separate csv
* added section to docs/index
@sebastianmika

This comment has been minimized.

Copy link
Contributor Author

sebastianmika commented May 9, 2019

So: I added a benchmark for using cattrs - this is actually getting closer to what pydantic does. Still, to keep the types of libs separated I added an extra section for the benchmarks and also extended the benchmark script such that SAVE=1 ATTRS=1 python run.py would now update the csv for that section.

Benchmarks are run using the cases.json you shared

@samuelcolvin

This comment has been minimized.

Copy link
Owner

samuelcolvin commented May 11, 2019

I'm not sure I see the point in having tests with bot attr and cattr, I think we should use just one.

Also, maybe I was wasn't that clear before, but what I was suggesting for the second benchmark set was to only validate the standard type stuff attr is designed (as far as I understand it) to check, and avoid all the more complex checks shown in the first set of benchmarks. Eg. have a second simpler pydantic model for this benchmark and remove all the custom validation logic from the attr case.

@sebastianmika

This comment has been minimized.

Copy link
Contributor Author

sebastianmika commented May 17, 2019

Well, after all I do not care much. I find it this way, i.e. have the same benchmark but with differently flavoured tools, more informative. I tells me that when my required validations and/or non standard types are limited - or if I am willing to write custom validations logic - that I can gain considerable speed-ups.

Actually we use attr + cattr to do pretty much the same we would use pydantic for. And the custom logic required (i.e. validators) is really limited.

So - what now? If you find a "restricted" benchmark more informative I can see if I find the time to strip down the case here (to attr then, not cattr). If not, I personally find this benchmark helpful as it is (actually both).

@samuelcolvin

This comment has been minimized.

Copy link
Owner

samuelcolvin commented May 18, 2019

So - what now? If you find a "restricted" benchmark more informative I can see if I find the time to strip down the case here (to attr then, not cattr). If not, I personally find this benchmark helpful as it is (actually both).

Sorry to take up more of your time, but yes I think the second benchmarks should be restricted - the kind of thing you might check in a package to make sure people are using it (roughly) correctly, rather than what you would check in a public API.

If this isn't clear, let me know and I'll give an exact example.

If this sounds like too much work, let me know and I'll try to do it when I get time. Thanks again.

@samuelcolvin samuelcolvin mentioned this pull request May 24, 2019
0 of 4 tasks complete
@dmontagu

This comment has been minimized.

Copy link
Collaborator

dmontagu commented May 24, 2019

In my own implementation of the benchmark using attrs+cattrs (#543 referenced above), I was getting ~3x faster execution for attrs+cattrs relative to pydantic.

Trying to get to the bottom of why the attrs+cattrs benchmark was faster (and if anything could be easily modified to catch up), I did some line-by-line profiling, and while nothing stood out to me as an obvious bottleneck, I did notice that you pay a price for some of the validations even when they are disabled (e.g., when validating a string, anystr_length_validator gets called even if there are no length restrictions on the instance).

@samuelcolvin Is there a way to actually modify the set of validators that gets called for a given model? That seems more reflective of my understanding of how attrs+cattrs performs validation, and I would expect it to help speed things up if, for instance, you only want type validation. It looked to me like something was happening where there was iteration over each field's list of validators, but if I touched that list (even just removing items from it) it surprisingly caused everything to run much more slowly.

@sebastianmika for what it's worth, if you are trying to resolve a bottleneck, I was able to get a ~30% speed improvement in the pydantic benchmark by cythonizing just fields.py, main.py and validators.py with zero changes to the underlying source (except I had to rename uses of include to in main.py as include is reserved in cython).

@samuelcolvin

This comment has been minimized.

Copy link
Owner

samuelcolvin commented May 25, 2019

discussion on performance improvements moved to #547.

@samuelcolvin samuelcolvin self-assigned this Jun 4, 2019
@samuelcolvin

This comment has been minimized.

Copy link
Owner

samuelcolvin commented Jun 4, 2019

@sebastianmika thanks for this I'm going to make some changes to get it the way I want it (partially back to how it started 😄).

Will work on it when I get a chance.

@sebastianmika

This comment has been minimized.

Copy link
Contributor Author

sebastianmika commented Jun 18, 2019

@samuelcolvin Cool - and sorry for my lameness. I took the freedom to make yet another proposal now that pydantic got much faster with cython - great work! But still, feel free to just change it to your liking.

@samuelcolvin

This comment has been minimized.

Copy link
Owner

samuelcolvin commented Jun 18, 2019

Thanks so much, I'll look into it when I get a chance and will response/make some more changes.

@samuelcolvin

This comment has been minimized.

Copy link
Owner

samuelcolvin commented Aug 6, 2019

Thanks so much for this, sorry I haven't dealt with it. I'm going to defer further changes to benchmarks until after v1 (see #576), but after that I promise to deal with this.

Sorry again for the delay.

@samuelcolvin samuelcolvin removed the deferred label Nov 8, 2019
@samuelcolvin samuelcolvin changed the title benchmark for attr benchmark for cattrs Nov 8, 2019
@samuelcolvin

This comment has been minimized.

Copy link
Owner

samuelcolvin commented Nov 8, 2019

Sorry for the extremely long delay on this.

I've updated your PR to:

  • work with the latest benchmarking code
  • remove attrs since as described above I think it's slightly different from pydantic and I think the cattrs example is a better match for pydantic
  • do all he setup logic on __init__ to match the other libraries.

I'll merge this once it's passed.

samuelcolvin added 2 commits Nov 8, 2019
@samuelcolvin samuelcolvin merged commit b87ca4e into samuelcolvin:master Nov 8, 2019
10 of 11 checks passed
10 of 11 checks passed
continuous-integration/travis-ci/pr The Travis CI build is in progress
Details
Header rules No header rules processed
Details
Pages changed 6 new files uploaded
Details
Redirect rules No redirect rules processed
Details
Mixed content No mixed content detected
Details
codecov/project 100% remains the same compared to a34118f
Details
deploy/netlify Deploy preview ready!
Details
samuelcolvin.pydantic Build #20191108.30 succeeded
Details
samuelcolvin.pydantic (Job Python36) Job Python36 succeeded
Details
samuelcolvin.pydantic (Job Python37) Job Python37 succeeded
Details
samuelcolvin.pydantic (Job Python38) Job Python38 succeeded
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.