benchmark for cattrs

[sebastianmika]

Change Summary

Added a benchmark for attr (http://www.attrs.org/en/stable/index.html). Run them on my machine (Mac) in a fresh venv with python 3.7.3

Checklist

• Unit tests for the changes exist NA
• Tests pass on CI and coverage remains at 100%
• Documentation reflects the changes where applicable
• HISTORY.rst has been updated
  • if this is the first change since a release, please add a new section
  • include the issue number or this pull request number #<number>
  • include your github username @<whomever>

[codecov]

Codecov Report

Merging #513 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff          @@
##           master   #513   +/-   ##
=====================================
  Coverage     100%   100%           
=====================================
  Files          19     19           
  Lines        3237   3237           
  Branches      642    642           
=====================================
  Hits         3237   3237

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a34118f...aaf313b. Read the comment docs.

[samuelcolvin]

Thanks, I'm keen to include attr as it's definitely a reasonable alternative to pydantic in some circumstances, however I'm not sure this is currently a fair comparison.

Hand written checks are always going to be much faster, but they're not the same thing:

1. You have to write your own validators while all the other libraries bring their own.
2. The checks are very different, for example pydantic and the other libraries will check for lots of different date formats where your dtconv method simply checks against a single date format.

IMHO this isn't a mistake in your PR, but rather that attr generally has a different use case from the other libraries tested here:

• DRF is solely used for deserialising data submitted to an HTTP API (and serialising it, but that's not tested here)
• marshmallow and trafaret are more general purpose, but are mostly used in the same API case I think
• pydantic can be used for that but (often through dataclasses) is also used to formalise internal interfaces
• attr is (as far as I've seen) mostly used for internal interfaces and functions/classes exported from libraries

Perhaps best to add another benchmark for simple type checks were only attr and pydantic are benchmarked and leave attr out of the "web API style" benchmark.

What do you think?

Also I have a dataset I've always used for generating benchmarks. It was generated with the same random case generator and it's not perfect, but I've always used it in the past so for consistency I think best to carry on using it. Just put cases.json into benchmarks/cases.json.

[sebastianmika]

Thanks, I'm keen to include attr as it's definitely a reasonable alternative to pydantic in some circumstances, however I'm not sure this is currently a fair comparison.

Hand written checks are always going to be much faster, but they're not the same thing:

I agree, I just came across via FastAPI and I was testing whether using it with pydantic would resolve one of our bottlenecks. Since it didn't I ran this test directly and thought it might be worthwhile to commit.

I originally tried to use cattr - but there is a known bug in deserializing None strings to "None" which I didn't want to fix here - and it spoils the comparability.

Perhaps best to add another benchmark for simple type checks were only attr and pydantic are benchmarked and leave attr out of the "web API style" benchmark.

What do you think?

I surely agree. Would you then add another section in the docs? Shall I?

Also I have a dataset I've always used for generating benchmarks. It was generated with the same random case generator and it's not perfect, but I've always used it in the past so for consistency I think best to carry on using it. Just put cases.json into benchmarks/cases.json.

[samuelcolvin]

I surely agree. Would you then add another section in the docs? Shall I?

If you're keen that would be great. Yes, another sub-session just below the current benchmarks.

[sebastianmika]

So: I added a benchmark for using cattrs - this is actually getting closer to what pydantic does. Still, to keep the types of libs separated I added an extra section for the benchmarks and also extended the benchmark script such that SAVE=1 ATTRS=1 python run.py would now update the csv for that section.

Benchmarks are run using the cases.json you shared

[samuelcolvin]

I'm not sure I see the point in having tests with bot attr and cattr, I think we should use just one.

Also, maybe I was wasn't that clear before, but what I was suggesting for the second benchmark set was to only validate the standard type stuff attr is designed (as far as I understand it) to check, and avoid all the more complex checks shown in the first set of benchmarks. Eg. have a second simpler pydantic model for this benchmark and remove all the custom validation logic from the attr case.

[sebastianmika]

Well, after all I do not care much. I find it this way, i.e. have the same benchmark but with differently flavoured tools, more informative. I tells me that when my required validations and/or non standard types are limited - or if I am willing to write custom validations logic - that I can gain considerable speed-ups.

Actually we use attr + cattr to do pretty much the same we would use pydantic for. And the custom logic required (i.e. validators) is really limited.

So - what now? If you find a "restricted" benchmark more informative I can see if I find the time to strip down the case here (to attr then, not cattr). If not, I personally find this benchmark helpful as it is (actually both).

[samuelcolvin]

So - what now? If you find a "restricted" benchmark more informative I can see if I find the time to strip down the case here (to attr then, not cattr). If not, I personally find this benchmark helpful as it is (actually both).

Sorry to take up more of your time, but yes I think the second benchmarks should be restricted - the kind of thing you might check in a package to make sure people are using it (roughly) correctly, rather than what you would check in a public API.

If this isn't clear, let me know and I'll give an exact example.

If this sounds like too much work, let me know and I'll try to do it when I get time. Thanks again.

[dmontagu]

In my own implementation of the benchmark using attrs+cattrs (#543 referenced above), I was getting ~3x faster execution for attrs+cattrs relative to pydantic.

Trying to get to the bottom of why the attrs+cattrs benchmark was faster (and if anything could be easily modified to catch up), I did some line-by-line profiling, and while nothing stood out to me as an obvious bottleneck, I did notice that you pay a price for some of the validations even when they are disabled (e.g., when validating a string, anystr_length_validator gets called even if there are no length restrictions on the instance).

@samuelcolvin Is there a way to actually modify the set of validators that gets called for a given model? That seems more reflective of my understanding of how attrs+cattrs performs validation, and I would expect it to help speed things up if, for instance, you only want type validation. It looked to me like something was happening where there was iteration over each field's list of validators, but if I touched that list (even just removing items from it) it surprisingly caused everything to run much more slowly.

@sebastianmika for what it's worth, if you are trying to resolve a bottleneck, I was able to get a ~30% speed improvement in the pydantic benchmark by cythonizing just fields.py, main.py and validators.py with zero changes to the underlying source (except I had to rename uses of include to in main.py as include is reserved in cython).

[samuelcolvin]

discussion on performance improvements moved to #547.

[samuelcolvin]

@sebastianmika thanks for this I'm going to make some changes to get it the way I want it (partially back to how it started 😄).

Will work on it when I get a chance.

[sebastianmika]

@samuelcolvin Cool - and sorry for my lameness. I took the freedom to make yet another proposal now that pydantic got much faster with cython - great work! But still, feel free to just change it to your liking.

[samuelcolvin]

Thanks so much, I'll look into it when I get a chance and will response/make some more changes.

[samuelcolvin]

Thanks so much for this, sorry I haven't dealt with it. I'm going to defer further changes to benchmarks until after v1 (see #576), but after that I promise to deal with this.

Sorry again for the delay.

[samuelcolvin]

Sorry for the extremely long delay on this.

I've updated your PR to:

• work with the latest benchmarking code
• remove attrs since as described above I think it's slightly different from pydantic and I think the cattrs example is a better match for pydantic
• do all he setup logic on __init__ to match the other libraries.

I'll merge this once it's passed.