add "north star" benchmark

[davidhewitt]

Change Summary

Adds a new set of test_north_star benchmarks which we can potentially use to get a general sense of performance for something which (I hope) looks like a typical use-case.

The commit contains a new file generate_data.py which creates a ~500KB file to be used as input for the benchmarks. I didn't check this file in yet, although I am happy to (or consider alternative strategies like storing in S3).

The north star benchmarks measure time for successful validation for each of json and python inputs, in both lax and strict mode. There is also a comparison to json.dumps to measure our overhead. On my machine, this is current measurements:

------------------------------------------------------ benchmark: 8 tests -----------------------------------------------------
Name (time in ms)                              Min               Mean            StdDev            Outliers  Rounds  Iterations
-------------------------------------------------------------------------------------------------------------------------------
test_north_star_dump_python                 1.7044 (1.0)       2.2383 (1.0)      2.6097 (7.09)         8;58     584           1
test_north_star_json_loads                  1.9442 (1.14)      2.3952 (1.07)     2.3015 (6.25)         6;48     515           1
test_north_star_json_dumps                  2.7657 (1.62)      3.1403 (1.40)     0.3681 (1.0)         38;23     361           1
test_north_star_dump_json                   3.3257 (1.95)      3.7396 (1.67)     0.5104 (1.39)        23;18     298           1
test_north_star_validate_python_strict     31.7533 (18.63)    37.3106 (16.67)    7.9220 (21.52)         4;4      31           1
test_north_star_validate_python            36.9103 (21.66)    42.7102 (19.08)    9.4304 (25.62)         4;4      28           1
test_north_star_validate_json_strict       37.3472 (21.91)    42.2550 (18.88)    7.6634 (20.82)         4;4      27           1
test_north_star_validate_json              38.0183 (22.31)    43.1739 (19.29)    7.5644 (20.55)         4;4      27           1
-------------------------------------------------------------------------------------------------------------------------------

Here are the v1 measurements by comparison:

---------------------------------------------------- benchmark: 4 tests ---------------------------------------------------
Name (time in ms)                          Min               Mean            StdDev            Outliers  Rounds  Iterations
---------------------------------------------------------------------------------------------------------------------------
test_north_star_dump_python_v1         23.2602 (1.0)      25.3978 (1.0)      3.9667 (2.28)          2;6      43           1
test_north_star_dump_json_v1           36.3348 (1.56)     38.3833 (1.51)     1.7393 (1.0)          10;1      28           1
test_north_star_validate_python_v1     83.0300 (3.57)     87.0504 (3.43)     6.8162 (3.92)          1;1      13           1
test_north_star_validate_json_v1       85.2474 (3.66)     91.6602 (3.61)     6.7360 (3.87)          2;2      12           1
---------------------------------------------------------------------------------------------------------------------------

Related issue number

N/A

Checklist

• Unit tests for the changes exist
• Documentation reflects the changes where applicable
• Pydantic tests pass with this pydantic-core (except for expected changes)
• My PR is ready to review, please add a comment including the phrase "please review" to assign reviewers

[davidhewitt]

(CI failures are expected because I haven't checked in the 500KB sample data file pending a decision about how to store it most appropriately.)

[dmontagu]

@davidhewitt would it be easy for you to produce this table for v1 for comparison? I am just curious. Also, these are all validation (not serialization) benchmarks, right?

[davidhewitt]

Added serialization benchmarks and also equivalent v1 functionality as new commits. Updated the OP text to contain all these measurements.

[adriangb]

I think committing the file is okay if it's 500kb. It won't be changing much so it's a one-time download.

[davidhewitt]

Moved to pydantic/pydantic#6547