Profiling

[jbrockmendel]

We should put together a standardized profiling suite. The pandas folks use asv, not sure what other options they considered.

Tangentially related: @pganssle when running afl the other day, did you put together something to run through various input options? I poked at it long enough to get to the curses screen but not enough to pull anything meaningful out of the results.

[pganssle]

I agree. Is there a framework like this similar to pytest that we can use, or are we in the wild west of everyone-rolls-their-own? (Or maybe we can actually use pytest for this).

I've found a lot of places where simple optimizations can have huge advantages here.

Regarding the afl stuff, I just wrote a little script that throws stuff at the parser. Not sure where else fuzzing would be useful.

[jbrockmendel]

For performance tracking pandas uses asv, but that's not exactly profiling. (Also, for some reason the runs I do locally have way higher variance than most others). Still close enough to the topic to consider. (Possibly a good topic for a sprinter?)

I'm pretty sure pytest-profile or pytest-profiling plugins are out there. We could have a CI environment that runs the tests with that enabled. Not sure how to store the results. That seems like a problem that has probably been solved elsewhere.