Performance testing suite design

[shcheklein]

Opening this issue to start the discussion on how should we approach the implementation of the DVC performance tests (aka time testing, aka benchmarking, etc).

Motivation:

As we go we should see performance degradation. For example, someone made a heavy import that affects CLI time to show something (this dvc help that should run in ms). Or, we implemented some dvc checkout logic that affects performance. Now, we have to run every time manually some additional checks. It's fragile, people tend to forget to run it, etc.

Requirements:

• Cross-platform - Windows, Max, Linux. There are some system-specific scenarios we would like to monitor and catch (100K+ files in NTFS dir). run benchmarks on Mac and Windows dvc-bench#14
• Supports automation - run at least nightly on more or less same stack of machines to being able to compare to previous runs. nightly builds for dvc master dvc-bench#15
• Ideally, creates a ticket with p1/p0 automatically to investigate on a certain threshold.
• Ideally, can try binary search to figure which commit affected this. Ping engineering on noticable performance degradation dvc-bench#8
• At least nightly reports to see changes. nightly builds for dvc master dvc-bench#15
• Support for bash/python scripts to run. using python for now because bash is hard to run on windows. And it is easier to profile with python.
• Easy to run a specific test/all tests locally with a specific DVC installation.
• Easy way to add new tests.

Implementations:

• We should definitely take a look at other projects (cpython? databases?), available cloud solutions for this?
• How do we implement this: separate repo dvc-test vs a directory in the main repo tests/time or tests/bench or whatnot.
  • dvc-test seems to be more flexible to my mind, I don't see any downsides if it's implemented right
  • directory approach: @Suor could you put a comment - what specific advantages do you see here that could not implemented in the directory one?

@iterative/engineering any thoughts? any comments are welcome - I'll be editing this ticket to come up with a good summary.

[pared]

Sharing some of my thoughts

Reproducibility

We need to make sure tests are run at machines with +/- constant computation power, so that ourt tests will not be affected by hardware upgrades. Probably we should consider caching info about benchmark system hardware, to be able to notice if change in performance happend due to hardware, or even rerun test (for example for major versions) when test machine has change, so that we have some context.

Monitoring

At least nightly reports to see changes.

We should probably be storing somewhere our tests results for different revisions and visualise them so that we have easy to read chart performance(time). Something like grafana/kibana, or even just pyplot chart embedded in html would be nice indicator that could let us know that some change introduced major slowdown.

We can take as an example the pyperformance benchmark suite.

• --affinity option to force running the commands on a given CPU list. (We can leverage cgroups in Linux)
• Real examples

I don't find any problem with having this benchmarks under the DVC repository.

Django also has their own benchmarks.

[dmpetrov]

Re separate repo dvc-test vs a directory in the main repo tests/time...

@Suor brought a good point - people will need the tests in the development stage. So, keeping test in dvc repro will make the tests more useful. However, for the automation (running tests periodically, creates a ticket etc) a separate system and repo might be needed.

It looks like it makes sense to start developing the perf-tests in dvc repo.

[shcheklein]

I don't see a connection between "needed in the development stage" and one repo vs two repos. I've seen this organized very well, and I don't remember any problems with running the test harness from a separate repo - the difference should be - cd ../tests vs cd bench.

On the other hand It's clear how creating it in the same repo limits the flexibility and overcomplicates the repo.

• One travis config when you need two.
• Different requirements in terms of flake, black, other linters.
• Will have to always remember to exclude it from different analyzers.
• If it grows beyond just perf testing and will include integration tests (that can and should serve as perf tests as well).
• if we decide to save results via DVC (and we should), save data that is used to test it (and again we should!) it means that dvc core repo will handle all of this, will potentially have a branch to run and save results - not nice.

Also, it feels natural - because it should be decoupled from a specific DVC installation it is testing - it should be one of the core functions to test any of the provided DVC packages (pip, deb, etc) and this logic is a higher level.

[Suor]

I see it completely differently. @shcheklein brought a long list of things that are potentially desirable, it is however a big question whether we really need all of them or will need at some point. Even if we will decide that we do need all of those then it's non-obvious when it will be implemented, it might easily turn into another dead horse like dvc-test repo is.

I am proposing to start from something, which can benefit us immediately and which could be started written right away, - bench scenarios to be run manually to reproduce performance issues. And that makes little sense to separate those into some other repo. If in the future we will implement the sophisticated system proposed by Ivan we can reuse these bench scenarios either moving them out or not.

[shcheklein]

@Suor I can agree that some things are advanced and it'll take a while to build them. Some of the things in that list should be done day zero/day one (e.g. automation, reporting on a nightly basis from the same environment, handling data to run these tests, etc). They will already require creating a proper framework.

Also, I still don't any reasonable downsides starting it as a separate repo. No one mentioned it to me still. All the arguments (easy to run, benefit immediately, etc) don't contradict starting with a separate project.

[pared]

I would like to support @shcheklein on this one.
Some tests might require us to create dvc repo, first real-use case scenario that comes to my mind is timing adding some well known dataset (CIFAR, MNIST or even ImageNet). Do we really want to keep sample repos under main repo?

If we want simple benchmark that starts benefiting us immediately, why not create sub-repos under
dvc-test and just time -ing dvc repro each one?

I also don't see how would we store performance tests results in main dvc repo in a way that would be easily accessible. dvc-test would be our playground, so we could even embed pyplot charts in README.

Why don't we start right away with a separate directory (looks like almost everyone is in favor) and then move it to this repository if needed (or if it turns out as a simpler solution)?

[jorgeorpinel]

+1 for separate iterative/dvcbench repo

Side note: If we could figure out a way to extract the time stats from each passed build in https://travis-ci.com/iterative/dvc/pull_requests. E.g.

could that give us an idea of how the performance of dvc has evolved so far?

[pared]

@jorgeorpinel Hmm, that could be something that we should be monitoring, but main problem with build time (so far) was setup of environment as I recall, and @efiop was handling that in few pull requests. (#2361) I would agree that monitoring build time is important, but I would not consider it performance metric. We could try to extract tests run time, thought (pytest reports that in log), that is something that could be performance metric.

[shcheklein]

@pared @jorgeorpinel yep, also, it's not a good metric since number of tests is changing.

@iterative/engineering is there an agreement that we can commit to on this one?