Benchmark restructuring and memory profiling #642

benchislett · 2021-03-03T05:37:56Z

Issues

Problems I've been having with benchmarks:

Unclear which tests exist, and for what. Unclear what each file contains, which led to duplication of benchmark code
Unable to isolate setup and runtime for a given benchmark, so that higher-order composition can yield analysis over arbitrary system metrics (thus enabling memory and network profiling) (Memory benchmarks #638 [FEATURE] Benchmarking memory #516)
No consistent way to run benchmarks
No clean way to tinker with visualizations of benchmarks

Progress

First thing is to refactor all the existing benchmarks. The style I went for is benchmark_title.py which contains two exported functions benchmark_title_setup which takes many parameters, and benchmark_title_run which takes a parameter tuple returned by the previous setup call. All files have been refactored in this way excepting the tiledb/zarr benchmarks, see c7bbbd0 (ping @DebadityaPal). In the future, if our benchmarks become sufficiently complex, we can further refactor these into classes.

Next we need to sort out which suites we want to run and when, how we want to store output, and what we want to profile. Time is obvious, but also memory and network. As of right now, time, memory, and network benchmarks are all working. We'll need to choose exactly which suites we want to run and store, but this is now a trivial extension.

The next step after this PR is to configure the full suite, output to a file (yaml or similar), and run the visualizations on that (or do them in the same notebook and store them as artifacts as well).

TL;DR

Time, memory, network benchmarks for all metrics are live. This PR is ready to go!

Plan: each benchmark will export two functions, filename_setup and filename_run, which designate how the benchmark is to be run. setup will take any required parameters and return a processed param tuple to be passed as argument to the runner. The runner is designed to be as slim as possible so we only measure the crucial code. Then, we can externally call these functions and time/profile/benchmark on the runtime of the function call, allowing for a great increase of control. Further refactors along this design coming soon.

Following the previous commit, this refactors the benchmark_dataset_iter into separate files with the same design as the now-refactored `benchmark_compress_hub.py`. One step closer to full control

It'll be nice to keep track of this as well. Might be subsumed by the dataset_comparison file, but I'll get to that next.

Improves `benchmark_access_hub_full.py` and uses that as a base for `benchmark_access_hub_slice.py` which replaces functionality from `benchmark_random_access.py` (now deleted).

Existing refactored benchmarks now cover all cases once present in this file.

Until these can be converted, I want to have a distinction to know what is and isn't compatible with the new runner (next few commits). This will probably be fixed before going in

github-actions · 2021-03-03T05:38:34Z

Locust summary

Git references

Initial: c214a53
Terminal: 63504ce

benchmarks/benchmark_access_hub_full.py

Changes:

Name: benchmark_access_hub_full_setup
Type: function
Changed lines: 7
Total lines: 7
Name: benchmark_access_hub_full_run
Type: function
Changed lines: 4
Total lines: 4

benchmarks/benchmark_access_hub_slice.py

Changes:

Name: benchmark_access_hub_slice_setup
Type: function
Changed lines: 7
Total lines: 7
Name: benchmark_access_hub_slice_run
Type: function
Changed lines: 4
Total lines: 4

benchmarks/benchmark_compress_hub.py

Changes:

Name: benchmark_compress_hub_setup
Type: function
Changed lines: 15
Total lines: 15
Name: benchmark_compress_hub_run
Type: function
Changed lines: 3
Total lines: 3

benchmarks/benchmark_compress_pillow.py

Changes:

Name: benchmark_compress_pillow_setup
Type: function
Changed lines: 3
Total lines: 3
Name: benchmark_compress_pillow_run
Type: function
Changed lines: 5
Total lines: 5

benchmarks/benchmark_iterate_hub_local_pytorch.py

Changes:

Name: benchmark_iterate_hub_local_pytorch_setup
Type: function
Changed lines: 10
Total lines: 10
Name: benchmark_iterate_hub_local_pytorch_run
Type: function
Changed lines: 4
Total lines: 4

benchmarks/benchmark_iterate_hub_local_tensorflow.py

Changes:

Name: benchmark_iterate_hub_local_tensorflow_setup
Type: function
Changed lines: 9
Total lines: 9
Name: benchmark_iterate_hub_local_tensorflow_run
Type: function
Changed lines: 4
Total lines: 4

benchmarks/benchmark_iterate_hub_pytorch.py

Changes:

Name: benchmark_iterate_hub_pytorch_setup
Type: function
Changed lines: 7
Total lines: 7
Name: benchmark_iterate_hub_pytorch_run
Type: function
Changed lines: 4
Total lines: 4

benchmarks/benchmark_iterate_hub_tensorflow.py

Changes:

Name: benchmark_iterate_hub_tensorflow_setup
Type: function
Changed lines: 5
Total lines: 5
Name: benchmark_iterate_hub_tensorflow_run
Type: function
Changed lines: 4
Total lines: 4

benchmarks/legacy_benchmark_sequential_access.py

Changes:

Name: time_batches
Type: function
Changed lines: 23
Total lines: 23
Name: time_tiledb
Type: function
Changed lines: 56
Total lines: 56
Name: time_zarr
Type: function
Changed lines: 36
Total lines: 36
Name: time_hub
Type: function
Changed lines: 11
Total lines: 11

benchmarks/legacy_benchmark_sequential_write.py

Changes:

Name: time_batches
Type: function
Changed lines: 25
Total lines: 25
Name: time_tiledb
Type: function
Changed lines: 25
Total lines: 25
Name: time_zarr
Type: function
Changed lines: 12
Total lines: 12
Name: time_hub
Type: function
Changed lines: 22
Total lines: 22

DebadityaPal · 2021-03-03T10:06:17Z

@benchislett I'll change the zarr/tiledb benchmarks to follow the style the other benchmarks are following.

haiyangdeperci · 2021-03-08T06:38:11Z

I'll take a look on the benchmarks, thanks a lot for your wonderful input.
@benchislett Could you run fix the linting in the meantime?

benchislett · 2021-03-08T15:30:04Z

Yeah, I haven't been able to get the linter going nicely in my IDE so I usually just save it until my last commit and fix it all then. Coming soon.

haiyangdeperci · 2021-03-08T15:33:23Z

@benchislett sure, no worries 👍

haiyangdeperci · 2021-03-24T22:42:14Z

@benchislett Could you give me an estimate with regards to when you will incorporate these changes?

benchislett · 2021-03-25T02:24:47Z

As soon as I get a spare couple hours. Probably this weekend. By Monday at the latest

haiyangdeperci · 2021-03-25T09:53:11Z

@benchislett Amazing! Thank you. Monday sounds great!

haiyangdeperci · 2021-03-29T11:47:38Z

@benchislett Have you encountered any obstacles while working on this PR? Perhaps, I could help.

benchislett · 2021-03-29T15:36:39Z

@benchislett Have you encountered any obstacles while working on this PR? Perhaps, I could help.

Thanks, but there really haven't been any issues other than a distinct lack of time. This weekend was crunch time for me at work and I was busy non-stop. I'll have my commits in tonight and we can go from there

haiyangdeperci · 2021-03-29T16:04:45Z

@benchislett No worries, I just wanted to make sure everything is all right. That's great. Looking forward to the update 💯

haiyangdeperci · 2021-03-30T09:29:10Z

@benchislett Thanks for the changes in the notebook, I quite appreciate those. Would you mind fixing the linting though? Just run black over the code, I assume.

benchislett added 11 commits March 2, 2021 22:45

Refactor dataset iteration benchmarks

50df193

Following the previous commit, this refactors the benchmark_dataset_iter into separate files with the same design as the now-refactored `benchmark_compress_hub.py`. One step closer to full control

Add full dataset compute benchmark

fd31f34

It'll be nice to keep track of this as well. Might be subsumed by the dataset_comparison file, but I'll get to that next.

Refactor benchmark_random_access into new format

85ac898

Improves `benchmark_access_hub_full.py` and uses that as a base for `benchmark_access_hub_slice.py` which replaces functionality from `benchmark_random_access.py` (now deleted).

Remove unused line in benchmark_iterate_hub TF

b4a49da

Local variants of iteration benchmarks using tfds

1fa0c7d

Remove dataset compare benchmarks

a2d8206

Existing refactored benchmarks now cover all cases once present in this file.

Rename remaining un-refactored benchmarks "legacy"

c7bbbd0

Until these can be converted, I want to have a distinction to know what is and isn't compatible with the new runner (next few commits). This will probably be fixed before going in

Fix minor issues in total access benchmarks

e8535d1

Initial prototype for benchmark runner notebook

698ff7c

Update benchmark runner notebook

63504ce

benchislett mentioned this pull request Mar 3, 2021

Memory benchmarks #638

Closed

benchislett added 5 commits March 29, 2021 19:48

Merge remote-tracking branch 'upstream/master'

ecbe360

Add psutil to benchmark requirements

b222c73

Fix pytorch and tensorflow local benchmarks

f4d9135

Add network benchmarking and expand suites

08d6c4f

Update .gitignore with benchmark local data

abb18f6

haiyangdeperci self-requested a review March 30, 2021 14:08

benchislett added 2 commits March 30, 2021 10:21

Auto-fix issues with black

2eecc18

Add time to network monitor output to plot better

86ccd58

haiyangdeperci approved these changes Mar 30, 2021

View reviewed changes

haiyangdeperci merged commit da105b0 into activeloopai:master Mar 30, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Benchmark restructuring and memory profiling #642

Benchmark restructuring and memory profiling #642

benchislett commented Mar 3, 2021 •

edited

Loading

github-actions bot commented Mar 3, 2021

DebadityaPal commented Mar 3, 2021

haiyangdeperci commented Mar 8, 2021

benchislett commented Mar 8, 2021

haiyangdeperci commented Mar 8, 2021

haiyangdeperci commented Mar 24, 2021

benchislett commented Mar 25, 2021

haiyangdeperci commented Mar 25, 2021

haiyangdeperci commented Mar 29, 2021

benchislett commented Mar 29, 2021

haiyangdeperci commented Mar 29, 2021

haiyangdeperci commented Mar 30, 2021

Benchmark restructuring and memory profiling #642

Benchmark restructuring and memory profiling #642

Conversation

benchislett commented Mar 3, 2021 • edited Loading

Issues

Progress

TL;DR

github-actions bot commented Mar 3, 2021

Locust summary

Git references

DebadityaPal commented Mar 3, 2021

haiyangdeperci commented Mar 8, 2021

benchislett commented Mar 8, 2021

haiyangdeperci commented Mar 8, 2021

haiyangdeperci commented Mar 24, 2021

benchislett commented Mar 25, 2021

haiyangdeperci commented Mar 25, 2021

haiyangdeperci commented Mar 29, 2021

benchislett commented Mar 29, 2021

haiyangdeperci commented Mar 29, 2021

haiyangdeperci commented Mar 30, 2021

benchislett commented Mar 3, 2021 •

edited

Loading