Add blazemeter taurus bench-script #148

rflorenc · 2016-01-27T11:00:10Z

This adds support for executing blazemeter taurus (bzt) performance tests.

The main dependency is bzt, which can be installed using the pip package management system for python.

http://gettaurus.org/docs/Installation/#CentOS

jeremyeder · 2016-01-27T12:15:10Z

Thanks Ricardo, here are my notes from looking it over:

Can you commit a README file with documentation, including some of the more complex options?
There's no --help option?
RHEL ships PyYAML-3.10-11. I hope that will suffice?
Usage of pip for bzt...this would be the first time we're using pip in pbench. For the rest of the benchmarks, we've created RPMs. I think we should be consistent about that.
Looking at http://gettaurus.org/docs/Installation/ I see we will need a few other packages pulled in as well (wait, are these dependencies for building bzt via pip? Anyway I think right now pbench_benchmark scripts automatically install any dependencies via yum/rpm. This script should do the same.

portante · 2016-01-27T14:26:28Z

@rflorenc, thanks this is a great start. We can work through the pip -> rpm dependencies.

atheurer · 2016-01-27T15:13:32Z

Really glad to see a new benchmark script here. I think this is a good time to create a checklist for benchmarks. We can maintain this list in the top of the benchmark script, so users know what common pbench features are present in that script. For example:

[y/n] package dependency/installation management
[y/n] tools start/stop/post-processing calls
[y/n] multiple benchmark iterations
[y/n] multiple test samples per benchmark iteration (to compute average and stddev)
[y/n] unittest
[y/n] iteration result metrics in indexable json format (no benchmarks have this yet)

others?

Good examples of the first 5 features are in pbench_uperf

The travis build environment (ubuntu precise) supports python3.2 only. Recent pip apparently has *stopped* supporting python3.2. We had to go through hoops to make the previous environment work, but that is now broken. So we go through a different set of hoops that make the build work, but there now seem to be unit test failures that were not there before: they are under investigation.

rflorenc · 2016-01-28T09:46:52Z

Thank you all for the feedback!

@jeremyeder ,

Sure, I'll commit the README file soon.
The help option is already included. Run it with -h or --help.
Yes, that version of PyYAML is OK.
Yes, those are the base dependencies required to install bzt through pip.

The indexing tests are failing in the new travis-ci environment: the pretty-printing is done slightly differently locally, causing spurious failures. I'll revisit them after the 0.36 pbench-agent release (which does not depend on these tests at all).

Fix travis build.

Added a description in the beginning of the file similar to the one found in the other bench tools.

rflorenc · 2016-01-29T14:26:19Z

I totally agree that we should be consistent with the way the rest of the bench scripts are solving their dependencies through rpm, but in this case since it's a python wrapper for an external tool (also python) you would potentially avoid problems in the future.

This is where bzt takes its dependencies from:
[https://github.com/Blazemeter/taurus/blob/master/setup.py#L66]

Pip takes care of updating/upgrading the app as needed from the python package index.

It only output the first one, but it is called with "$@", which expands to multiple arguments. Opportunistically, delete duplicate run of util-scripts unittests from .travis.yml.

Make debug_log deal with all its arguments.

…yeder/pbench into jeremyeder-fix_openvswitch_dump_ports

Try first without a protocol argument, but if that fails, try with OpenFlow13.

…orts Jeremyeder fix openvswitch dump ports

This convention was used in the util scripts renaming, and will be the established convention henceforth.

Clarified message logged for ssh connection error

Rename benchmark scripts from pbench_foo to pbench-foo.

All the util scripts now have a prefix of pbench-. Move non-pbench scripts to contrib/ subdirectory. contrib/ is not added to the PATH so there will be no collision. If users want to use one or more of these, they will need to call them by full pathname or add contrib/ to their PATH. Add a backward-compatibility wrapper temporarily. We maintain compatibility with the old util-scripts names *for one release only*. The old names will be symlinked to this wrapper in a postinstall action when the RPM is installed. The wrapper does some minimal checking, issues a big fat warning and then executes the script under its new name.

Rename util scripts to avoid collisions.

…scenario files. Added initial platform detection code.

rflorenc · 2016-02-08T10:15:39Z

@jeremyeder , @portante
PyYAML module is no longer a dependency, since bzt accepts json scenario files as input.
Added some initial code to facilitate rpm installs based on the linux distro.

jeremyeder · 2016-02-09T11:52:04Z

@rflorenc ok -- I read the readme. The content is good, I just wonder if it's obvious enough being inside the pbench_bzt file. Maybe we need man pages, or to put README_pbench_bzt.md in the higher-level "doc" folder in pbench? I think there is markdown2man converter somewhere that could make this very easy.

Anyway...

@atheurer has laid out a list of requirements, where does pbench_bzt stand on these 6 items?
#148 (comment)

rflorenc · 2016-02-09T15:32:58Z

We could have several README_pbench_$tool.md under pbench/agent/bench-scripts/doc with the initial description every bench script has. And maybe later turn those into man pages.

The scripts would just contain the checklist @atheurer suggested, in the beginning.

Rename user-benchmark to pbench-user-benchmark.

reduce sampling interval for perf to 100Hz

Its functions have been taken over by pbench-clear-tools: pbench-clear-tools --group=foo --name=bar will delete the "bar" tool from the "foo" tool group. With no options, it will clear all tools from all groups. With only a --name option, it will delete the named tool from the default group *only*. Fix typo in pbench-bc-wrapper.

Delete obsolete pbench-unregister-tools.

fix syntax to freq argument

Version bump.

ndokos · 2016-02-16T18:46:00Z

Can you rebase on latest master and rename it to pbench-bzt?

Added a description in the beginning of the file similar to the one found in the other bench tools.

…scenario files. Added initial platform detection code.

…pbench_bzt

rflorenc · 2016-02-18T11:02:02Z

@ndokos , done.

ndokos · 2016-02-18T15:47:04Z

Thanks!

ndokos · 2016-02-25T17:54:57Z

I think the plan for moving ahead with this is:

Create a bench-scripts/config subdir and move pbench-bzt into it (the contrib directory is not in PATH).
To use it currently, invoke it by its full pathname under pbench-user-benchmark:

pbench-user-benchmark <opts> -- /opt/pbench-agent/bench-scripts/contrib/pbench-bzt ...

When pbench-bzt becomes a full benchscript (start/stop/postprocess etc), then it can be moved back into pbench-scripts.
At that time, it will need unit tests.

Agreed?

rflorenc · 2016-02-25T19:33:21Z

Yes, thanks. Meanwhile I'll look further into the missing bits.

ndokos · 2016-03-21T21:35:16Z

I cleaned up the history and moved the script to bench-scripts/contrib. I did that in PR #207 since I wanted to get this in v0.37. Apologies if I'm out of order. Please check #207 to make sure that everything is as it should be. Thanks! Closing this.

jeremyeder and others added 2 commits January 14, 2016 13:41

must specify openflow version

8222c22

Add blazemeter taurus bench-script

66a2cbb

ndokos and others added 3 commits January 28, 2016 11:23

Leave out server tests 7.6 and 7.7.

c7c5451

The indexing tests are failing in the new travis-ci environment: the pretty-printing is done slightly differently locally, causing spurious failures. I'll revisit them after the 0.36 pbench-agent release (which does not depend on these tests at all).

Merge pull request #150 from ndokos/wip-travis

e91d2f1

Fix travis build.

add README

023b018

Added a description in the beginning of the file similar to the one found in the other bench tools.

ndokos and others added 12 commits February 2, 2016 13:01

Make debug_log deal with all its arguments.

c791a53

It only output the first one, but it is called with "$@", which expands to multiple arguments. Opportunistically, delete duplicate run of util-scripts unittests from .travis.yml.

Merge pull request #136 from ndokos/wip-debug-log-args

b7b1e74

Make debug_log deal with all its arguments.

Merge branch 'fix_openvswitch_dump_ports' of https://github.com/jerem…

fcd54ba

…yeder/pbench into jeremyeder-fix_openvswitch_dump_ports

Check ovs-ofctl exit status.

a85efef

Try first without a protocol argument, but if that fails, try with OpenFlow13.

Merge pull request #153 from ndokos/jeremyeder-fix_openvswitch_dump_p…

a5ec4d8

…orts Jeremyeder fix openvswitch dump ports

Clarified message logged for ssh connection error

1b8b1bf

Rename benchmark scripts from pbench_foo to pbench-foo.

c464409

This convention was used in the util scripts renaming, and will be the established convention henceforth.

Merge pull request #155 from ofthecure/master

4f79fbc

Clarified message logged for ssh connection error

Merge pull request #156 from ndokos/wip-rename-benchscripts

a73e562

Rename benchmark scripts from pbench_foo to pbench-foo.

Merge pull request #130 from ndokos/wip-rename-util-scripts

d8e2104

Rename util scripts to avoid collisions.

Removed yaml (PyYaml) dependency. pbench_bzt now uses/generates json …

b6725a1

…scenario files. Added initial platform detection code.

ndokos and others added 4 commits February 9, 2016 15:52

Rename user-benchmark to pbench-user-benchmark.

bfd3fa3

Merge pull request #159 from ndokos/wip-rename-user-benchmarks

5ce1c37

Rename user-benchmark to pbench-user-benchmark.

reduce sampling interval for perf to 100Hz

dc2eadd

Merge pull request #161 from jeremyeder/perf-freq-100

126e2e8

reduce sampling interval for perf to 100Hz

ndokos and others added 6 commits February 10, 2016 17:44

Merge pull request #162 from ndokos/wip-clear-tools

87bfdf1

Delete obsolete pbench-unregister-tools.

fix syntax to freq argument

dcbaa8c

Merge pull request #163 from jeremyeder/fix-perf-100-patch

d82b80e

fix syntax to freq argument

Version bump.

b93dd9b

Merge pull request #169 from ndokos/wip-version-0.36

6be09f9

Version bump.

rflorenc and others added 10 commits February 18, 2016 10:26

Rename pbench_bzt to pbench-bzt

ce44847

Include description

013ec61

Removed old pbench_bzt

f1ffba1

Add blazemeter taurus bench-script

e983540

add README

8468fc6

Added a description in the beginning of the file similar to the one found in the other bench tools.

Removed yaml (PyYaml) dependency. pbench_bzt now uses/generates json …

934bf35

…scenario files. Added initial platform detection code.

Rename pbench_bzt to pbench-bzt

ecabc91

Include description

53bc5cf

Removed old pbench_bzt

bb0ae45

Merge branch 'pbench_bzt' of https://github.com/rflorenc/pbench into …

3d74cbd

…pbench_bzt

ndokos mentioned this pull request Mar 21, 2016

Add blazemeter taurus script to bench-scripts/contrib. #204

Merged

ndokos closed this Mar 21, 2016

portante self-assigned this May 9, 2022

Add blazemeter taurus bench-script #148

Add blazemeter taurus bench-script #148

Uh oh!

Conversation

rflorenc commented Jan 27, 2016

Uh oh!

jeremyeder commented Jan 27, 2016

Uh oh!

portante commented Jan 27, 2016

Uh oh!

atheurer commented Jan 27, 2016

Uh oh!

rflorenc commented Jan 28, 2016

Uh oh!

rflorenc commented Jan 29, 2016

Uh oh!

rflorenc commented Feb 8, 2016

Uh oh!

jeremyeder commented Feb 9, 2016

Uh oh!

rflorenc commented Feb 9, 2016

Uh oh!

ndokos commented Feb 16, 2016

Uh oh!

rflorenc commented Feb 18, 2016

Uh oh!

ndokos commented Feb 18, 2016

Uh oh!

ndokos commented Feb 25, 2016

Uh oh!

rflorenc commented Feb 25, 2016

Uh oh!

ndokos commented Mar 21, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants