Support for test listing, running and metrics via v3 #14

diego-plan9 · 2017-05-03T14:51:06Z

Additions to the cli for listing Tests, starting (with streaming) TestRuns, and listing Metrics, in tandem with the changes on the SDK at li-clutter-org/loadimpact-sdk-python#7.

Add a "test" click command group containing a "list" command for displaying an overview of the tests for the projects of the organizations the user has access to (by default); or only for the tests of particular projects via a project_id parameter.

Add a "metric" click command group containing a "list" command for displaying an overview of the metrics for a test run; optinally filterng by metric type.

Add metric streaming, via a configurable "--quiet" flag and multiple "--metric" parameters.

coveralls · 2017-05-03T14:53:59Z

Coverage increased (+0.2%) to 73.698% when pulling 8d052a4 on feature/apiv3-test-support into e68ca92 on develop.

robingustafsson · 2017-05-04T23:11:18Z

I'm testing this. Ran into an issue when listing my large list of tests :-)

Traceback (most recent call last):
  File "/usr/local/bin/loadimpact", line 9, in <module>
    load_entry_point('loadimpact-cli==1.0.6', 'console_scripts', 'loadimpact')()
  File "build/bdist.macosx-10.12-intel/egg/loadimpactcli/loadimpact_cli.py", line 41, in run_cli
  File "/Library/Python/2.7/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/Library/Python/2.7/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/Library/Python/2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Python/2.7/site-packages/click/core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Python/2.7/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Library/Python/2.7/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "build/bdist.macosx-10.12-intel/egg/loadimpactcli/test_commands.py", line 57, in list_tests
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe3' in position 59: ordinal not in range(128)

Change the format for the "config summary" column (to "XX users YYs") and colorize the "last run status" column on "test list" output.

coveralls · 2017-05-05T13:04:08Z

Coverage increased (+0.9%) to 74.396% when pulling 5205ca0 on feature/apiv3-test-support into e68ca92 on develop.

Introduce a `util.Metric` class for converting between different representations (raw format expected by the api, user friendly name, parameter name). Revise `metric list` command for displaying the name of the metric when used as a parameter. Revise `test run` syntax and output, displaying the metric values in a single row, and accepting standard metrics and custom metrics.

Add unit tests for `util.Metric` (along with its usage) and for the expected flow of calls during metric streaming, fixing a number of issues in the process: * fix unicode handling on test_commands * add equality methods to metrics and metric types * fix parameter handling for non-standard metrics, not appending the load zone identifier automatically * revise docstrings and minor tweaks

coveralls · 2017-05-10T12:59:53Z

Coverage increased (+6.6%) to 80.119% when pulling 4d4ca9a on feature/apiv3-test-support into e68ca92 on develop.

robingustafsson

Looking good, added some comments regarding general UX of output

robingustafsson · 2017-05-19T10:20:18Z

loadimpactcli/test_commands.py

+
+            stream = test_run.result_stream([m.str_raw(True) for m in metrics])
+
+            click.echo(pprint_header(metrics))


I think we should wait with printing the header until we get the first results and instead print "Initializing test..." or equivalent until the first results arrive (maybe even print the "live feedback" messages until we reach the "running" state).

robingustafsson · 2017-05-19T10:23:13Z

loadimpactcli/test_commands.py

+    Return a pretty printed string with the header of the metric streaming
+    output, consisting of the name of the metrics including the parameters.
+    """
+    return u'\t'.join([u'TIMESTAMP:'] + [u'{0}:'.format(m.str_ui(True)) for m in returned_metrics])


Is there some way to make the header columns better align with the position of the result rows? Would it be bad to reserve a fixed space for each column?

Also could we perhaps insert the header every 20 or so result rows, not to forget what each column represents. Thoughts?

I mostly went with using a simple \t between columns as it seemed to provide consistency with the rest of the commands's output, but at the expense of readability indeed! I will play a bit with the aligning, as I fully agree it would be a welcome and doable improvement.

robingustafsson · 2017-05-19T10:26:19Z

loadimpactcli/test_commands.py

+
+@test.command('list', short_help='List tests.')
+@click.option('--project_id', 'project_ids', multiple=True, help='Id of the project to list tests from.')
+def list_tests(project_ids):


Could we limit this to the 10-20 or so tests with the most recent test runs and then add a line of output saying how to print/list more tests (using an --all or --limit flag perhaps)? This would greatly enhance the UX for someone like me who has 100s of tests across orgs and projects 😄

Yes! Did not account for such hard core users, but it does make much sense.

robingustafsson · 2017-05-19T10:28:06Z

loadimpactcli/test_commands.py

+        for id_ in set(project_ids):
+            tests.extend(client.list_tests(project_id=id_))
+
+        click.echo("ID:\tNAME:\tLAST RUN DATE:\tLAST RUN STATUS:\tCONFIG:")


Same as comment below regarding fixed column width, is that possible? Saying for example we show the first 32 chars of the test name and then add "...", maybe have a flag --full to show full information?

Add a `--limit` argument (defaulting to 20) that controls the number of tests that are displayed during `test list`. Fix PEP8 issues and a small bug on userscenario.

coveralls · 2017-05-23T12:35:54Z

Coverage increased (+6.7%) to 80.276% when pulling eaae558 on feature/apiv3-test-support into e68ca92 on develop.

Change the output of the "test run" command, showing an initialization message once the test run is requested and delay printing the header until the actual metrics start to arrive, additionally printing them every 20 lines for clarity. Update docs.

Use fixed widths for the columns on the output of the `test list` command, introducing a `util.ColumnFormatter` helper and aiming for reasonable defaults for each column's width. The previous behaviour (using just '\t' as a separator between each column, with no truncating or further processing of the cells) is preserved by introducing a `--full_width` flag.

Use fixed with and truncation (optionally disabled by `--full_width`) for the `test run` command, analogously to the output of `test list`.

coveralls · 2017-05-24T10:34:27Z

Coverage increased (+7.5%) to 81.006% when pulling ad82357 on feature/apiv3-test-support into e68ca92 on develop.

coveralls · 2017-05-24T11:26:09Z

Coverage increased (+7.5%) to 81.006% when pulling 9b6eea8 on feature/apiv3-test-support into e68ca92 on develop.

coveralls · 2017-05-24T11:26:09Z

Coverage increased (+7.5%) to 81.006% when pulling 9b6eea8 on feature/apiv3-test-support into e68ca92 on develop.

coveralls · 2017-05-24T12:10:35Z

Coverage increased (+7.6%) to 81.146% when pulling c00e6e0 on feature/apiv3-test-support into e68ca92 on develop.

Diego M. Rodriguez added 12 commits April 21, 2017 15:26

[feat] add "test list" command

ae8dded

Add a "test" click command group containing a "list" command for displaying an overview of the tests for the projects of the organizations the user has access to (by default); or only for the tests of particular projects via a project_id parameter.

[feat] add "test run" command

c0922b1

[feat] add "metric list" command

3142fdc

Add a "metric" click command group containing a "list" command for displaying an overview of the metrics for a test run; optinally filterng by metric type.

[fix] use post data correctly on filtering metrics

7fdacba

[feat] add streaming to the "test run" command

f650dd9

Add metric streaming, via a configurable "--quiet" flag and multiple "--metric" parameters.

[fix] more useful output of "test run" stream

2f1ed2a

[fix] summarize config on "test list", sort

25aada2

[test] tests for test commands, fix function name

76e7666

[test] tests for metric commands

facd772

[feat] show last run status on test list output

a5cd76e

[doc] update README.md with new commands

f75479b

[fix] fix python3 comp., metric ordering

8d052a4

diego-plan9 requested review from martinfijal and robingustafsson May 3, 2017 14:51

Diego M. Rodriguez added 2 commits May 5, 2017 11:18

[fix] use unicode for commands output

9fe1786

[feat] test list: change config summary, colorize

5205ca0

Change the format for the "config summary" column (to "XX users YYs") and colorize the "last run status" column on "test list" output.

Diego M. Rodriguez added 4 commits May 9, 2017 16:01

[docs] update README.md with metric changes

8734baf

[chore] add enum dependency for circleci

6e46bae

robingustafsson reviewed May 19, 2017

View reviewed changes

[feat] limit number of displayed tests during list

eaae558

Add a `--limit` argument (defaulting to 20) that controls the number of tests that are displayed during `test list`. Fix PEP8 issues and a small bug on userscenario.

Diego M. Rodriguez added 3 commits May 23, 2017 15:09

[feat] improve column display of "test run"

3dcc8ba

Use fixed with and truncation (optionally disabled by `--full_width`) for the `test run` command, analogously to the output of `test list`.

[style] PEP8 adjustments

ad82357

[fix] fix python3 unicode conversion

9b6eea8

[fix] fix python3 unicode conversion

c00e6e0

robingustafsson merged commit 3e7a8bd into develop Oct 3, 2017

robingustafsson deleted the feature/apiv3-test-support branch October 27, 2017 16:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for test listing, running and metrics via v3 #14

Support for test listing, running and metrics via v3 #14

diego-plan9 commented May 3, 2017

coveralls commented May 3, 2017 •

edited

Loading

robingustafsson commented May 4, 2017 •

edited

Loading

coveralls commented May 5, 2017 •

edited

Loading

coveralls commented May 10, 2017 •

edited

Loading

robingustafsson left a comment

robingustafsson May 19, 2017

robingustafsson May 19, 2017

diego-plan9 May 19, 2017

robingustafsson May 19, 2017

diego-plan9 May 19, 2017

robingustafsson May 19, 2017

coveralls commented May 23, 2017 •

edited

Loading

coveralls commented May 24, 2017

coveralls commented May 24, 2017

coveralls commented May 24, 2017 •

edited

Loading

coveralls commented May 24, 2017 •

edited

Loading


		stream = test_run.result_stream([m.str_raw(True) for m in metrics])

		click.echo(pprint_header(metrics))

Support for test listing, running and metrics via v3 #14

Support for test listing, running and metrics via v3 #14

Conversation

diego-plan9 commented May 3, 2017

coveralls commented May 3, 2017 • edited Loading

robingustafsson commented May 4, 2017 • edited Loading

coveralls commented May 5, 2017 • edited Loading

coveralls commented May 10, 2017 • edited Loading

robingustafsson left a comment

Choose a reason for hiding this comment

robingustafsson May 19, 2017

Choose a reason for hiding this comment

robingustafsson May 19, 2017

Choose a reason for hiding this comment

diego-plan9 May 19, 2017

Choose a reason for hiding this comment

robingustafsson May 19, 2017

Choose a reason for hiding this comment

diego-plan9 May 19, 2017

Choose a reason for hiding this comment

robingustafsson May 19, 2017

Choose a reason for hiding this comment

coveralls commented May 23, 2017 • edited Loading

coveralls commented May 24, 2017

coveralls commented May 24, 2017

coveralls commented May 24, 2017 • edited Loading

coveralls commented May 24, 2017 • edited Loading

coveralls commented May 3, 2017 •

edited

Loading

robingustafsson commented May 4, 2017 •

edited

Loading

coveralls commented May 5, 2017 •

edited

Loading

coveralls commented May 10, 2017 •

edited

Loading

coveralls commented May 23, 2017 •

edited

Loading

coveralls commented May 24, 2017 •

edited

Loading

coveralls commented May 24, 2017 •

edited

Loading