Skip to content

Conversation

@narasimhan-v
Copy link
Collaborator

When multiple tests are run, along with kernel messages output,
it becomes difficult to read the status of the tests. This commits
tries to solve that by displaying the status in the summary.

Signed-off-by: Narasimhan V sim@linux.vnet.ibm.com

@narasimhan-v
Copy link
Collaborator Author

# python avocado-setup.py --run-suite host_io_fc_fvt
13:29:21 INFO    : Check for environment
13:29:21 INFO    : Creating temporary mux dir
13:29:23 INFO    : 
13:29:23 INFO    : Running Host Tests Suite io_fc_fvt_fs_mark_fs_mark
13:29:23 INFO    : Running: /usr/local/bin/avocado run avocado-misc-tests/io/disk/fs_mark.py -m /root/sim/tests/tests/avocado-misc-tests/io/disk/fs_mark.py.data/fs_mark.yaml --force-job-id 083ddd31ec8b3dec4b008c86ebc2d49e8348b36a  --job-results-dir /root/sim/tests/results
/usr/local/lib/python3.6/site-packages/avocado_framework-78.0-py3.6.egg/avocado/plugins/run.py:287: FutureWarning: The following arguments will be changed to boolean soon: sysinfo, output-check, failfast, keep-tmp and ignore-missing-references.
  FutureWarning)
Error running method "pre_tests" of plugin "fetchasset": 'Attribute' object has no attribute 'id'
JOB ID     : 083ddd31ec8b3dec4b008c86ebc2d49e8348b36a
JOB LOG    : /root/sim/tests/results/job-2020-04-21T13.29-083ddd3/job.log
 (1/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-ext4-1d3d: ERROR: 'FSMark' object has no attribute 'link' (1.21 s)
 (2/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-xfs-8b3d: ERROR: 'FSMark' object has no attribute 'link' (1.21 s)
 (3/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-btrfs-209e: CANCEL: btrfs is not supported with                                 RHEL 7.4 onwards (0.71 s)
RESULTS    : PASS 0 | ERROR 2 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 1
JOB TIME   : 3.46 s
JOB HTML   : /root/sim/tests/results/job-2020-04-21T13.29-083ddd3/results.html
13:29:27 INFO    : 
13:29:27 INFO    : 
13:29:27 INFO    : Running Host Tests Suite io_fc_fvt_fs_mark_fs_mark.2
13:29:27 INFO    : Running: /usr/local/bin/avocado run avocado-misc-tests/io/disk/fs_mark.py -m /root/sim/tests/tests/avocado-misc-tests/io/disk/fs_mark.py.data/fs_mark.yaml --force-job-id 510e502727f066cb881e8ab3f61bd82d0a72c21f  --job-results-dir /root/sim/tests/results
/usr/local/lib/python3.6/site-packages/avocado_framework-78.0-py3.6.egg/avocado/plugins/run.py:287: FutureWarning: The following arguments will be changed to boolean soon: sysinfo, output-check, failfast, keep-tmp and ignore-missing-references.
  FutureWarning)
Error running method "pre_tests" of plugin "fetchasset": 'Attribute' object has no attribute 'id'
JOB ID     : 510e502727f066cb881e8ab3f61bd82d0a72c21f
JOB LOG    : /root/sim/tests/results/job-2020-04-21T13.29-510e502/job.log
 (1/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-ext4-1d3d: ERROR: 'FSMark' object has no attribute 'link' (1.23 s)
 (2/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-xfs-8b3d: ERROR: 'FSMark' object has no attribute 'link' (1.21 s)
 (3/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-btrfs-209e: CANCEL: btrfs is not supported with                                 RHEL 7.4 onwards (0.72 s)
RESULTS    : PASS 0 | ERROR 2 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 1
JOB TIME   : 3.51 s
JOB HTML   : /root/sim/tests/results/job-2020-04-21T13.29-510e502/results.html
13:29:31 INFO    : 
13:29:32 INFO    : Summary of test results can be found below:
TestSuite                                                                   TestRun    Summary             
 
host_io_fc_fvt_fs_mark_fs_mark                                              Run        Successfully executed
/root/sim/tests/results/job-2020-04-21T13.29-083ddd3/job.log
|  total 3  ||  pass 0  ||  cancel 1  ||  errors 2  ||  failures 0  ||  skip 0  |
 
host_io_fc_fvt_fs_mark_fs_mark.2                                            Run        Successfully executed
/root/sim/tests/results/job-2020-04-21T13.29-510e502/job.log
|  total 3  ||  pass 0  ||  cancel 1  ||  errors 2  ||  failures 0  ||  skip 0  |
13:29:32 INFO    : Removing temporary mux dir
# python avocado-setup.py --run-suite host_io_fc_fvt --additional-args " --dry-run"
13:28:12 INFO    : Check for environment
13:28:12 INFO    : Creating temporary mux dir
13:28:13 INFO    : 
13:28:14 INFO    : Running Host Tests Suite io_fc_fvt_fs_mark_fs_mark
13:28:14 INFO    : Running: /usr/local/bin/avocado run avocado-misc-tests/io/disk/fs_mark.py -m /root/sim/tests/tests/avocado-misc-tests/io/disk/fs_mark.py.data/fs_mark.yaml --force-job-id 6b0c0143bd32b2d120d2008a021e73e8125170b4  --dry-run --job-results-dir /root/sim/tests/results
/usr/local/lib/python3.6/site-packages/avocado_framework-78.0-py3.6.egg/avocado/plugins/run.py:287: FutureWarning: The following arguments will be changed to boolean soon: sysinfo, output-check, failfast, keep-tmp and ignore-missing-references.
  FutureWarning)
JOB ID     : 6b0c0143bd32b2d120d2008a021e73e8125170b4
JOB LOG    : /root/sim/tests/results/job-2020-04-21T13.28-6b0c014/job.log
 (1/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-ext4-1d3d: CANCEL: Test cancelled due to --dry-run (0.60 s)
 (2/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-xfs-8b3d: CANCEL: Test cancelled due to --dry-run (0.24 s)
 (3/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-btrfs-209e: CANCEL: Test cancelled due to --dry-run (0.23 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 3
JOB TIME   : 1.39 s
JOB HTML   : /root/sim/tests/results/job-2020-04-21T13.28-6b0c014/results.html
13:28:16 INFO    : 
13:28:16 INFO    : 
13:28:16 INFO    : Running Host Tests Suite io_fc_fvt_fs_mark_fs_mark.2
13:28:16 INFO    : Running: /usr/local/bin/avocado run avocado-misc-tests/io/disk/fs_mark.py -m /root/sim/tests/tests/avocado-misc-tests/io/disk/fs_mark.py.data/fs_mark.yaml --force-job-id 72301915c3d5cfe5cd9ac4e6d42a6f88a9cf02c6  --dry-run --job-results-dir /root/sim/tests/results
/usr/local/lib/python3.6/site-packages/avocado_framework-78.0-py3.6.egg/avocado/plugins/run.py:287: FutureWarning: The following arguments will be changed to boolean soon: sysinfo, output-check, failfast, keep-tmp and ignore-missing-references.
  FutureWarning)
JOB ID     : 72301915c3d5cfe5cd9ac4e6d42a6f88a9cf02c6
JOB LOG    : /root/sim/tests/results/job-2020-04-21T13.28-7230191/job.log
 (1/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-ext4-1d3d: CANCEL: Test cancelled due to --dry-run (0.23 s)
 (2/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-xfs-8b3d: CANCEL: Test cancelled due to --dry-run (0.24 s)
 (3/3) avocado-misc-tests/io/disk/fs_mark.py:FSMark.test;dir-disk-btrfs-209e: CANCEL: Test cancelled due to --dry-run (0.24 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 3
JOB TIME   : 1.00 s
JOB HTML   : /root/sim/tests/results/job-2020-04-21T13.28-7230191/results.html
13:28:18 INFO    : 
13:28:18 INFO    : Summary of test results can be found below:
TestSuite                                                                   TestRun    Summary             
 
host_io_fc_fvt_fs_mark_fs_mark                                              Not_Run    Unable to find job log file

 
host_io_fc_fvt_fs_mark_fs_mark.2                                            Not_Run    Unable to find job log file

13:28:18 INFO    : Removing temporary mux dir

@narasimhan-v narasimhan-v requested a review from sathnaga April 21, 2020 08:44
Copy link
Collaborator

@PraveenPenguin PraveenPenguin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@narasimhan-v narasimhan-v force-pushed the results_summary_status branch from 46a1f5f to 236a0a2 Compare April 27, 2020 08:22
avocado-setup.py Outdated
result_link += "/job.log\n"
with open(result_json, encoding = "utf-8") as fp:
result_state = json.load(fp)
for state in ['total', 'pass', 'cancel', 'errors', 'failures', 'skip']:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add warn as well

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harish-24 WARN is not found in the json file. So I did not add it.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@narasimhan-v I see a warn in there,

{
    "cancel": 0,
    "debuglog": "/home/satheesh/data/gits/github/avocado/job-2020-03-30T12.39-8a51c2c/job.log",
    "errors": 0,
    "failures": 0,
    "interrupt": 0,
    "job_id": "8a51c2c2ab2e400fc5089b3e6fd26244e383427e",
    "pass": 1,
    "skip": 0,
    "tests": [
        {
            "end": 1585552154.4747539,
            "fail_reason": null,
            "id": "1-/bin/true",
            "logdir": "/home/satheesh/data/gits/github/avocado/job-2020-03-30T12.39-8a51c2c/test-results/1-_bin_true",
            "logfile": "/home/satheesh/data/gits/github/avocado/job-2020-03-30T12.39-8a51c2c/test-results/1-_bin_true/debug.log",
            "start": 1585552154.219356,
            "status": "PASS",
            "time": 0.2553977966308594,
            "whiteboard": ""
        }
    ],
    "time": 0.2553977966308594,
    "total": 1,
    "warn": 0

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, let me recheck and add this. Thanks @harish-24 and @sathnaga

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems I had older avocado in the system I tested.
Will add them.

commit c6b0c8180cd38b66d29729eb825bc0d4155a33c8
Author: Wainer dos Santos Moschetta <wainersm@redhat.com>
Date:   Fri Jan 10 15:25:48 2020 -0300

    plugins: jsonresult: add missing warn and interrupt fields

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@harish-24 please check now. Added warn and interrupt

avocado-setup.py Outdated
result_link += "/job.log\n"
with open(result_json, encoding = "utf-8") as fp:
result_state = json.load(fp)
for state in ['total', 'pass', 'cancel', 'errors', 'failures', 'skip']:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@narasimhan-v I see a warn in there,

{
    "cancel": 0,
    "debuglog": "/home/satheesh/data/gits/github/avocado/job-2020-03-30T12.39-8a51c2c/job.log",
    "errors": 0,
    "failures": 0,
    "interrupt": 0,
    "job_id": "8a51c2c2ab2e400fc5089b3e6fd26244e383427e",
    "pass": 1,
    "skip": 0,
    "tests": [
        {
            "end": 1585552154.4747539,
            "fail_reason": null,
            "id": "1-/bin/true",
            "logdir": "/home/satheesh/data/gits/github/avocado/job-2020-03-30T12.39-8a51c2c/test-results/1-_bin_true",
            "logfile": "/home/satheesh/data/gits/github/avocado/job-2020-03-30T12.39-8a51c2c/test-results/1-_bin_true/debug.log",
            "start": 1585552154.219356,
            "status": "PASS",
            "time": 0.2553977966308594,
            "whiteboard": ""
        }
    ],
    "time": 0.2553977966308594,
    "total": 1,
    "warn": 0

avocado-setup.py Outdated
with open(result_json, encoding = "utf-8") as fp:
result_state = json.load(fp)
for state in ['total', 'pass', 'cancel', 'errors', 'failures', 'skip']:
result_link += "| %s %s |" % (state, str(result_state[state]))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and better to add a try: except here inorder to handle a missing key in different avocado version if any...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure

When multiple tests are run, along with kernel messages output,
it becomes difficult to read the status of the tests. This commits
tries to solve that by displaying the status in the summary.

Signed-off-by: Narasimhan V <sim@linux.vnet.ibm.com>
@narasimhan-v narasimhan-v force-pushed the results_summary_status branch from 236a0a2 to 4711ef2 Compare April 28, 2020 07:04
@narasimhan-v narasimhan-v requested a review from sathnaga April 28, 2020 07:04
Previously we were having set length of spaces needed in the
result summary. Now, we compute the length of items displayed in
first column of summary, and use it to compute space needed.

Signed-off-by: Narasimhan V <sim@linux.vnet.ibm.com>
@narasimhan-v
Copy link
Collaborator Author

12:34:48 INFO    : Summary of test results can be found below:
TestSuite                                                                                TestRun    Summary
 
host_io_fc_fvt_fs_mark_fs_mark                                                           Run        Successfully executed
/root/tests/results/job-2020-04-28T12.34-9823f5a/job.log
| PASS 0 || CANCEL 0 || ERRORS 3 || FAILURES 0 || SKIP 0 || WARN 0 || INTERRUPT 0 |
 
host_io_fc_fvt_warntest                                                                  Run        Successfully executed
/root/tests/results/job-2020-04-28T12.34-3250862/job.log
| PASS 0 || CANCEL 0 || ERRORS 0 || FAILURES 0 || SKIP 0 || WARN 1 || INTERRUPT 0 |

@sathnaga sathnaga merged commit 8225de2 into lop-devops:master May 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants