Mechanism to add parameters to results.json file

[marioaag]

I'm interested to add extra details to the results.json file in the tests section, is it a mechanism to include extra values?
Example:
''''
"tests": [
{
"end": 4.214863779,
"fail_reason": "",
"id": "160621-162417945045.BlueGenParallel.-1-/Users/marioalvarado/git/iLabConfig/test/avocado/testcases/CheckHostStatus/CheckHostStatus.py:CheckHostStatus.test_os_info",
"logdir": "/tmp/avocado_logs/job-2021-06-16T11.24-51227b3/test-results/160621-162417945045.BlueGenParallel.-1-_Users_marioalvarado_git_iLabConfig_test_avocado_testcases_CheckHostStatus_CheckHostStatus.py_CheckHostStatus.test_os_info",
"logfile": "",
"start": 0.757385825,
"status": "PASS",
"tags": {},
"time": 3.457477954,
"whiteboard": ""
},
'''

I would like to add extra values here.

And also another quick question, what is the meaning of the values in the start and end values? what is the format used?

[beraldoleal]

@marioaag about the extra items to add here, we are open to add new things if needed. So you can open an issue and we discuss/implement there. Depending on the item I can help adding support.

[marioaag]

Thanks! I'm interested on see the config parameters by test and this looks like a good place to get them added, that could be posible?

[beraldoleal]

Have you tried ~/avocado/job-results/latest/jobdata/args.json ? There is run.test_parameters there.

[marioaag]

yep but that info is at job level, it is not at test level, when an specific information is defined at the runnable level, like this:
'''
runnable = Runnable('avocado-instrumented',
test_ref,
config={'run.test_parameters': variant_parameters})
'''
That info is not display or logged but it is available Test object level as self._config
I believe that has the config information for each one of the test available could be always helpful, in my specific scenario it will give me access in more elegant way that what I'm doing right now to the run.test_parameters info.

Note:
I'm using nrunner

[richtja]

@marioaag about your second question start and end values are start_time and end_time of test. They are gathered by time.monotonic()

[clebergnu]

Hi @marioaag,

IIUC, you want to show "some data" in the results.json, right? This "some data", in a generic format, is what users are expected to put in the whiteboard.

There's an example in examples/tests/whiteboard.py. If you run it, you'd get:

{
    "cancel": 0,
    "debuglog": "/home/cleber/avocado/job-results/job-2021-06-21T17.00-8bbb97b/job.log",
    "errors": 0,
    "failures": 0,
    "interrupt": 0,
    "job_id": "8bbb97b03ab2f7efed82e8c4150150cc9efd8e5c",
    "pass": 1,
    "skip": 0,
    "tests": [
        {
            "end": 1829550.444271247,
            "fail_reason": null,
            "id": "1-examples/tests/whiteboard.py:WhiteBoard.test",
            "logdir": "/home/cleber/avocado/job-results/job-2021-06-21T17.00-8bbb97b/test-results/1-examples_tests_whiteboard.py_WhiteBoard.test",
            "logfile": "/home/cleber/avocado/job-results/job-2021-06-21T17.00-8bbb97b/test-results/1-examples_tests_whiteboard.py_WhiteBoard.test/debug.log",
            "start": 1829550.442510152,
            "status": "PASS",
            "tags": {},
            "time": 0.0017610948998481035,
            "whiteboard": "TXkgbWVzc2FnZSBlbmNvZGVkIGluIGJhc2U2NA==\n"
        }
    ],
    "time": 0.0017610948998481035,
    "total": 1,
    "warn": 0
}

You should be able to easily get that data, and de-serialize it, no matter what it contains (which can be extra JSON content.

Hope this helps!

[clebergnu]

And btw @marioaag , there's documentation here about the whiteboard: https://avocado-framework.readthedocs.io/en/89.0/guides/writer/chapters/writing.html#saving-test-generated-custom-data

[marioaag]

That feature covers what I'm trying to do, thanks!
But giving it a try with nrunner looks like I hit a bug...
The raw copy of the whiteboard has the output as expected but the results.json doesn't

{
    "cancel": 0,
    "debuglog": "/tmp/avocado_logs/job-2021-06-23T12.07-d97272c/job.log",
    "errors": 0,
    "failures": 0,
    "interrupt": 0,
    "job_id": "d97272c1d442f54ab040337effadacb1612c9a5e",
    "pass": 1,
    "skip": 0,
    "tests": [
        {
            "end": 0.828379635,
            "fail_reason": "<unknown>",
            "id": "230621-170746400131.BlueGenParallel.-1-/Users/marioalvarado/git/iLabConfig/test/avocado/testcases/CheckHostStatus/CheckHostStatus.py:CheckHostStatus.test_os_info",
            "logdir": "/tmp/avocado_logs/job-2021-06-23T12.07-d97272c/test-results/230621-170746400131.BlueGenParallel.-1-_Users_marioalvarado_git_iLabConfig_test_avocado_testcases_CheckHostStatus_CheckHostStatus.py_CheckHostStatus.test_os_info",
            "logfile": "<unknown>",
            "start": 0.482796272,
            "status": "PASS",
            "tags": {},
            "time": 0.345583363,
            "whiteboard": "<unknown>"
        }
    ],
    "time": 0.345583363,
    "total": 1,
    "warn": 0
}

[clebergnu]

@marioaag yes, looks like a bug! Thanks for spotting it.

BTW, we're in a "nrunner bug hunting/squashing" mode, in hope of making it the default runner as soon as in version 90.0. Would you, please, open an issue about this and tag it with nrunner and bug? Many thanks!

[marioaag]

Opened the ticket:
#4717

I don't have permissions to assign labels

[marioaag]

Even though the whiteboard solution covers in some way what I'm trying to do, don't you thing that with the possibility of have different configurations for each one of the test, get the test config values and parameters values by default in the results.json file will be useful? In my specific scenario will help to don't need to add the whiteboard print in each one of the tests.

[clebergnu]

@marioaag thanks for the issue. I've added the labels (and I'll check if GitHub allows us to open that for all contributors).

About your question, I believe some users will consider the results.json really about actual results, while some will understand that input information (such as config and parameters) are also relevant. IMO, it could be an optional, opt-in, kind of feature to add config and parameters to the results.json file.

BTW, the original idea for the input information is to be able to preserve the "recipes" for each task, so that they can even be replayed with a single command if necessary. If you look at what avocado list --resolver --write-recipes-to-directory=/your/dir -- your-tests does, you will get the general idea. Those files can be executed with avocado-runner-$type runnable-run-recipe.

Now about the recipe files, I'm 100% onboard with improving it so that it contains, by default, the configuration and parameters used. These can be preserved in the individual test results directories.

[clebergnu]

@willianrampazzo tagging you so that you keep this in mind while working on issue #4569.

[marioaag]

I like the idea having those files available in the individual test results directory and generated automatically that will work for what I'm doing and for a lot of other things!

[clebergnu]

I like the idea having those files available in the individual test results directory and generated automatically that will work for what I'm doing and for a lot of other things!

Care to open another issue? 😉

[marioaag]

Done, issue #4745.
Again don't have permissions to assign labels :(

[clebergnu]

Thanks @marioaag , I've added the labels.