Missing testcases in squad

[feine-moehre]

I miss test cases in SQUAD (0.64.1).

I have a lava Job with 3 Testsuites: 0_rtc-test, 1_simple-ntp, lava.

The job results are attachted. lava_1687.txt

The test case time-stamp-difference is failed.

Now I look up the results in SQUAD:

In this view, some testcases are missing especially the failed test case time-stamp-difference.

I tried to fetch manually the job result via XML RPC on ipython. So I can confirm, that the result of especially time-stamp-difference is available via XML RPC.

The test job is fetched correctly by SQUAD. There is no failure reported. Refetching changes nothing.

[mwasilew]

It's not missing (most likely). Please check your list of metrics in squad. Since time-stamp-difference has accompanying measurement in LAVA it's converted to metric.

[feine-moehre]

Ok. in the metrics section, after some clicks I can see only the measurement, no unit and especially no test case result. There is nothing which tells me that this test case is failed. So the main point of an test case, pass or fail, is not displayed.

[mwasilew]

We don't support units at the moment. As for pass fail, if you need to see it, you should remove measurement in LAVA. You can't have both. It's either Test (pass or fail) or a Metric (benchmark value).

[feine-moehre]

So I have to call lava-test-case as workaround twice?

lava-test-case time-stamp-difference --result $RESULT
lava-test-case time-stamp-difference-metric --result $RESULT --measurement $DIFF --units "sec"

This workaround works but it is very unpleasant. Because the -metric call needs also the --result parameter. Otherwise LAVA will no handle the testcase.

[mwasilew]

I know. It's LAVA limitation. You can try to fix it from both ends:

• allow lava-test-case call to accept either measurement or result
• extract both result and measurement on SQUAD end (not that hard to do)

The later one should be configurable though as I don't think this is a good default.

Answering your question - this will work.

[feine-moehre]

• In my view changing this behaviour in LAVA would break things. Because LAVA can cope with that.
• Handling that in SQUAD would be better. But a configurable behaviour is no really nice because some people use one way, some another. So test cases are less portable.

If SQUAD extracts result and measurement as a new (default) behaviour what would break? When you use two calls of lava-test-case, one for result and a corresponding one for measurement. with this behaviour you only one result more and the measurement is still present.

[mwasilew]

I have a slightly different view. IMHO LAVA is not doing the right thing. It should only execute the test cases and it can't know whether a measurement of X constitutes 'pass' or 'fail'. At least not in all cases. You can only tell that if you have a hard coded threshold in the test cases. In many cases you only look for trend in the measurement in which case single point can never be 'pass' or 'fail'. For this reason SQUAD either uses test result (when no measurement is present) or discards it when there is a measurement. As I wrote, this isn't terribly hard to change, but I would like it to be configurable for each project (this is the smallest granularity we can currently get). This is where the change should happen:
https://github.com/Linaro/squad/blob/master/squad/ci/backend/lava.py#L306
This is example on how backend gets it's settings:
https://github.com/Linaro/squad/blob/master/squad/ci/backend/lava.py#L344

[feine-moehre]

This issue has huge potential to break something. And I don't want to modify code which might help only me.

As a first workaround I use a convenience function to submit result and measurement in two calls of lava-test-case.

[mwasilew]

No worries, I'll try to get this done some time next week.