Android: initial version of the flakey.txt file #197
Conversation
android/flakey.txt
Outdated
| #Hikey | ||
| E com.android.cts.usb.TestUsbTest#testUsbSerialReadOnDeviceMatches ALL HiKey ALL | ||
|
|
||
| ## Flakey Cases |
There was a problem hiding this comment.
Please remove the trailing whitespace
android/flakey.txt
Outdated
| # TESTNAME | ||
| # | ||
| # KERNEL_VERSIONS | ||
| # : ( ALL | 5.4 | 4.19 | 4.14 | 4.9 | 4.4 ) |
There was a problem hiding this comment.
What is 'ALL' meaning in this context?
There was a problem hiding this comment.
Hi Anders - it means that the given classification 'E / F etc' is valid for ALL kernels under test; like a '*', should match all kernels.
android/flakey.txt
Outdated
| # : ( ALL | 5.4 | 4.19 | 4.14 | 4.9 | 4.4 ) | ||
| # | ||
| # HARDWARE | ||
| # : ( ALL | HiKey | HiKey960 | db845 ) |
There was a problem hiding this comment.
a few questions, 'ALL', what does that mean in this context?
the three hardware names, 'HiKey', 'HiKey960' and 'db845' why doesn't they match the environment name in SQUAD?
What do they match if any?
There was a problem hiding this comment.
any idea's about this @sumitsemwal @liuyq ?
There was a problem hiding this comment.
The hardware matches the hardware defined in androidreportconfig.py like here:
hardware
squad environment is not used by the current android report tool,.
There was a problem hiding this comment.
right this is use in this way that you pull stuff from the DB to harmonise the data.
And you do that from all the SQUAD projects starting with '4.14-stable-*' or?
Will andoid move to use one project for one kernel release and different environments (DUT's) inside that project or isn't that feasible due to the different Android versions?
There was a problem hiding this comment.
Could not understand how your question is related to this PR:(
About using environments in one project, current squad does not support it well for android report. different Android versions would be one case.
And about the future, there is no plan, I think as a user of squad, the one supported by squad already and easy to be used for the setup will be used, projects based, environment based, or mixture.
And for the HARDWARE items defined here, I think it's possible to be the corresponding environment value too when environment setup is used. Or when migrated to the format here that LTS uses, maybe environment value will be used(though not in this file that time)
android/flakey.txt
Outdated
| # : ( ALL | HiKey | HiKey960 | db845 ) | ||
| # | ||
| # ANDROID_VERSIONS | ||
| # : ( ALL | Android8 | Android9 | Android10 | Android11 | AOSP ) |
There was a problem hiding this comment.
What does 'ALL' mean in this context?
Why isn't Android(12|13|14|15) on the list?
There was a problem hiding this comment.
Similar to above - 'ALL' means a wild-card match to all values of the variable - here it is about ANDROID_VERSIONS.
Good catch about the 12/13/14 list - those should be added. And 8/9/10 should be removed I suppose.
| # : STATE TESTNAME (KERNEL_VERSIONS) (HARDWARE) (ANDROID_VERSIONS) | ||
| # | ||
| # STATE | ||
| # I | F | B | E |
There was a problem hiding this comment.
Maybe explain what the different STATE's means?
There was a problem hiding this comment.
yes, should be explained here.
IIRC, I was Investigating, F was flakey, E was expected failure. I don't recall what B was?
There was a problem hiding this comment.
ok thank you @sumitsemwal , @liuyq do you know what B stands for?
There was a problem hiding this comment.
No, I don't know that.
There was a problem hiding this comment.
maybe we should drop it from the list?
There was a problem hiding this comment.
OK cool, but we are sure about the others? maybe we should explain them in a comment so we don't forget them moving forward?
There was a problem hiding this comment.
updated with comments on the meaning
|
updated the hardware, android versions and the kernel versions list with the new version. |
which is used by the LKFT Android projects to
define the known flakey tests.
It's in a different format from the current LTS list
at moment, which will be changed to the same format
in the next step, that the work will be tracked here:
https://linaro.atlassian.net/browse/LCAA-360
Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>
|
Hi, @roxell |
Yes my concern is still that we change the device name of the DUT in two many places but hey thats for another day... but it is important to push back as much as possible so its not changed in squad-report or here it should be the device name in LAVA that we are using and that should be stored in squad and used here and in squad-report... I merged this for now. |
OK, I think I get your point now. I will try to see if it could be updated to use the lava device name. rb5 there looks a bit user friendly than "qrb5165-rb5", it's shorter and easier to read.
Thanks for merge it! |
which is used by the LKFT Android projects to
define the known flakey tests.
It's in a different format from the current LTS list at moment, which will be changed to the same format in the next step, that the work will be tracked here:
https://linaro.atlassian.net/browse/LCAA-360