CFA: invalid index exception

[pohly]

Sometimes, under uncertain circumstances, parsing manifest files fails because a line does not contain two words, leading to this exception (https://ostroproject.org/jenkins/job/build_intel-quark/1997/console):

ERROR: Exception in plugin ISA_LicenseChecker process_report():
03:46:29.670 Traceback (most recent call last):
03:46:29.670   File "/var/lib/jenkins/ostro-worker-19-slot-0-OTusb/ostro-os/meta-security-isafw/lib/isafw/isafw.py", line 133, in call_plugins
03:46:29.670     method(*parameters, **keywords)
03:46:29.670   File "/var/lib/jenkins/ostro-worker-19-slot-0-OTusb/ostro-os/meta-security-isafw/lib/isafw/isaplugins/ISA_la_plugin.py", line 262, in process_report
03:46:29.670     return LicenseChecker.process_report()
03:46:29.670   File "/var/lib/jenkins/ostro-worker-19-slot-0-OTusb/ostro-os/meta-security-isafw/lib/isafw/isaplugins/ISA_la_plugin.py", line 127, in process_report
03:46:29.670     self.process_pkg_list()
03:46:29.670   File "/var/lib/jenkins/ostro-worker-19-slot-0-OTusb/ostro-os/meta-security-isafw/lib/isafw/isaplugins/ISA_la_plugin.py", line 145, in process_pkg_list
03:46:29.670     orig_pkg_name = line.split()[2]
03:46:29.670 IndexError: list index out of range

That was with meta-security-isafw 89d4453.

At the very least we should add better error handling for the input files. Then if those errors trigger, we may get more information about the root cause of the problem.

[ereshetova]

I made this fix now: 18c9b4c
It would gracefully handle the condition and also allow to produce the reports to see what is wrong. I wasn't able to reproduce locally, so real cause is still unknown, but this commit should help to show it.

[ereshetova]

@pohly I cannot reproduce this issue anymore, can you?

[pohly]

We still need to check for "UNKNOWN" in the log files and figure out the root cause. I'd prefer to keep this open until we know that.

[ereshetova]

Problem is that I cannot find any UKNOWN in the logs...

[pohly]

Is that perhaps because none of the packages where the problem occurs need to be reported? In that case we will have to add more debug output and check the isafw_lalog for that output.

[ereshetova]

No, the file where now UNKNOWN should appear is the pkg list of the image: where first field is binary pkg name, second, pkg version and third one is source pkg. It has nothing to do with the license check yet at that point. I also made a small correction check for empty line there, I wonder if this was the case that was failing.

[pohly]

If it was the empty line, then why did we sometimes get that empty line and sometimes not? This still worries me. Which file is the code parsing?

[ereshetova]

@pohly, the file that is parsed is always saved. Here is example: http://ostro.ostc.intel.com/download/builds/meta-security-isafw_pull-requests/66/2016-07-06_02-10-08-build-85/isafw/intel-corei7-64/pkglist

[pohly]

@ereshetova is that "isafw/intel-corei7-64/pkglist" the input file (the one which triggered the index exception), or the output file (where we should find the "UNKNOWN" valued in the third column)? Probably the latter. My question was about the former - what file is getting parsed by the ISAFW code?

[ereshetova]

@pohly, actually I got you confused and was confused myself :( Task switching doesn't work very well recently for me. So, the isafw/intel-corei7-64/pkglist is the input file that triggers the exception. The output file doesn't exist, because the output data is only maintained in memory struct. I will add dump of that data now to the log to see if I see UNKNOWN present.

[pohly]

@ereshetova I know exactly what you mean with multitasking ;-} Okay, so it is as I thought ("none of the packages where the problem occurs need to be reported") and we need additional debug output to root-cause the problem.

[ereshetova]

@pohly. Actually after sleeping and thinking on clear head to this: I have the data already dumped in logs. Here for example: http://ostro.ostc.intel.com/download/builds/meta-security-isafw_pull-requests/66/2016-07-06_02-10-08-build-85/isafw/intel-quark/isafw-logs/isafw_lalog

If you search for UNKNOWN, you will find entries like:
kernel-modu+git0+80d463be82-r0.0 from bundle-ostro-image-swupd-qa-bundle-b UNKNOWN

where the first item, binary pkg name, then image name and then source pkg name that is missing for some reason and therefore fix was writing UNKNOWN.

Now why this happens. If you check pkg list here (http://ostro.ostc.intel.com/download/builds/meta-security-isafw_pull-requests/66/2016-07-06_02-10-08-build-85/isafw/intel-quark/pkglist ), you can find that this is the entry for this binary pkg:

kernel-modu+git0+80d463be82-r0.0 linux-firmware-ralink-license

So, it does have source pkg name, what it is missing is the version in between (means supplied as empty from bitbake), so there are only two arguments. I didn't expect this to be possible before, but I will make a better line parsing now given that such things happen.

[ereshetova]

there are also entries where source pkg name is also missing:
I am trying to investigate why...
kernel-module-nls-ascii 4.4.13+git1+44a6105173_2ae6amodule-adf4350