splunklib.searchcommands.splunk_csv.dict_reader raises KeyError when supports_multivalues=True

[brattonc]

This error seems to occur because dict_reader.py populates the fieldnames property with only the non __mv_* field names, which causes super(DictReader, self).next() to return a dict where the __mv_* fields are grouped together in a single dictionary entry.

Example:

1. Splunk passes a set of field names to the custom command:

   test1,test2,__mv_test1,__mv_test2
   

2. splunk_csv.DictReader.fieldnames gets called which results in:

   csv.DictReader._fieldnames = ["test1", "test2","__mv_test1","__mv_test2"]
   splunk_csv.DictReader.__fieldnames = ["test1", "test2"]
   splunk_csv.DictReader.__mv_fieldnames = [("test1","__mv_test1"), ("test2","__mv_test2")]
   return __fieldnames
   

   Note that __fieldnames is returned.

3. splunk_csv.DictReader.next is called, which in turn calls csv.DictReader.next and reads an event from Splunk:

   "1 2 3","9193","$1$,$2$,$3$",
   

4. csv.DictReader.next combines the event and fieldnames property...

   d = dict(zip(self.fieldnames, row))
   lf = len(self.fieldnames)
   lr = len(row)
   if lf < lr:
       d[self.restkey] = row[lf:]
   elif lf > lr:
       for key in self.fieldnames[lr:]:
           d[key] = self.restval
   return d
   

   Because this method uses fieldnames, d now contains:

   [("test1","1 2 3"),("test2","9193"),(self.restkey,["$1$,$2$,$3$",""])]
   

5. splunk_csv.DictReader.next iterates over self.__mv_fieldnames and attempts to look up "__mv_test1" in the dictionary, which results in a KeyError.

Running Splunk 6.2.4
Python SDK 1.3.1

[shakeelmohamed]

Thanks @brattonc, we're working on updating search commands. We'll be sure to address this when those changes are released

[ahoying]

For my relatively simple streaming command, switching fieldnames to return _fieldnames instead of __fieldnames fixed this bug without any noticeable side effects. I'm running Python SDK 1.4.0 on Splunk 6.2.4.

[shakeelmohamed]

We've just released v1.5.0 to address this, thanks again for the bug report @ahoying @brattonc