KeyError

[stdkoehler]

Hello, I'm trying to read an MDF4 file and have 2 issues.

1. In mdfinfo4 at line 536
   self['cc_ref'][i] = CCBlock(fid, self['cc_ref'][i])['unit']['Comment']
   I receive a KeyError for 'name'. As the only entry of CCBlock with a Comment key was an entry called unit I replaced those. Several CCBlocks didn't have unit either which I countered by introducing

try:
     self['cc_ref'][i] = CCBlock(fid, self['cc_ref'][i])['unit']['Comment']
except Exception, e:
     print repr(e)

I'm not sure how much this impacts the further processing. mdfreader.mdfinfo() is working now though as it seems

2. Continuing mdfreader.mdf() I receive the following errors:

('Unexpected error:', (<type 'exceptions.ImportError'>, ImportError('No module named dataRead',), <traceback object at 0x7fbe92be2f38>))
dataRead crashed, back to python data reading
('Unexpected error:', (<type 'exceptions.ImportError'>, ImportError('No module named dataRead',), <traceback object at 0x7fbe8ac113b0>))
dataRead crashed, back to python data reading
Traceback (most recent call last):
File "/workspace/Eclipse/Python/ReadMDF4/src/ReadMDF4.py", line 12, in
yop2 = mdfreader.mdf("./file.MF4")
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdf.py", line 106, in init
self.read(fileName, channelList=channelList, convertAfterRead=convertAfterRead, filterChannelNames=filterChannelNames)
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdfreader.py", line 347, in read
self.read4(self.fileName, info, multiProc, channelList, convertAfterRead, filterChannelNames=False)
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdf4reader.py", line 1282, in read4
buf.read(channelList) # reads raw data from data block with DATA and DATABlock classes
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdf4reader.py", line 371, in read
self[recordID]['data'] = self.load(record, zip=None, nameList=channelList, sortedFlag=True)
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdf4reader.py", line 453, in load
temps['data'] = DATABlock(record, parent_block=data_block, channelList=nameList, sortedFlag=sortedFlag)
File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdf4reader.py", line 92, in DATABlock
return fromstring(parent_block['data'], dtype=record.numpyDataRecordFormat, shape=record.numberOfRecords, names=record.dataRecordName)
File "/Programme/anaconda/lib/python2.7/site-packages/numpy/core/records.py", line 709, in fromstring
_array = recarray(shape, descr, buf=datastring, offset=offset)
File "/Programme/anaconda/lib/python2.7/site-packages/numpy/core/records.py", line 423, in new
strides=strides, order=order)
TypeError: buffer is too small for requested array

[ratal]

Hi,
For the first point, I will have a look and improve robustness.
Regarding the second point, it seems you try to read a file that is asking too much memory. You can try to use convertAfterRead=False parameter or ask to load only a set of channels you are interested in instead of the full file data.

[stdkoehler]

Hey ratal, thanks for the input! I tried convertAfterRead=False which didn't help though. The error occurs before reaching line 1367 of mdf4reader where the boolean is checked:

        if convertAfterRead:
            self._convertAllChannel4()

is never reached.

I would like to try to load only a set of channels, which can be done via argument channelList = ['channel', 'list']. However, I don't know the data and am not sure what to put as "channel" or "list"

EDIT: I tried mdfreader.mdfinfo("./file.MF4").listChannels("./file.MF4") to get the channel names. However, I once again received a KeyError at mdfinfo4 line 990:
for cn in range(nChannel):

                if not toChangeIndex[cn]:
                    # offset all indexes of indexes to be moved
                    self['CNBlock'][dg][cg][cn + nChannel] = self['CNBlock'][dg][cg].pop(cn)
                    self['CCBlock'][dg][cg][cn + nChannel] = self['CCBlock'][dg][cg].pop(cn)

File "/Programme/anaconda/lib/python2.7/site-packages/mdfreader/mdfinfo4.py", line 990, in readCGBlock
self['CCBlock'][dg][cg][cn + nChannel] = self['CCBlock'][dg][cg].pop(cn)
KeyError: 1

[ratal]

Indeed documentation is maybe not so clear about channel listing. Below is the correct procedure:
from mdfreader.mdfinfo4 import info4
yop=info4()
yop.listChannels4('yourfile.mf4')
Looking a bit more deeply, there is possibility your file uses nested channel conversions. It was not so much tested so far (no file to experiment) and not really implemented.
It could then create a wrong representation of stored data and ask for too much data to be loaded ending with memory issues.
If you still have problem, best way would be to send an example of this file or at least a representation of its structure, like a info4 dict.

[stdkoehler]

I still receive the same error as mentioned above, even if I use the approach you just described. I don't know whether the file is nested but it's possible. Sadly, I can't send you the data file as it is confidential from a project I work with several partners. A info4 dict might be possible though, if you tell me how I create it.

[ratal]

After thinking, it might be complicated for you and me to use this info4 dict. Instead, could you use mdfvalidator from vector (free use) and export as an xml the file structure ? It will help me to understand it.

[ratal]

2 feedback since 2 weeks, closed

[swoldetsadick]

I have the same issue but with a relatively small file 2,46 Mb. I would think it is a nested data format problem.
self['cc_ref'][i] = CCBlock(fid, self['cc_ref'][i])['name']['Comment']
does not exist.
self['cc_ref'][i] = CCBlock(fid, self['cc_ref'][i])['unit']['Comment'] does but I do not know if I should make the change.