ValueError: Data size is not a multiple of the chunk size #40

Nodd · 2015-11-04T09:47:36Z

I'm trying to open a 4.4GB tdms file, and I get the following error from tdmsinfo:

Traceback (most recent call last):
  File "/usr/bin/tdmsinfo", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python3.5/site-packages/nptdms/tdmsinfo.py", line 26, in main
    tdmsfile = tdms.TdmsFile(args.tdms_file)
  File "/usr/lib/python3.5/site-packages/nptdms/tdms.py", line 153, in __init__
    self._read_segments(tdms_file)
  File "/usr/lib/python3.5/site-packages/nptdms/tdms.py", line 166, in _read_segments
    tdms_file, self.objects, previous_segment)
  File "/usr/lib/python3.5/site-packages/nptdms/tdms.py", line 456, in read_metadata
    self.calculate_chunks()
  File "/usr/lib/python3.5/site-packages/nptdms/tdms.py", line 485, in calculate_chunks
    "chunk size %d" % (total_data_size, data_size))
ValueError: Data size 4435200000 is not a multiple of the chunk size 4000256

I saw that there are closed issues with the same error, but I just installed v0.7.1 so I use the last version.
The error appear on multiple files. Those files can be read with a LabView program so they are valid. Also LabView didn't crash while writing the files.

I'm on Arch Linux 64 bits with python 3.5 with 32 GB of RAM.

The text was updated successfully, but these errors were encountered:

Nodd · 2015-11-04T09:49:34Z

with -d and -p options, there is more information:

DEBUG:nptdms.tdms:Reading segment at 0
DEBUG:nptdms.tdms:Property kTocDAQmxRawData is False
DEBUG:nptdms.tdms:Property kTocRawData is True
DEBUG:nptdms.tdms:Property kTocNewObjList is True
DEBUG:nptdms.tdms:Property kTocInterleavedData is False
DEBUG:nptdms.tdms:Property kTocBigEndian is False
DEBUG:nptdms.tdms:Property kTocMetaData is True
DEBUG:nptdms.tdms:Reading metadata at 28
DEBUG:nptdms.tdms:Creating a new segment object
DEBUG:nptdms.tdms:Reading metadata for object /
DEBUG:nptdms.tdms:Object has no data in this segment
DEBUG:nptdms.tdms:Reading 1 properties
DEBUG:nptdms.tdms:Property name (tdsTypeString): testcase7_manual_smb100_trigger_500MCarrier_CH0
DEBUG:nptdms.tdms:Creating a new segment object
DEBUG:nptdms.tdms:Reading metadata for object /'Group Name'
DEBUG:nptdms.tdms:Object has no data in this segment
DEBUG:nptdms.tdms:Reading 0 properties
DEBUG:nptdms.tdms:Creating a new segment object
DEBUG:nptdms.tdms:Reading metadata for object /'Group Name'/'Channel Name'
DEBUG:nptdms.tdms:Object data type: tdsTypeI16
DEBUG:nptdms.tdms:Object number of values in segment: 2000128
DEBUG:nptdms.tdms:Reading 0 properties
INFO:nptdms.tdms:Read metadata: Took 0.9210000000000051 ms
Traceback (most recent call last):
  File "/usr/bin/tdmsinfo", line 11, in <module>
    sys.exit(main())
  File "/usr/lib/python3.5/site-packages/nptdms/tdmsinfo.py", line 26, in main
    tdmsfile = tdms.TdmsFile(args.tdms_file)
  File "/usr/lib/python3.5/site-packages/nptdms/tdms.py", line 153, in __init__
    self._read_segments(tdms_file)
  File "/usr/lib/python3.5/site-packages/nptdms/tdms.py", line 166, in _read_segments
    tdms_file, self.objects, previous_segment)
  File "/usr/lib/python3.5/site-packages/nptdms/tdms.py", line 456, in read_metadata
    self.calculate_chunks()
  File "/usr/lib/python3.5/site-packages/nptdms/tdms.py", line 485, in calculate_chunks
    "chunk size %d" % (total_data_size, data_size))
ValueError: Data size 4435200000 is not a multiple of the chunk size 4000256

Nodd · 2015-11-04T10:38:04Z

I was able to read the file simply by not raising the error:

        if total_data_size % data_size != 0:
            #raise ValueError(
            print(
                "Data size %d is not a multiple of the "
                "chunk size %d" % (total_data_size, data_size))
        #else:
        self.num_chunks = total_data_size // data_size

adamreeve · 2015-11-07T00:59:55Z

Hmm that's odd. Can you tell if you're missing any data at the beginning or end of the file compared to what LabView reads? The structure of the file looks pretty straightforward, so I'm not sure what's going wrong here. Does this only happen with very large files?

Nodd · 2015-11-07T01:32:42Z

I only have large files :(
I'm missing some data at the end of the file.

adamreeve · 2015-11-10T08:28:50Z

Any chance you can upload it somewhere so I can have a look? If you could generate a smaller file that showed the same issue that would be awesome.

Delicate-aRt · 2016-04-05T14:58:22Z

Hi guys,

Have same issue here:

b = TdmsFile('small_tdms_file.tdms')
Data size 1075200 is not a multiple of the chunk size 2293760

File is pretty small, about 1Mb. You may download it here.

adamreeve · 2016-04-06T10:24:19Z

Thanks, I can reproduce the error with that file so hopefully will be able to figure out what's going on. The metadata says this file should have 448 channels (/Measurement/0 to /Measurement/447), each with 1280 single-precision (32 bit) floating point values, but the file size is less than what that would total (the 2293760 byte chunk size) so that can't be right. Do you know how many values should be in each channel?

Delicate-aRt · 2016-04-06T11:31:36Z

I was provided only with file and kind of custom python parser that tries to read it.

After reviewing the code and intermediate result - you are right, there should be 448x1280 values, but in fact there is only 448x600 values.

I have no idea whether there should be more or less. And believe me, I completely understand how weird this sounds to you :)

Should I consider this as corrupted tdms file? Or something that can be handled somehow?

Nodd · 2016-04-06T16:34:10Z

It looks like the last chunk is not complete it the actual data is smaller that the chunk size. There is no padding to match the chunk size.
I don't know if it's legal, but it looks like it can happen easily.

adamreeve · 2016-04-09T03:01:57Z

Yeah it sounds like something that npTDMS should handle, although I couldn't find any mention of this being valid in NI's documentation and how exactly to determine which channels have data and how many values are in each channel if the size is less than expected. Hopefully we can just keep reading as normal up until hitting the actual end of the segment.

adamreeve · 2016-04-10T10:10:51Z

I haven't had a lot of time to look into this but have made a bit of progress and the changes in this branch should work for interleaved data, non-interleaved will be a bit more work: https://github.com/adamreeve/npTDMS/tree/not-multiple-chunk-size.

@Delicate-aRt, your file is interleaved but @Nodd's isn't.

Delicate-aRt · 2016-04-12T20:45:29Z

@adamreeve you are right: it is in interleaved mode. Sorry for not providing that details, TDMS is something completely new for me.

I'll check whether it works now and let you know!

adamreeve · 2016-04-17T10:23:05Z

@Nodd, are you able to test out the latest code on that not-multiple-chunk-size branch and let me know if that works and the data looks correct? I think it should work fine for you as your file only has one channel, but I'm not sure if it would be correct if the same issue happened in a file with multiple channels.

fgroes · 2016-10-25T08:29:33Z

hi, I have the same issue, I also fixed it temporarily by commenting out the exception and handling the case, that there is only one chunk, which is not long enough

I now tested your fix and it works for my data, please merge the changes into the next release

unfortunately I can't provide you with the date, because it belongs to a customer of ours

adamreeve · 2016-10-25T19:45:38Z

Thanks @fgroes, I've just released 0.8.2 with this fix in it.

nmgeek · 2019-02-08T00:13:40Z

I saw this same problem, today, with version 0.13.0. Could the bug have regressed?
Our LabView application can read the file without error. The data is interleaved.

nmgeek · 2019-02-08T00:21:28Z

It's not a regression. The problem was not fixed in 0.8.2 either ... for my TDMS file.
Here is the output from v0.8.2:

WARNING:nptdms.tdms:Data size 348793088 is not a multiple of the chunk size 2000000. Will attempt to read last chunk
Traceback (most recent call last):
  File "nptdms_test_env/bin/tdmsinfo", line 10, in <module>
    sys.exit(main())
  File "nptdms_test_env/local/lib/python2.7/site-packages/nptdms/tdmsinfo.py", line 26, in main
    tdmsfile = tdms.TdmsFile(args.tdms_file)
  File "nptdms_test_env/local/lib/python2.7/site-packages/nptdms/tdms.py", line 169, in __init__
    self._read_segments(tdms_file)
  File "nptdms_test_env/local/lib/python2.7/site-packages/nptdms/tdms.py", line 199, in _read_segments
    segment.read_raw_data(tdms_file)
  File "nptdms_test_env/local/lib/python2.7/site-packages/nptdms/tdms.py", line 521, in read_raw_data
    self._read_interleaved_numpy(f, data_objects, endianness)
  File "nptdms_test_env/local/lib/python2.7/site-packages/nptdms/tdms.py", line 552, in _read_interleaved_numpy
    combined_data = combined_data.reshape(-1, all_channel_bytes)
ValueError: cannot reshape array of size 793088 into shape (40)

nmgeek · 2019-02-08T00:25:26Z

The traceback produced by v0.13.0 is a bit different:

WARNING:nptdms.tdms:Data size 348793088 is not a multiple of the chunk size 2000000. Will attempt to read last chunk
Traceback (most recent call last):
  File "/usr/local/bin/tdmsinfo", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/dist-packages/nptdms/tdmsinfo.py", line 26, in main
    tdmsinfo(args.tdms_file, args.properties)
  File "/usr/local/lib/python2.7/dist-packages/nptdms/tdmsinfo.py", line 30, in tdmsinfo
    tdmsfile = tdms.TdmsFile(file)
  File "/usr/local/lib/python2.7/dist-packages/nptdms/tdms.py", line 94, in __init__
    self._read_segments(f)
  File "/usr/local/lib/python2.7/dist-packages/nptdms/tdms.py", line 124, in _read_segments
    segment.read_raw_data(f)
  File "/usr/local/lib/python2.7/dist-packages/nptdms/tdms.py", line 512, in read_raw_data
    self._read_interleaved_numpy(f, data_objects)
  File "/usr/local/lib/python2.7/dist-packages/nptdms/tdms.py", line 546, in _read_interleaved_numpy
    combined_data = combined_data.reshape(-1, all_channel_bytes)
ValueError: total size of new array must be unchanged

adamreeve mentioned this issue Oct 25, 2016

Handle truncated final chunk of data #49

Merged

adamreeve closed this as completed in #49 Oct 25, 2016

nmgeek mentioned this issue Feb 21, 2019

Data size is not a multiple of the chunk size #109

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ValueError: Data size is not a multiple of the chunk size #40

ValueError: Data size is not a multiple of the chunk size #40

Nodd commented Nov 4, 2015

Nodd commented Nov 4, 2015

Nodd commented Nov 4, 2015

adamreeve commented Nov 7, 2015

Nodd commented Nov 7, 2015

adamreeve commented Nov 10, 2015

Delicate-aRt commented Apr 5, 2016

adamreeve commented Apr 6, 2016

Delicate-aRt commented Apr 6, 2016

Nodd commented Apr 6, 2016

adamreeve commented Apr 9, 2016

adamreeve commented Apr 10, 2016

Delicate-aRt commented Apr 12, 2016

adamreeve commented Apr 17, 2016

fgroes commented Oct 25, 2016

adamreeve commented Oct 25, 2016

nmgeek commented Feb 8, 2019

nmgeek commented Feb 8, 2019

nmgeek commented Feb 8, 2019

ValueError: Data size is not a multiple of the chunk size #40

ValueError: Data size is not a multiple of the chunk size #40

Comments

Nodd commented Nov 4, 2015

Nodd commented Nov 4, 2015

Nodd commented Nov 4, 2015

adamreeve commented Nov 7, 2015

Nodd commented Nov 7, 2015

adamreeve commented Nov 10, 2015

Delicate-aRt commented Apr 5, 2016

adamreeve commented Apr 6, 2016

Delicate-aRt commented Apr 6, 2016

Nodd commented Apr 6, 2016

adamreeve commented Apr 9, 2016

adamreeve commented Apr 10, 2016

Delicate-aRt commented Apr 12, 2016

adamreeve commented Apr 17, 2016

fgroes commented Oct 25, 2016

adamreeve commented Oct 25, 2016

nmgeek commented Feb 8, 2019

nmgeek commented Feb 8, 2019

nmgeek commented Feb 8, 2019