Was unable to replace data for a track

[EricR86]

From orion.bu...@gmail.com on December 15, 2009 16:59:28

What steps will reproduce the problem? 1. Took standard genomedata.round5a.tier1 collection (created before update
to genomedata-close-data, so there are supercontigs without continuous
children.
2. Used genomedata-erase-track to remove the two DNaseI.*.Crawford tracks.
3. Replaced the data in parallel using genomedata-load-data.
4. Used genomedata-close-data to recompute the metadata for the new track.

This should succeed, making the genomedata collection usable with the new
data. Instead, just near the end of genomedata-close-data, the following
exceptions was raised:

writing metadata for chr1
scanning supercontig_0
BDP1.K562
BRF1.K562
cFos.GM12878
cFos.K562
cJun.K562
cMyc.K562.Iyer
cMyc.K562
CTCF.GM12878.Iyer
CTCF.GM12878
CTCF.GM12878.Stam
CTCF.K562.Bernstein
CTCF.K562.Iyer
CTCF.K562.Stam
DNaseI.GM12878.Crawford
HDF5-DIAG: Error detected in HDF5 (1.8.2) thread 0:
#000: H5Dio.c line 175 in H5Dread(): can't read data
major: Dataset
minor: Read failed
#1: H5Dio.c line 407 in H5D_read(): can't read data
major: Dataset
minor: Read failed
#2: H5Dchunk.c line 1517 in H5D_chunk_read(): unable to read raw data chunk
major: Low-level I/O
minor: Read failed
#3: H5Dchunk.c line 2443 in H5D_chunk_lock(): data pipeline read failed
major: Data filters
minor: Filter operation failed
#4: H5Z.c line 1008 in H5Z_pipeline(): filter returned failure during read
major: Data filters
minor: Read failed
#5: H5Zdeflate.c line 117 in H5Z_filter_deflate(): inflate() failed
major: Data filters
minor: Unable to initialize object
Traceback (most recent call last):
File
"/net/noble/vol2/home/stasis/arch/Linux/RHEL4/x86_64/bin/genomedata-close-data",
line 8, in
load_entry_point('genomedata==0.1.7.dev- r2670 ', 'console_scripts',
'genomedata-close-data')()
File
"/net/noble/vol2/home/stasis/arch/Linux/RHEL4/x86_64/lib/python2.5/genomedata-0.1.7.dev_r2670-py2.5.egg/genomedata/_close_data.py",
line 164, in main
return close_data(genomedatadir, **kwargs)
File
"/net/noble/vol2/home/stasis/arch/Linux/RHEL4/x86_64/lib/python2.5/genomedata-0.1.7.dev_r2670-py2.5.egg/genomedata/_close_data.py",
line 140, in close_data
write_metadata(chromosome, verbose=verbose)
File
"/net/noble/vol2/home/stasis/arch/Linux/RHEL4/x86_64/lib/python2.5/genomedata-0.1.7.dev_r2670-py2.5.egg/genomedata/_close_data.py",
line 74, in write_metadata
col = continuous[:, col_index]
File
"/net/gs/vol3/software/modules-sw/python/2.5.4/Linux/RHEL4/x86_64/lib/python2.5/site-packages/tables/array.py",
line 504, in getitem
arr = self._readSlice(startl, stopl, stepl, shape)
File
"/net/gs/vol3/software/modules-sw/python/2.5.4/Linux/RHEL4/x86_64/lib/python2.5/site-packages/tables/array.py",
line 571, in _readSlice
self._g_readSlice(startl, stopl, stepl, arr)
File "hdf5Extension.pyx", line 1057, in
tables.hdf5Extension.Array._g_readSlice
tables.exceptions.HDF5ExtError: Problems reading the array data.

Original issue: http://code.google.com/p/genomedata/issues/detail?id=4

[EricR86]

From orion.bu...@gmail.com on December 15, 2009 21:28:14

In a second trial, the program failed at precisely the same point. It should be noted that DNaseI.GM12878.Crawford was one of two tracks that were erased and re-loaded. It looks like the data for one
of the arrays is being corrupted in the process of either erasing the data or re-loading the data.

[EricR86]

From orion.bu...@gmail.com on January 04, 2010 16:31:08

This is the same error that HDF5 returns when a matrix is indexed only on the first
dimension (e.g. continuous[start:stop] raises an error whereas continuous[start:stop,
:] does not)

This might be related...

[EricR86]

From orion.bu...@gmail.com on February 24, 2010 14:01:11

This is fixed. The cause seemed to be overwriting the data with a new array instead
of just a constant NAN, thus filling up memory with empty data in the writing process.

Status: Fixed