[POD5 merging] RuntimeError: Invalid: Unexpected empty message in IPC file format #178
Description
Issue Description
Fail to merge POD5 files from a skipped run. Here is the command used:
POD5_DEBUG=1 pod5 filter \
--missing-ok \
--force-overwrite \
~/runs/mysample/pod5_skip/*.pod5 \
--ids mysample.read_ids.tsv \
--output mysample.cleaned.pod5
Where mysample.read_ids.tsv has been generated as follows:
pod5 view ~/runs/mysample/pod5_skip \
--include "read_id, channel, end_reason" | \
grep -v analysis_config_change | \
cut -f1 > mysample.read_ids.tsv
Logs
Please provide any log files. These can be generated by setting the
POD5_DEBUGenvironment variable e.g.POD5_DEBUG=1 pod5 view my.pod5
POD5_DEBUG=1 pod5 filter --missing-ok --force-overwrite ~/runs/mysample/pod5_skip/*.pod5 --ids mysample.read_ids.tsv --output mysample.pod5
Parsed 77028387 reads_ids from: mysample.read_ids.tsv
Found 77044516 read_ids from 78 inputs
Calculated 74051605 transfers
Filtering: 0%| |
Filtering: 1%|##1 | 1100000/74051605
Filtering: 3%|#### | 2072912/74051605 /
Filtering: 4%|###### | 3086216/74051605
Filtering: 5%|#######9 | 4069537/74051605
Filtering: 7%|#########9 | 5067626/74051605
Filtering: 8%|###########8 | 6053389/74051605
Filtering: 9%|#############7 | 7033279/74051605
Filtering: 11%|###############8 | 8070070/74051605
Filtering: 12%|#################7 | 9054808/74051605
Filtering: 14%|###################4 | 10025517/74051605
Filtering: 15%|#####################4 | 11036815/74051605
Filtering: 16%|#######################4 | 12074986/74051605
Filtering: 18%|#########################5 | 13130121/74051605
Filtering: 19%|###########################4 | 14100341/74051605
Filtering: 20%|#############################4 | 15139795/74051605
Filtering: 22%|###############################3 | 16133404/74051605
Filtering: 23%|#################################3 | 17124927/74051605
Filtering: 24%|###################################1 | 18084232/74051605 [26:17<1:21:20, 11466.87Read/s]Traceback (most recent call last):
File "/storage0/miniforge3/envs/nanopore/bin/pod5", line 10, in <module>
sys.exit(main())
^^^^^^
File "/storage0/miniforge3/envs/nanopore/lib/python3.12/site-packages/pod5/tools/main.py", line 60, in main
return run_tool(parser)
^^^^^^^^^^^^^^^^
File "/storage0/miniforge3/envs/nanopore/lib/python3.12/site-packages/pod5/tools/parsers.py", line 41, in run_tool
raise exc
File "/storage0/miniforge3/envs/nanopore/lib/python3.12/site-packages/pod5/tools/parsers.py", line 38, in run_tool
return tool_func(**kwargs)
^^^^^^^^^^^^^^^^^^^
File "/storage0/miniforge3/envs/nanopore/lib/python3.12/site-packages/pod5/tools/parsers.py", line 290, in run
return filter_pod5(**kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/storage0/miniforge3/envs/nanopore/lib/python3.12/site-packages/pod5/tools/utils.py", line 63, in wrapper
raise exc
File "/storage0/miniforge3/envs/nanopore/lib/python3.12/site-packages/pod5/tools/utils.py", line 60, in wrapper
ret = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/storage0/miniforge3/envs/nanopore/lib/python3.12/site-packages/pod5/tools/pod5_filter.py", line 166, in filter_pod5
filter_reads(dest=dest, sources=sources, duplicate_ok=duplicate_ok)
File "/storage0/miniforge3/envs/nanopore/lib/python3.12/site-packages/pod5/tools/utils.py", line 63, in wrapper
raise exc
File "/storage0/miniforge3/envs/nanopore/lib/python3.12/site-packages/pod5/tools/utils.py", line 60, in wrapper
ret = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/storage0/miniforge3/envs/nanopore/lib/python3.12/site-packages/pod5/tools/pod5_filter.py", line 102, in filter_reads
while repacker.currently_open_file_reader_count >= active_limit:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/storage0/miniforge3/envs/nanopore/lib/python3.12/site-packages/pod5/repack.py", line 25, in currently_open_file_reader_count
return self._repacker.currently_open_file_reader_count
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Invalid: Unexpected empty message in IPC file format
Specifications
- Pod5 Version: 0.3.15
- Python Version: 3.12.11
- Platform: Ubuntu 22.04 x86_64
The pod5 tool has worked for all the other runs. I really appreciate any help you could provide. Thank you.