Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error at the 3-unzip/reads step of pb-asssembly #370

Closed
lizhao007 opened this issue Jan 23, 2021 · 8 comments
Closed

Error at the 3-unzip/reads step of pb-asssembly #370

lizhao007 opened this issue Jan 23, 2021 · 8 comments

Comments

@lizhao007
Copy link

lizhao007 commented Jan 23, 2021 •
edited by pb-cdunn
Loading

Operating system
Linux mn02.hpc.hzau 3.10.0-862.el7.x86_64 #1 SMP Fri Apr 20 16:44:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Package name
I installed the last version of pb-assembly with conda and I have used the command: conda update --all.

Conda environment
$ conda denovo_asm

Describe the bug
I used the fc_unzip.cfg as this website suggested:

[General]
max_n_open_files = 1000

[Unzip]
input_fofn=input.fofn
input_bam_fofn=input_bam.fofn
polish_include_zmw_all_subreads = true

[job.defaults]
job_type=lsf
pwatcher_type=blocking
JOB_QUEUE=high
MB=8192
NPROC=4
njobs=8
submit =bsub -K -n ${NPROC} -q ${JOB_QUEUE} -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} ${JOB_SCRIPT} 

[job.step.unzip.track_reads]
njobs=1
NPROC=48
MB=393216
# uses minimap2 now
[job.step.unzip.blasr_aln]
njobs=50
NPROC=2
MB=32000
[job.step.unzip.phasing]
njobs=100
NPROC=2
MB=16384
[job.step.unzip.hasm]
njobs=1
NPROC=48
MB=393216
# uses arrow now
[job.step.unzip.quiver]
njobs=50
NPROC=12
MB=98304

The command is:

#BSUB -J pbassembly
#BSUB -n 1
#BSUB -R span[hosts=1]
#BSUB -o %J.out
#BSUB -e %J.err
#BSUB -q normal
source activate denovo_asm
fc_unzip.py fc_unzip.cfg

Error message
After the task is submitted, the following error message is generated.
3-unzip/reads/run-P60beead857406d.bash.stderr(this is the error after i changed 48 cores to 30 cores, its the same error)

before Read_Overlap record j = 11537261 out of 490442646 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00015/uow-00/raw_reads.112.las 
[161671]preparing tr_stage1
[161671]maxrss:    17872
[161671]terminating track_reads workers...
Traceback (most recent call last):
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/multiprocessing/pool.py", line 851, in next
    item = self._items.popleft()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/threading.py", line 302, in wait
    waiter.acquire()
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/multiprocessing/pool.py", line 856, in next
    self._cond.wait(timeout)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/threading.py", line 302, in wait
    waiter.acquire()
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 211, in <module>
    main()
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 207, in main
    track_reads(**vars(args))
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 156, in track_reads
    try_run_track_reads(n_core, min_len, bestn, db_fn, las_fofn_fn, output_fn)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 134, in try_run_track_reads
    run_track_reads(exe_pool, file_list, min_len, bestn, db_fn, output_fn)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 76, in run_track_reads
    for fn, res in exe_pool.imap(io.run_func, inputs):
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/multiprocessing/pool.py", line 863, in next
    raise TimeoutError from None
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/threading.py", line 250, in __exit__
    return self._lock.__exit__(*args)
RuntimeError: release unlocked lock
2021-01-23 11:21:21,214 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2021-01-23 11:21:21,214 - root - INFO - CD: '/public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads' -> '/public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads'
2021-01-23 11:21:21,214 - root - INFO - CD: '/public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads' -> '/public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads'
2021-01-23 11:21:21,215 - root - CRITICAL - Error in /public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py with args="{'json_fn': '/public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py", line 280, in <module>
    main()
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py", line 272, in main
    run(**vars(parsed_args))
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py", line 266, in run
    run_cfg_in_tmpdir(cfg, tmpdir, '.')
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py", line 241, in run_cfg_in_tmpdir
    run_bash(bash_template, myinputs, myoutputs, parameters)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py", line 200, in run_bash
    util.system(cmd)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/io.py", line 27, in syscall
    raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
+++ pwd
++ echo 'FAILURE. Running top in /public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ pstree -apl

real    939m39.878s
user    7m40.529s
sys     16m53.484s
+ finish
+ echo 'finish code: 1'

3-unzip/reads/run-P60beead857406d.bash.stdout

Sender: LSF System <lsfadmin@c01n01>
Subject: Job 23426821: <P60beead857406d> in cluster <hzauhpc> Exited

Job <P60beead857406d> was submitted from host <c02n07> by user <zhaoli> in cluster <hzauhpc> at Fri Jan 22 10:33:50 2021
Job was executed on host(s) <30*c01n01>, in queue <high>, as user <zhaoli> in cluster <hzauhpc> at Fri Jan 22 19:41:41 2021
</public/home/zhaoli> was used as the home directory.
</public/home/zhaoli/denovo-perennial/pbassembly> was used as the working directory.
Started at Fri Jan 22 19:41:41 2021
Terminated at Sat Jan 23 11:21:22 2021
Results reported at Sat Jan 23 11:21:22 2021

Your job looked like:

------------------------------------------------------------
# LSBATCH: User input
/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pwatcher/mains/job_start.sh
------------------------------------------------------------

TERM_REQUEUE_ADMIN: job killed and requeued by root or LSF administrator.

Resource usage summary:

    CPU time :                                   8074.00 sec.
    Max Memory :                                 346550 MB
    Average Memory :                             337231.53 MB
    Total Requested Memory :                     40960.00 MB
    Delta Memory :                               -305590.00 MB
    Max Swap :                                   305 MB
    Max Processes :                              39
    Max Threads :                                43
    Run time :                                   516 sec.
    Turnaround time :                            89252 sec.

After an error, the system automatically submitted to 48 SMP cores. After running for a week, it stopped. No useful files were generated

falcon-unzip 1.3.7 (pip thinks "falcon-unzip 1.3.7")
falcon-kit 1.8.1 (pip thinks "falcon-kit 1.8.1")
pypeflow 2.3.0
[INFO]Setup logging from file "None".
[INFO]Default logging setup:

[loggers]
keys=root

[handlers]
keys=stream,file_all

[formatters]
keys=form01,form02

[logger_root]
level=NOTSET
handlers=stream,file_all

[handler_stream]
class=StreamHandler
level=INFO
formatter=form02
args=(sys.stderr,)

[handler_file_all]
class=FileHandler
level=DEBUG
formatter=form01
args=('all.log', 'w')

[formatter_form01]
format=%(asctime)s - %(name)s:%(lineno)d - %(levelname)s - %(message)s

[formatter_form02]
format=[%(levelname)s]%(message)s

[INFO]Using config=
{'General': {'max_n_open_files': '1000'},
 'Unzip': {'input_bam_fofn': 'input_bam.fofn',
           'input_fofn': 'input.fofn',
           'polish_include_zmw_all_subreads': True,
           'polish_use_blasr': False,
           'polish_vc_ignore_error': False},
 'job.defaults': {'JOB_QUEUE': 'normal',
                  'MB': '8192',
                  'NPROC': '4',
                  'job_type': 'lsf',
                  'njobs': '8',
                  'pwatcher_type': 'blocking',
                  'submit': 'bsub -K -n ${NPROC} -q ${JOB_QUEUE} -J '
                            '${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} '
                            '${JOB_SCRIPT}',
                  'use_tmpdir': False},
 'job.high': {},
 'job.highmem': {},
 'job.step.unzip.blasr_aln': {'MB': '32000', 'NPROC': '2', 'njobs': '50'},
 'job.step.unzip.hasm': {'MB': '393216', 'NPROC': '48', 'njobs': '1'},
 'job.step.unzip.phasing': {'MB': '16384', 'NPROC': '2', 'njobs': '100'},
 'job.step.unzip.quiver': {'MB': '98304', 'NPROC': '12', 'njobs': '50'},
 'job.step.unzip.track_reads': {'MB': '393216', 'NPROC': '48', 'njobs': '1'},
 'max_n_open_files': '1000'}
[INFO]PATH=/public/home/zhaoli/software/anaconda3/envs/denovo_asm/bin:/opt/ibm/lsf/10.1/linux3.10-glibc2.17-x86_64/bin:/public/home/zhaoli/software/anaconda3/bin:/public/home/zhaoli/software/anaconda3/condabin:/public/home/zhaoli/perl5/bin:/public/home/software/opt/moudles/Modules/3.2.10/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/public/home/zhaoli/software/srato
olkit.2.8.2-1-centos_linux64/bin:/public/home/zhaoli/software/IGV_2.4.4:/public/home/zhaoli/software/MapSplice-v2.1.8/bin:/public/home/zhaoli/software/bedtools2-2.25.0/bin:/public/home/zhaoli/software/bcftools-1.3.1:/public/home/zhaoli/software/samtools-1.9:/public/home/zhaoli/software/hisat2-2.1.0:/public/home/zhaoli/software/java-1.8.0_144/bin:/public/home/zhaoli/software/GenomeAnalysisTK-3.7:/public/home/zhaoli/software/git/learngit/wtdbg2:/public/home/zhaoli/software/git/learngit/minimap2:/public/home/zhaoli/software/git/learngit/miniasm:/public/home/zhaoli/software/git/learngit/assembly-stats-master/build:/public/home/zhaoli/software/bin:/public/home/zhaoli/software/anaconda3/bin:/public/home/zhaoli/software/cufflinks-2.2.1.Linux_x86_64:/public/home/zhaoli/software/bowtie2-master:/public/home/zhaoli/software/MCScanX:/public/home/zhaoli/software/CAFE/bin:/public/home/zhaoli/software/batmeth2/bin:/public/home/zhaoli/software/FastUniq/source:/public/home/zhaoli/software/NECAT/Linux-amd64/bin:/public/home/zhaoli/git/learngit/idba/bin:/public/home/zhaoli/git/learngit/SOAPdenovo2:/public/home/zhaoli/git/learngit/STAR-Fusion:/public/home/zhaoli/git/learngit/STAR/bin/Linux_x86_64:/public/home/zhaoli/git/learngit/patch-memcpy:/public/home/zhaoli/git/learngit/bwa:/public/home/zhaoli/software/git/learngit/MECAT2/Linux-amd64/bin:/public/home/zhaoli/git/learngit/EasySpeciesTree:/public/home/zhaoli/git/learngit/ASTRAL/astral.5.7.3.jar:/public/home/zhaoli/software/anaconda3/bin/mafft:/public/home/zhaoli/software/anaconda3/bin/raxmlHPC:/public/home/zhaoli/software/anaconda3/bin/trimal:/public/home/zhaoli/software/anaconda3/bin/fasttree:/public/home/zhaoli/software/r8s1.81/src:/public/home/zhaoli/.local/bin:/public/home/zhaoli/bin
[INFO]$('which ls')
[INFO]$('which blasr')
[INFO]$('which samtools')
[INFO]$('which gcpp')
[INFO]$('which minimap2')
[INFO]$('which nucmer')
[INFO]$('which show-coords')
[INFO]$('which falconc')
[INFO]$('nucmer --version')
[INFO]$('minimap2 --version')
[INFO]$('racon --version')
[INFO]$ show-coords -h >
[INFO]$ samtools >
[INFO]samtools ['1', '10'] is >= 1.3
[INFO]CD: '0-rawreads' <- '/public/home/zhaoli/denovo-perennial/pbassembly'
[INFO]CD: '0-rawreads' -> '/public/home/zhaoli/denovo-perennial/pbassembly'
[INFO]CD: '1-preads_ovl' <- '/public/home/zhaoli/denovo-perennial/pbassembly'
[INFO]CD: '1-preads_ovl' -> '/public/home/zhaoli/denovo-perennial/pbassembly'
[INFO]Falcon directories up-to-date.
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.blocking' from '/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pwatcher/blocking.py'>
[INFO]job_type='lsf', (default)job_defaults={'job_type': 'lsf', 'pwatcher_type': 'blocking', 'JOB_QUEUE': 'normal', 'MB': '8192', 'NPROC': '4', 'njobs': '8', 'submit': 'bsub -K -n ${NPROC} -q ${JOB_QUEUE} -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} ${JOB_SCRIPT}', 'use_tmpdir': False}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 8; was None
[INFO]config=
 {'General': {'max_n_open_files': '1000'}, 'Unzip': {'input_fofn': 'input.fofn', 'input_bam_fofn': 'input_bam.fofn', 'polish_include_zmw_all_subreads': True, 'polish_vc_ignore_error': False, 'polish_use_blasr': False}, 'job.defaults': {'job_type': 'lsf', 'pwatcher_type': 'blocking', 'JOB_QUEUE': 'normal', 'MB': '8192', 'NPROC': '4', 'njobs': '8', 'submit': 'bsub -K -n ${NPROC} -q ${JOB_QUEUE} -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} ${JOB_SCRIPT}', 'use_tmpdir': False}, 'job.step.unzip.track_reads': {'njobs': '1', 'NPROC': '48', 'MB': '393216'}, 'job.step.unzip.blasr_aln': {'njobs': '50', 'NPROC': '2', 'MB': '32000'}, 'job.step.unzip.phasing': {'njobs': '100', 'NPROC': '2', 'MB': '16384'}, 'job.step.unzip.hasm': {'njobs': '1', 'NPROC': '48', 'MB': '393216'}, 'job.step.unzip.quiver': {'njobs': '50', 'NPROC': '12', 'MB': '98304'}, 'job.highmem': {}, 'job.high': {}, 'max_n_open_files': '1000'}
[INFO]Num unsatisfied: 2, graph: 5
[INFO]About to submit: Node(3-unzip/reads)
[INFO](slept for another 0.0s -- another 1 loop iterations)
[INFO]Popen: 'bsub -K -n 48 -q normal -J P60beead857406d -o /public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads/run-P60beead857406d.bash.stdout -e /public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads/run-P60beead857406d.bash.stderr /public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pwatcher/mains/job_start.sh'
<<Waiting for dispatch ...>>
[INFO](slept for another 0.30000000000000004s -- another 2 loop iterations)
[INFO](slept for another 1.2000000000000002s -- another 3 loop iterations)
...
[INFO](slept for another 3430.0s -- another 343 loop iterations)
[INFO](slept for another 3440.0s -- another 344 loop iterations)
<<Job is finished>>
[ERROR]Task Node(3-unzip/reads) failed with exit-code=1
[ERROR]Some tasks are recently_done but not satisfied: {Node(3-unzip/reads)}
[ERROR]ready: set()
        submitted: set()
[ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
[ERROR]Error for run(target=clr, config_fn=fc_unzip.cfg, logging_config_fn=None)
Traceback (most recent call last):
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 195, in run
    run_logged(target, config_fn)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 184, in run_logged
    unzip_all(config, unzip_config_fn)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 27, in unzip_all
    tasks_unzip.run_workflow(wf, config, unzip_config_fn)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/tasks/unzip.py", line 453, in run_workflow
    gen_parallel_tasks(
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/pype.py", line 106, in gen_parallel_tasks
    wf.refreshTargets()
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: {Node(3-unzip/reads)}
Traceback (most recent call last):
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/bin/fc_unzip.py", line 33, in <module>
    sys.exit(load_entry_point('falcon-unzip==1.3.7', 'console_scripts', 'fc_unzip.py')())
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/start_unzip.py", line 33, in main
    unzip.run(**vars(args))
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 195, in run
    run_logged(target, config_fn)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 184, in run_logged
    unzip_all(config, unzip_config_fn)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 27, in unzip_all
    tasks_unzip.run_workflow(wf, config, unzip_config_fn)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/tasks/unzip.py", line 453, in run_workflow
    gen_parallel_tasks(
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/pype.py", line 106, in gen_parallel_tasks
    wf.refreshTargets()
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
    self._refreshTargets(updateFreq, exitOnFailure)
  File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/simple_pwatcher_bridge.py", line 362, in _refreshTargets
    raise Exception(msg)
Exception: Some tasks are recently_done but not satisfied: {Node(3-unzip/reads)}

Expected behavior
Thanks to anyone willing to take a look.
@pb-cdunn
Copy link

It looks like your system eventually killed the job, which is goo.

It's import to run a small example first to test for any integration problems. After that, you can re-run 3-unzip/reads manually, off the cluster. Look in its stderr file for hints on how to run the same commands yourself. It's very difficult to debug from here, but there is likely a simple problem.

@lizhao007
Copy link
Author

In fact, I run the example data set FALCON-examples/run/greg200k-sv2 from https://github.com/PacificBiosciences/pb-assembly#example-data-set, but it's still reporting an error at 3-unzip/reads step. There is no change in fc_unzip.ctg.
The err.log is:
`falcon-unzip 1.3.7 (pip thinks "falcon-unzip 1.3.7")
falcon-kit 1.8.1 (pip thinks "falcon-kit 1.8.1")
pypeflow 2.3.0
[INFO]Setup logging from file "None".
[INFO]Default logging setup:

[loggers]
keys=root

[handlers]
keys=stream,file_all

[formatters]
keys=form01,form02

[logger_root]
level=NOTSET
handlers=stream,file_all

[handler_stream]
class=StreamHandler
level=INFO
formatter=form02
args=(sys.stderr,)

[handler_file_all]
class=FileHandler
level=DEBUG
formatter=form01
args=('all.log', 'w')

[formatter_form01]
format=%(asctime)s - %(name)s:%(lineno)d - %(levelname)s - %(message)s

[formatter_form02]
format=[%(levelname)s]%(message)s

[INFO]Using config=
{'General': {'max_n_open_files': '1000'},
'Unzip': {'input_bam_fofn': 'input_bam.fofn',
'input_fofn': 'input.fofn',
'plish_include_zmw_all_subreads': 'true',
'polish_include_zmw_all_subreads': False,
'polish_use_blasr': False,
'polish_vc_ignore_error': False},
'job.defaults': {'JOB_QUEUE': 'normal',
'MB': '8192',
'NPROC': '4',
'job_type': 'lsf',
'njobs': '8',
'pwatcher_type': 'blocking',
'submit': 'bsub -K -n ${NPROC} -q ${JOB_QUEUE} -J '
'${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} '
'${JOB_SCRIPT}',
'use_tmpdir': False},
'job.high': {},
'job.highmem': {},
'job.step.unzip.blasr_aln': {'MB': '32000', 'NPROC': '2', 'njobs': '50'},
'job.step.unzip.hasm': {'MB': '393216', 'NPROC': '48', 'njobs': '1'},
'job.step.unzip.phasing': {'MB': '16384', 'NPROC': '2', 'njobs': '100'},
'job.step.unzip.quiver': {'MB': '98304', 'NPROC': '12', 'njobs': '50'},
'job.step.unzip.track_reads': {'MB': '393216', 'NPROC': '48', 'njobs': '1'},
'max_n_open_files': '1000'}
[INFO]PATH=/public/home/zhaoli/software/anaconda3/envs/denovo_asm/bin:/public/home/zhaoli/software/anaconda3/condabin:/opt/ibm/lsf/10.1/linux3.10-glibc2.17-x86_64/bin:/public/home/zhaoli/perl5/bin:/public/home/software/opt/moudles/Modules/3.2.10/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/public/home/zhaoli/software/sratoolkit.2.8.2-1-centos_linux64/bin:/public/home/zhaoli/software/IGV_2.4.4:/public/home/zhaoli/software/MapSplice-v2.1.8/bin:/public/home/zhaoli/software/bedtools2-2.25.0/bin:/public/home/zhaoli/software/bcftools-1.3.1:/public/home/zhaoli/software/samtools-1.9:/public/home/zhaoli/software/hisat2-2.1.0:/public/home/zhaoli/software/java-1.8.0_144/bin:/public/home/zhaoli/software/GenomeAnalysisTK-3.7:/public/home/zhaoli/software/git/learngit/wtdbg2:/public/home/zhaoli/software/git/learngit/minimap2:/public/home/zhaoli/software/git/learngit/miniasm:/public/home/zhaoli/software/git/learngit/assembly-stats-master/build:/public/home/zhaoli/software/bin:/public/home/zhaoli/software/anaconda3/bin:/public/home/zhaoli/software/cufflinks-2.2.1.Linux_x86_64:/public/home/zhaoli/software/bowtie2-master:/public/home/zhaoli/software/MCScanX:/public/home/zhaoli/software/CAFE/bin:/public/home/zhaoli/software/batmeth2/bin:/public/home/zhaoli/software/FastUniq/source:/public/home/zhaoli/software/NECAT/Linux-amd64/bin:/public/home/zhaoli/git/learngit/idba/bin:/public/home/zhaoli/git/learngit/SOAPdenovo2:/public/home/zhaoli/git/learngit/STAR-Fusion:/public/home/zhaoli/git/learngit/STAR/bin/Linux_x86_64:/public/home/zhaoli/git/learngit/patch-memcpy:/public/home/zhaoli/git/learngit/bwa:/public/home/zhaoli/git/learngit/EasySpeciesTree:/public/home/zhaoli/git/learngit/ASTRAL/astral.5.7.3.jar:/public/home/zhaoli/software/anaconda3/bin/mafft:/public/home/zhaoli/software/anaconda3/bin/raxmlHPC:/public/home/zhaoli/software/anaconda3/bin/trimal:/public/home/zhaoli/software/anaconda3/bin/fasttree:/public/home/zhaoli/software/r8s1.81/src:/public/home/zhaoli/.local/bin:/public/home/zhaoli/bin
[INFO]$('which ls')
[INFO]$('which blasr')
[INFO]$('which samtools')
[INFO]$('which gcpp')
[INFO]$('which minimap2')
[INFO]$('which nucmer')
[INFO]$('which show-coords')
[INFO]$('which falconc')
[INFO]$('nucmer --version')
[INFO]$('minimap2 --version')
[INFO]$('racon --version')
[INFO]$ show-coords -h >
[INFO]$ samtools >
[INFO]samtools ['1', '10'] is >= 1.3
[INFO]CD: '0-rawreads' <- '/public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2'
[INFO]CD: '0-rawreads' -> '/public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2'
[INFO]CD: '1-preads_ovl' <- '/public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2'
[INFO]CD: '1-preads_ovl' -> '/public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2'
[INFO]Falcon directories up-to-date.
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.blocking' from '/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pwatcher/blocking.py'>
[INFO]job_type='lsf', (default)job_defaults={'job_type': 'lsf', 'pwatcher_type': 'blocking', 'JOB_QUEUE': 'normal', 'MB': '8192', 'NPROC': '4', 'njobs': '8', 'submit': 'bsub -K -n ${NPROC} -q ${JOB_QUEUE} -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} ${JOB_SCRIPT}', 'use_tmpdir': False}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 8; was None
[INFO]config=
{'General': {'max_n_open_files': '1000'}, 'Unzip': {'input_fofn': 'input.fofn', 'input_bam_fofn': 'input_bam.fofn', 'plish_include_zmw_all_subreads': 'true', 'polish_vc_ignore_error': False, 'polish_use_blasr': False, 'polish_include_zmw_all_subreads': False}, 'job.defaults': {'job_type': 'lsf', 'pwatcher_type': 'blocking', 'JOB_QUEUE': 'normal', 'MB': '8192', 'NPROC': '4', 'njobs': '8', 'submit': 'bsub -K -n ${NPROC} -q ${JOB_QUEUE} -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} ${JOB_SCRIPT}', 'use_tmpdir': False}, 'job.step.unzip.track_reads': {'njobs': '1', 'NPROC': '48', 'MB': '393216'}, 'job.step.unzip.blasr_aln': {'njobs': '50', 'NPROC': '2', 'MB': '32000'}, 'job.step.unzip.phasing': {'njobs': '100', 'NPROC': '2', 'MB': '16384'}, 'job.step.unzip.hasm': {'njobs': '1', 'NPROC': '48', 'MB': '393216'}, 'job.step.unzip.quiver': {'njobs': '50', 'NPROC': '12', 'MB': '98304'}, 'job.highmem': {}, 'job.high': {}, 'max_n_open_files': '1000'}
[INFO]Num unsatisfied: 0, graph: 5
[INFO]Setting max_jobs to 50; was 8
[INFO]Num unsatisfied: 0, graph: 8
[INFO]Setting max_jobs to 8; was 50
[INFO]Num unsatisfied: 1, graph: 11
[INFO]About to submit: Node(3-unzip/2-htigs/split)
[INFO](slept for another 0.0s -- another 1 loop iterations)
[INFO]Popen: 'bsub -K -n 48 -q normal -J P0a8d4231d83506 -o /public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2/3-unzip/2-htigs/split/run-P0a8d4231d83506.bash.stdout -e /public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2/3-unzip/2-htigs/split/run-P0a8d4231d83506.bash.stderr /public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pwatcher/mains/job_start.sh'
<<Waiting for dispatch ...>>
[INFO](slept for another 0.30000000000000004s -- another 2 loop iterations)
[INFO](slept for another 1.2000000000000002s -- another 3 loop iterations)
[INFO](slept for another 2.9999999999999996s -- another 4 loop iterations)
[INFO](slept for another 6.0s -- another 5 loop iterations)
[INFO](slept for another 10.500000000000002s -- another 6 loop iterations)
[INFO](slept for another 16.800000000000004s -- another 7 loop iterations)
[INFO](slept for another 25.20000000000001s -- another 8 loop iterations)
[INFO](slept for another 36.000000000000014s -- another 9 loop iterations)
[INFO](slept for another 49.499999999999986s -- another 10 loop iterations)
[INFO](slept for another 65.99999999999994s -- another 11 loop iterations)
[INFO](slept for another 85.7999999999999s -- another 12 loop iterations)
[INFO](slept for another 109.19999999999983s -- another 13 loop iterations)
[INFO](slept for another 135.5999999999998s -- another 14 loop iterations)
[INFO](slept for another 150.0s -- another 15 loop iterations)
[INFO](slept for another 160.0s -- another 16 loop iterations)
[INFO](slept for another 170.0s -- another 17 loop iterations)
[INFO](slept for another 180.0s -- another 18 loop iterations)
[INFO](slept for another 190.0s -- another 19 loop iterations)
[INFO](slept for another 200.0s -- another 20 loop iterations)
[INFO](slept for another 210.0s -- another 21 loop iterations)
[INFO](slept for another 220.0s -- another 22 loop iterations)
[INFO](slept for another 230.0s -- another 23 loop iterations)
[INFO](slept for another 240.0s -- another 24 loop iterations)
[INFO](slept for another 250.0s -- another 25 loop iterations)
[INFO](slept for another 260.0s -- another 26 loop iterations)
[INFO](slept for another 270.0s -- another 27 loop iterations)
[INFO](slept for another 280.0s -- another 28 loop iterations)
[INFO](slept for another 290.0s -- another 29 loop iterations)
[INFO](slept for another 300.0s -- another 30 loop iterations)
[INFO](slept for another 310.0s -- another 31 loop iterations)
[INFO](slept for another 320.0s -- another 32 loop iterations)
[INFO](slept for another 330.0s -- another 33 loop iterations)
[INFO](slept for another 340.0s -- another 34 loop iterations)
[INFO](slept for another 350.0s -- another 35 loop iterations)
[INFO](slept for another 360.0s -- another 36 loop iterations)
[INFO](slept for another 370.0s -- another 37 loop iterations)
[INFO](slept for another 380.0s -- another 38 loop iterations)
[INFO](slept for another 390.0s -- another 39 loop iterations)
[INFO](slept for another 400.0s -- another 40 loop iterations)
[INFO](slept for another 410.0s -- another 41 loop iterations)
[INFO](slept for another 420.0s -- another 42 loop iterations)
[INFO](slept for another 430.0s -- another 43 loop iterations)
[INFO](slept for another 440.0s -- another 44 loop iterations)
[INFO](slept for another 450.0s -- another 45 loop iterations)
[INFO](slept for another 460.0s -- another 46 loop iterations)
[INFO](slept for another 470.0s -- another 47 loop iterations)
[INFO](slept for another 480.0s -- another 48 loop iterations)
[INFO](slept for another 490.0s -- another 49 loop iterations)
[INFO](slept for another 500.0s -- another 50 loop iterations)
[INFO](slept for another 510.0s -- another 51 loop iterations)
[INFO](slept for another 520.0s -- another 52 loop iterations)
[INFO](slept for another 530.0s -- another 53 loop iterations)
[INFO](slept for another 540.0s -- another 54 loop iterations)
[INFO](slept for another 550.0s -- another 55 loop iterations)
[INFO](slept for another 560.0s -- another 56 loop iterations)
[INFO](slept for another 570.0s -- another 57 loop iterations)
[INFO](slept for another 580.0s -- another 58 loop iterations)
[INFO](slept for another 590.0s -- another 59 loop iterations)
[INFO](slept for another 600.0s -- another 60 loop iterations)
[INFO](slept for another 610.0s -- another 61 loop iterations)
[INFO](slept for another 620.0s -- another 62 loop iterations)
[INFO](slept for another 630.0s -- another 63 loop iterations)
[INFO](slept for another 640.0s -- another 64 loop iterations)
[INFO](slept for another 650.0s -- another 65 loop iterations)
[INFO](slept for another 660.0s -- another 66 loop iterations)
[INFO](slept for another 670.0s -- another 67 loop iterations)
[INFO](slept for another 680.0s -- another 68 loop iterations)
[INFO](slept for another 690.0s -- another 69 loop iterations)
[INFO](slept for another 700.0s -- another 70 loop iterations)
[INFO](slept for another 710.0s -- another 71 loop iterations)
[INFO](slept for another 720.0s -- another 72 loop iterations)
[INFO](slept for another 730.0s -- another 73 loop iterations)
[INFO](slept for another 740.0s -- another 74 loop iterations)
[INFO](slept for another 750.0s -- another 75 loop iterations)
[INFO](slept for another 760.0s -- another 76 loop iterations)
[INFO](slept for another 770.0s -- another 77 loop iterations)
[INFO](slept for another 780.0s -- another 78 loop iterations)
[INFO](slept for another 790.0s -- another 79 loop iterations)
[INFO](slept for another 800.0s -- another 80 loop iterations)
[INFO](slept for another 810.0s -- another 81 loop iterations)
[INFO](slept for another 820.0s -- another 82 loop iterations)
[INFO](slept for another 830.0s -- another 83 loop iterations)
[INFO](slept for another 840.0s -- another 84 loop iterations)
[INFO](slept for another 850.0s -- another 85 loop iterations)
[INFO](slept for another 860.0s -- another 86 loop iterations)
[INFO](slept for another 870.0s -- another 87 loop iterations)
[INFO](slept for another 880.0s -- another 88 loop iterations)
[INFO](slept for another 890.0s -- another 89 loop iterations)
[INFO](slept for another 900.0s -- another 90 loop iterations)
[INFO](slept for another 910.0s -- another 91 loop iterations)
[INFO](slept for another 920.0s -- another 92 loop iterations)
[INFO](slept for another 930.0s -- another 93 loop iterations)
[INFO](slept for another 940.0s -- another 94 loop iterations)
[INFO](slept for another 950.0s -- another 95 loop iterations)
[INFO](slept for another 960.0s -- another 96 loop iterations)
[INFO](slept for another 970.0s -- another 97 loop iterations)
[INFO](slept for another 980.0s -- another 98 loop iterations)
[INFO](slept for another 990.0s -- another 99 loop iterations)
[INFO](slept for another 1000.0s -- another 100 loop iterations)
[INFO](slept for another 1010.0s -- another 101 loop iterations)
[INFO](slept for another 1020.0s -- another 102 loop iterations)
[INFO](slept for another 1030.0s -- another 103 loop iterations)
[INFO](slept for another 1040.0s -- another 104 loop iterations)
[INFO](slept for another 1050.0s -- another 105 loop iterations)
[INFO](slept for another 1060.0s -- another 106 loop iterations)
[INFO](slept for another 1070.0s -- another 107 loop iterations)
[INFO](slept for another 1080.0s -- another 108 loop iterations)
[INFO](slept for another 1090.0s -- another 109 loop iterations)
[INFO](slept for another 1100.0s -- another 110 loop iterations)
[ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/bin/fc_unzip.py", line 33, in
sys.exit(load_entry_point('falcon-unzip==1.3.7', 'console_scripts', 'fc_unzip.py')())
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/start_unzip.py", line 33, in main
unzip.run(**vars(args))
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 195, in run
run_logged(target, config_fn)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 184, in run_logged
unzip_all(config, unzip_config_fn)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 27, in unzip_all
tasks_unzip.run_workflow(wf, config, unzip_config_fn)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/tasks/unzip.py", line 527, in run_workflow
gen_parallel_tasks(
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/pype.py", line 106, in gen_parallel_tasks
wf.refreshTargets()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
self._refreshTargets(updateFreq, exitOnFailure)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/simple_pwatcher_bridge.py", line 340, in _refreshTargets
time.sleep(sleep_time)
KeyboardInterrupt`
I am confused what went wrong.

@pb-cdunn
Copy link

pb-cdunn commented Feb 2, 2021 

[INFO]Popen: 'bsub -K -n 48 -q normal -J P0a8d4231d83506 -o /public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2/3-unzip/2-htigs/split/run-P0a8d4231d83506.bash.stdout -e /public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2/3-unzip/2-htigs/split/run-P0a8d4231d83506.bash.stderr /public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pwatcher/mains/job_start.sh'

That's not finishing. It might be hanging, or it could just be very, very time-consuming. I'd try running that (a script underneath it) myself and debug from there.

@lizhao007
Copy link
Author

Thanks a lot. Now I have successfully passed all steps with the test data under LSF cluster, but it still reports an error in 3-unzip/reads/ step with my own data. The last 100 lines of reported error in 3-unzip/reads/run-P60beead857406d.bash.stderr is:
`before Read_Overlap record j = 411000000 out of 562406860 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00255/uow-00/raw_reads.329.las Thu Feb 4 20:05:50 2021
before Read_Overlap record j = 460000000 out of 566511838 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00254/uow-00/raw_reads.328.las Thu Feb 4 20:05:51 2021
before Read_Overlap record j = 569000000 out of 570365563 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00253/uow-00/raw_reads.327.las Thu Feb 4 20:05:51 2021
before Read_Overlap record j = 118000000 out of 550930393 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00260/uow-00/raw_reads.333.las Thu Feb 4 20:05:51 2021
before Read_Overlap record j = 344000000 out of 458600561 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00256/uow-00/raw_reads.33.las Thu Feb 4 20:05:51 2021
before Read_Overlap record j = 176000000 out of 535385431 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00259/uow-00/raw_reads.332.las Thu Feb 4 20:05:52 2021
before Read_Overlap record j = 257000000 out of 568463961 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00257/uow-00/raw_reads.330.las Thu Feb 4 20:05:52 2021
before Read_Overlap record j = 192000000 out of 537292492 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00258/uow-00/raw_reads.331.las Thu Feb 4 20:05:52 2021
before Read_Overlap record j = 461000000 out of 566511838 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00254/uow-00/raw_reads.328.las Thu Feb 4 20:05:52 2021
before Read_Overlap record j = 412000000 out of 562406860 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00255/uow-00/raw_reads.329.las Thu Feb 4 20:05:53 2021
before Read_Overlap record j = 570000000 out of 570365563 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00253/uow-00/raw_reads.327.las Thu Feb 4 20:05:53 2021
before Read_Overlap record j = 119000000 out of 550930393 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00260/uow-00/raw_reads.333.las Thu Feb 4 20:05:53 2021
before Read_Overlap record j = 345000000 out of 458600561 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00256/uow-00/raw_reads.33.las Thu Feb 4 20:05:54 2021

completed loop record j = 570365563 out of 570365563 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00253/uow-00/raw_reads.327.las Thu Feb 4 20:05:54 2021

before Read_Overlap record j = 177000000 out of 535385431 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00259/uow-00/raw_reads.332.las Thu Feb 4 20:05:54 2021
before Read_Overlap record j = 258000000 out of 568463961 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00257/uow-00/raw_reads.330.las Thu Feb 4 20:05:54 2021
before Read_Overlap record j = 193000000 out of 537292492 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00258/uow-00/raw_reads.331.las Thu Feb 4 20:05:55 2021
before Read_Overlap record j = 462000000 out of 566511838 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00254/uow-00/raw_reads.328.las Thu Feb 4 20:05:55 2021
before Read_Overlap record j = 413000000 out of 562406860 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00255/uow-00/raw_reads.329.las Thu Feb 4 20:05:55 2021
before Read_Overlap record j = 346000000 out of 458600561 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00256/uow-00/raw_reads.33.las Thu Feb 4 20:05:56 2021
before Read_Overlap record j = 120000000 out of 550930393 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00260/uow-00/raw_reads.333.las Thu Feb 4 20:05:56 2021
before Read_Overlap record j = 178000000 out of 535385431 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00259/uow-00/raw_reads.332.las Thu Feb 4 20:05:56 2021
before Read_Overlap record j = 259000000 out of 568463961 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00257/uow-00/raw_reads.330.las Thu Feb 4 20:05:56 2021
before Read_Overlap record j = 194000000 out of 537292492 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00258/uow-00/raw_reads.331.las Thu Feb 4 20:05:57 2021
before Read_Overlap record j = 463000000 out of 566511838 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00254/uow-00/raw_reads.328.las Thu Feb 4 20:05:57 2021
before Read_Overlap record j = 414000000 out of 562406860 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00255/uow-00/raw_reads.329.las Thu Feb 4 20:05:57 2021
before Read_Overlap record j = 347000000 out of 458600561 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00256/uow-00/raw_reads.33.las Thu Feb 4 20:05:58 2021
before Read_Overlap record j = 179000000 out of 535385431 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00259/uow-00/raw_reads.332.las Thu Feb 4 20:05:58 2021
before Read_Overlap record j = 121000000 out of 550930393 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00260/uow-00/raw_reads.333.las Thu Feb 4 20:05:58 2021
before Read_Overlap record j = 260000000 out of 568463961 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00257/uow-00/raw_reads.330.las Thu Feb 4 20:05:58 2021
before Read_Overlap record j = 195000000 out of 537292492 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00258/uow-00/raw_reads.331.las Thu Feb 4 20:05:59 2021
before Read_Overlap record j = 464000000 out of 566511838 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00254/uow-00/raw_reads.328.las Thu Feb 4 20:05:59 2021
before Read_Overlap record j = 348000000 out of 458600561 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00256/uow-00/raw_reads.33.las Thu Feb 4 20:05:59 2021
before Read_Overlap record j = 415000000 out of 562406860 at /public/home/zhaoli/denovo-perennial/pbassembly/0-rawreads/las-merge-gathered/../las-merge-runs/m_00255/uow-00/raw_reads.329.las Thu Feb 4 20:05:59 2021
[895707]preparing tr_stage1
[895707]maxrss: 19408
[895707]terminating track_reads workers...
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/util/io.py", line 68, in run_func
ret = func(*args)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 28, in run_tr_stage1
return fn, tr_stage1(reader.readlines, min_len, bestn, rid_to_ctg)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 37, in tr_stage1
for l in readlines():
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/util/io.py", line 205, in readlines
output, _ = self.proc.communicate()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/subprocess.py", line 1011, in communicate
stdout = self.stdout.read()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
MemoryError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/util/io.py", line 73, in run_func
raise Exception(traceback.format_exc())
Exception: Traceback (most recent call last):
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/util/io.py", line 68, in run_func
ret = func(*args)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 28, in run_tr_stage1
return fn, tr_stage1(reader.readlines, min_len, bestn, rid_to_ctg)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 37, in tr_stage1
for l in readlines():
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/util/io.py", line 205, in readlines
output, _ = self.proc.communicate()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/subprocess.py", line 1011, in communicate
stdout = self.stdout.read()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
MemoryError

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 211, in
main()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 207, in main
track_reads(**vars(args))
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 156, in track_reads
try_run_track_reads(n_core, min_len, bestn, db_fn, las_fofn_fn, output_fn)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 134, in try_run_track_reads
run_track_reads(exe_pool, file_list, min_len, bestn, db_fn, output_fn)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 76, in run_track_reads
for fn, res in exe_pool.imap(io.run_func, inputs):
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/multiprocessing/pool.py", line 868, in next
raise value
Exception: Traceback (most recent call last):
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/util/io.py", line 68, in run_func
ret = func(*args)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 28, in run_tr_stage1
return fn, tr_stage1(reader.readlines, min_len, bestn, rid_to_ctg)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 37, in tr_stage1
for l in readlines():
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/util/io.py", line 205, in readlines
output, _ = self.proc.communicate()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/subprocess.py", line 1011, in communicate
stdout = self.stdout.read()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
MemoryError

2021-02-04 20:18:27,487 - root - WARNING - Call '/bin/bash user_script.sh' returned 256.
2021-02-04 20:18:27,487 - root - INFO - CD: '/public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads' -> '/public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads'
2021-02-04 20:18:27,488 - root - INFO - CD: '/public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads' -> '/public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads'
2021-02-04 20:18:27,488 - root - CRITICAL - Error in /public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py with args="{'json_fn': '/public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads/task.json',\n 'timeout': 30,\n 'tmpdir': None}"
Traceback (most recent call last):
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py", line 280, in
main()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py", line 272, in main
run(**vars(parsed_args))
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py", line 266, in run
run_cfg_in_tmpdir(cfg, tmpdir, '.')
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py", line 241, in run_cfg_in_tmpdir
run_bash(bash_template, myinputs, myoutputs, parameters)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/do_task.py", line 200, in run_bash
util.system(cmd)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/io.py", line 27, in syscall
raise Exception(msg)
Exception: Call '/bin/bash user_script.sh' returned 256.
+++ pwd
++ echo 'FAILURE. Running top in /public/home/zhaoli/denovo-perennial/pbassembly/3-unzip/reads (If you see -terminal database is inaccessible- you are using the python bin-wrapper, so you will not get diagnostic info. No big deal. This process is crashing anyway.)'
++ rm -f top.txt
++ which python
++ which top
++ env -u LD_LIBRARY_PATH top -b -n 1
++ env -u LD_LIBRARY_PATH top -b -n 1
++ pstree -apl

real 691m7.210s
user 8253m7.991s
sys 1820m38.160s

  • finish
  • echo 'finish code: 1'`

@pb-cdunn
Copy link 

File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 28, in run_tr_stage1
return fn, tr_stage1(reader.readlines, min_len, bestn, rid_to_ctg)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/rr_ctg_track.py", line 37, in tr_stage1
for l in readlines():
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/util/io.py", line 205, in readlines
output, _ = self.proc.communicate()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/subprocess.py", line 1011, in communicate
stdout = self.stdout.read()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
MemoryError

I guess it's running out of memory.

@lizhao007
Copy link
Author

Yes, but I do not know which step is out of memory and have no idea to fix it. This is the fc_unzip.cfg I used, when I doubled the required memory, it reported the same error.
`[General]
max_n_open_files = 1000

[Unzip]
input_fofn=input.fofn
input_bam_fofn=input_bam.fofn
polish_include_zmw_all_subreads = true

[job.defaults]
job_type=lsf
pwatcher_type=blocking
JOB_QUEUE=high
MB=8192
NPROC=4
njobs=8
submit =bsub -K -n ${NPROC} -q ${JOB_QUEUE} -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} ${JOB_SCRIPT}

[job.step.unzip.track_reads]
njobs=1
NPROC=30
MB=393216

uses minimap2 now

[job.step.unzip.blasr_aln]
njobs=50
NPROC=2
MB=32000
[job.step.unzip.phasing]
njobs=100
NPROC=2
MB=16384
[job.step.unzip.hasm]
njobs=1
NPROC=30
MB=393216

uses arrow now

[job.step.unzip.quiver]
njobs=50
NPROC=12
MB=98304`
And the output directory in pbassembly/3-unzip/reads is:
drwxr-xr-x 2 zhaoli LLi 4096 Jan 22 10:33 dump_pread_ids
drwxr-xr-x 2 zhaoli LLi 4096 Jan 22 10:33 dump_rawread_ids
drwxr-xr-x 2 zhaoli LLi 4096 Jan 22 10:33 get_read_ctg_map
-rw-r--r-- 1 zhaoli LLi 5966863885 Jan 31 04:15 rawread_to_contigs
-rwxr-xr-x 1 zhaoli LLi 253 Feb 18 09:44 run-P60beead857406d.bash
-rw-r--r-- 1 zhaoli LLi 30211915 Feb 18 09:52 run-P60beead857406d.bash.stderr
-rw-r--r-- 1 zhaoli LLi 302926 Feb 18 09:52 run-P60beead857406d.bash.stdout
-rw-r--r-- 1 zhaoli LLi 205 Feb 18 09:44 run.sh
-rw-r--r-- 1 zhaoli LLi 797 Feb 18 09:44 task.json
-rw-r--r-- 1 zhaoli LLi 578 Feb 18 09:44 task.sh
-rw-r--r-- 1 zhaoli LLi 643 Feb 18 09:44 template.sh
-rw-r--r-- 1 zhaoli LLi 34906 Feb 18 09:52 top.txt
-rw-r--r-- 1 zhaoli LLi 772 Feb 18 09:44 user_script.sh
Thanks for your help.

@pb-cdunn
Copy link

You can go into 3-unzip/reads an run a .sh file yourself. Look at user_script.sh. If you run that from that directory on your local machine, you can watch for memory usage of each process.

We strongly recommend that you switch to HiFi (i.e. CCS) reads and use ipa.

@wzxie
Copy link

wzxie commented May 23, 2021 •
edited
Loading

In fact, I run the example data set FALCON-examples/run/greg200k-sv2 from https://github.com/PacificBiosciences/pb-assembly#example-data-set, but it's still reporting an error at 3-unzip/reads step. There is no change in fc_unzip.ctg.
The err.log is:
`falcon-unzip 1.3.7 (pip thinks "falcon-unzip 1.3.7")
falcon-kit 1.8.1 (pip thinks "falcon-kit 1.8.1")
pypeflow 2.3.0
[INFO]Setup logging from file "None".
[INFO]Default logging setup:

[loggers]
keys=root

[handlers]
keys=stream,file_all

[formatters]
keys=form01,form02

[logger_root]
level=NOTSET
handlers=stream,file_all

[handler_stream]
class=StreamHandler
level=INFO
formatter=form02
args=(sys.stderr,)

[handler_file_all]
class=FileHandler
level=DEBUG
formatter=form01
args=('all.log', 'w')

[formatter_form01]
format=%(asctime)s - %(name)s:%(lineno)d - %(levelname)s - %(message)s

[formatter_form02]
format=[%(levelname)s]%(message)s

[INFO]Using config=
{'General': {'max_n_open_files': '1000'},
'Unzip': {'input_bam_fofn': 'input_bam.fofn',
'input_fofn': 'input.fofn',
'plish_include_zmw_all_subreads': 'true',
'polish_include_zmw_all_subreads': False,
'polish_use_blasr': False,
'polish_vc_ignore_error': False},
'job.defaults': {'JOB_QUEUE': 'normal',
'MB': '8192',
'NPROC': '4',
'job_type': 'lsf',
'njobs': '8',
'pwatcher_type': 'blocking',
'submit': 'bsub -K -n ${NPROC} -q ${JOB_QUEUE} -J '
'${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} '
'${JOB_SCRIPT}',
'use_tmpdir': False},
'job.high': {},
'job.highmem': {},
'job.step.unzip.blasr_aln': {'MB': '32000', 'NPROC': '2', 'njobs': '50'},
'job.step.unzip.hasm': {'MB': '393216', 'NPROC': '48', 'njobs': '1'},
'job.step.unzip.phasing': {'MB': '16384', 'NPROC': '2', 'njobs': '100'},
'job.step.unzip.quiver': {'MB': '98304', 'NPROC': '12', 'njobs': '50'},
'job.step.unzip.track_reads': {'MB': '393216', 'NPROC': '48', 'njobs': '1'},
'max_n_open_files': '1000'}
[INFO]PATH=/public/home/zhaoli/software/anaconda3/envs/denovo_asm/bin:/public/home/zhaoli/software/anaconda3/condabin:/opt/ibm/lsf/10.1/linux3.10-glibc2.17-x86_64/bin:/public/home/zhaoli/perl5/bin:/public/home/software/opt/moudles/Modules/3.2.10/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/public/home/zhaoli/software/sratoolkit.2.8.2-1-centos_linux64/bin:/public/home/zhaoli/software/IGV_2.4.4:/public/home/zhaoli/software/MapSplice-v2.1.8/bin:/public/home/zhaoli/software/bedtools2-2.25.0/bin:/public/home/zhaoli/software/bcftools-1.3.1:/public/home/zhaoli/software/samtools-1.9:/public/home/zhaoli/software/hisat2-2.1.0:/public/home/zhaoli/software/java-1.8.0_144/bin:/public/home/zhaoli/software/GenomeAnalysisTK-3.7:/public/home/zhaoli/software/git/learngit/wtdbg2:/public/home/zhaoli/software/git/learngit/minimap2:/public/home/zhaoli/software/git/learngit/miniasm:/public/home/zhaoli/software/git/learngit/assembly-stats-master/build:/public/home/zhaoli/software/bin:/public/home/zhaoli/software/anaconda3/bin:/public/home/zhaoli/software/cufflinks-2.2.1.Linux_x86_64:/public/home/zhaoli/software/bowtie2-master:/public/home/zhaoli/software/MCScanX:/public/home/zhaoli/software/CAFE/bin:/public/home/zhaoli/software/batmeth2/bin:/public/home/zhaoli/software/FastUniq/source:/public/home/zhaoli/software/NECAT/Linux-amd64/bin:/public/home/zhaoli/git/learngit/idba/bin:/public/home/zhaoli/git/learngit/SOAPdenovo2:/public/home/zhaoli/git/learngit/STAR-Fusion:/public/home/zhaoli/git/learngit/STAR/bin/Linux_x86_64:/public/home/zhaoli/git/learngit/patch-memcpy:/public/home/zhaoli/git/learngit/bwa:/public/home/zhaoli/git/learngit/EasySpeciesTree:/public/home/zhaoli/git/learngit/ASTRAL/astral.5.7.3.jar:/public/home/zhaoli/software/anaconda3/bin/mafft:/public/home/zhaoli/software/anaconda3/bin/raxmlHPC:/public/home/zhaoli/software/anaconda3/bin/trimal:/public/home/zhaoli/software/anaconda3/bin/fasttree:/public/home/zhaoli/software/r8s1.81/src:/public/home/zhaoli/.local/bin:/public/home/zhaoli/bin
[INFO]$('which ls')
[INFO]$('which blasr')
[INFO]$('which samtools')
[INFO]$('which gcpp')
[INFO]$('which minimap2')
[INFO]$('which nucmer')
[INFO]$('which show-coords')
[INFO]$('which falconc')
[INFO]$('nucmer --version')
[INFO]$('minimap2 --version')
[INFO]$('racon --version')
[INFO]$ show-coords -h >
[INFO]$ samtools >
[INFO]samtools ['1', '10'] is >= 1.3
[INFO]CD: '0-rawreads' <- '/public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2'
[INFO]CD: '0-rawreads' -> '/public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2'
[INFO]CD: '1-preads_ovl' <- '/public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2'
[INFO]CD: '1-preads_ovl' -> '/public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2'
[INFO]Falcon directories up-to-date.
[INFO]In simple_pwatcher_bridge, pwatcher_impl=<module 'pwatcher.blocking' from '/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pwatcher/blocking.py'>
[INFO]job_type='lsf', (default)job_defaults={'job_type': 'lsf', 'pwatcher_type': 'blocking', 'JOB_QUEUE': 'normal', 'MB': '8192', 'NPROC': '4', 'njobs': '8', 'submit': 'bsub -K -n ${NPROC} -q ${JOB_QUEUE} -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} ${JOB_SCRIPT}', 'use_tmpdir': False}, use_tmpdir=False, squash=False, job_name_style=0
[INFO]Setting max_jobs to 8; was None
[INFO]config=
{'General': {'max_n_open_files': '1000'}, 'Unzip': {'input_fofn': 'input.fofn', 'input_bam_fofn': 'input_bam.fofn', 'plish_include_zmw_all_subreads': 'true', 'polish_vc_ignore_error': False, 'polish_use_blasr': False, 'polish_include_zmw_all_subreads': False}, 'job.defaults': {'job_type': 'lsf', 'pwatcher_type': 'blocking', 'JOB_QUEUE': 'normal', 'MB': '8192', 'NPROC': '4', 'njobs': '8', 'submit': 'bsub -K -n ${NPROC} -q ${JOB_QUEUE} -J ${JOB_NAME} -o ${JOB_STDOUT} -e ${JOB_STDERR} ${JOB_SCRIPT}', 'use_tmpdir': False}, 'job.step.unzip.track_reads': {'njobs': '1', 'NPROC': '48', 'MB': '393216'}, 'job.step.unzip.blasr_aln': {'njobs': '50', 'NPROC': '2', 'MB': '32000'}, 'job.step.unzip.phasing': {'njobs': '100', 'NPROC': '2', 'MB': '16384'}, 'job.step.unzip.hasm': {'njobs': '1', 'NPROC': '48', 'MB': '393216'}, 'job.step.unzip.quiver': {'njobs': '50', 'NPROC': '12', 'MB': '98304'}, 'job.highmem': {}, 'job.high': {}, 'max_n_open_files': '1000'}
[INFO]Num unsatisfied: 0, graph: 5
[INFO]Setting max_jobs to 50; was 8
[INFO]Num unsatisfied: 0, graph: 8
[INFO]Setting max_jobs to 8; was 50
[INFO]Num unsatisfied: 1, graph: 11
[INFO]About to submit: Node(3-unzip/2-htigs/split)
[INFO](slept for another 0.0s -- another 1 loop iterations)
[INFO]Popen: 'bsub -K -n 48 -q normal -J P0a8d4231d83506 -o /public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2/3-unzip/2-htigs/split/run-P0a8d4231d83506.bash.stdout -e /public/home/zhaoli/denovo-perennial/pbassembly/FALCON-examples/run/greg200k-sv2/3-unzip/2-htigs/split/run-P0a8d4231d83506.bash.stderr /public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pwatcher/mains/job_start.sh'
<<Waiting for dispatch ...>>
[INFO](slept for another 0.30000000000000004s -- another 2 loop iterations)
[INFO](slept for another 1.2000000000000002s -- another 3 loop iterations)
[INFO](slept for another 2.9999999999999996s -- another 4 loop iterations)
[INFO](slept for another 6.0s -- another 5 loop iterations)
[INFO](slept for another 10.500000000000002s -- another 6 loop iterations)
[INFO](slept for another 16.800000000000004s -- another 7 loop iterations)
[INFO](slept for another 25.20000000000001s -- another 8 loop iterations)
[INFO](slept for another 36.000000000000014s -- another 9 loop iterations)
[INFO](slept for another 49.499999999999986s -- another 10 loop iterations)
[INFO](slept for another 65.99999999999994s -- another 11 loop iterations)
[INFO](slept for another 85.7999999999999s -- another 12 loop iterations)
[INFO](slept for another 109.19999999999983s -- another 13 loop iterations)
[INFO](slept for another 135.5999999999998s -- another 14 loop iterations)
[INFO](slept for another 150.0s -- another 15 loop iterations)
[INFO](slept for another 160.0s -- another 16 loop iterations)
[INFO](slept for another 170.0s -- another 17 loop iterations)
[INFO](slept for another 180.0s -- another 18 loop iterations)
[INFO](slept for another 190.0s -- another 19 loop iterations)
[INFO](slept for another 200.0s -- another 20 loop iterations)
[INFO](slept for another 210.0s -- another 21 loop iterations)
[INFO](slept for another 220.0s -- another 22 loop iterations)
[INFO](slept for another 230.0s -- another 23 loop iterations)
[INFO](slept for another 240.0s -- another 24 loop iterations)
[INFO](slept for another 250.0s -- another 25 loop iterations)
[INFO](slept for another 260.0s -- another 26 loop iterations)
[INFO](slept for another 270.0s -- another 27 loop iterations)
[INFO](slept for another 280.0s -- another 28 loop iterations)
[INFO](slept for another 290.0s -- another 29 loop iterations)
[INFO](slept for another 300.0s -- another 30 loop iterations)
[INFO](slept for another 310.0s -- another 31 loop iterations)
[INFO](slept for another 320.0s -- another 32 loop iterations)
[INFO](slept for another 330.0s -- another 33 loop iterations)
[INFO](slept for another 340.0s -- another 34 loop iterations)
[INFO](slept for another 350.0s -- another 35 loop iterations)
[INFO](slept for another 360.0s -- another 36 loop iterations)
[INFO](slept for another 370.0s -- another 37 loop iterations)
[INFO](slept for another 380.0s -- another 38 loop iterations)
[INFO](slept for another 390.0s -- another 39 loop iterations)
[INFO](slept for another 400.0s -- another 40 loop iterations)
[INFO](slept for another 410.0s -- another 41 loop iterations)
[INFO](slept for another 420.0s -- another 42 loop iterations)
[INFO](slept for another 430.0s -- another 43 loop iterations)
[INFO](slept for another 440.0s -- another 44 loop iterations)
[INFO](slept for another 450.0s -- another 45 loop iterations)
[INFO](slept for another 460.0s -- another 46 loop iterations)
[INFO](slept for another 470.0s -- another 47 loop iterations)
[INFO](slept for another 480.0s -- another 48 loop iterations)
[INFO](slept for another 490.0s -- another 49 loop iterations)
[INFO](slept for another 500.0s -- another 50 loop iterations)
[INFO](slept for another 510.0s -- another 51 loop iterations)
[INFO](slept for another 520.0s -- another 52 loop iterations)
[INFO](slept for another 530.0s -- another 53 loop iterations)
[INFO](slept for another 540.0s -- another 54 loop iterations)
[INFO](slept for another 550.0s -- another 55 loop iterations)
[INFO](slept for another 560.0s -- another 56 loop iterations)
[INFO](slept for another 570.0s -- another 57 loop iterations)
[INFO](slept for another 580.0s -- another 58 loop iterations)
[INFO](slept for another 590.0s -- another 59 loop iterations)
[INFO](slept for another 600.0s -- another 60 loop iterations)
[INFO](slept for another 610.0s -- another 61 loop iterations)
[INFO](slept for another 620.0s -- another 62 loop iterations)
[INFO](slept for another 630.0s -- another 63 loop iterations)
[INFO](slept for another 640.0s -- another 64 loop iterations)
[INFO](slept for another 650.0s -- another 65 loop iterations)
[INFO](slept for another 660.0s -- another 66 loop iterations)
[INFO](slept for another 670.0s -- another 67 loop iterations)
[INFO](slept for another 680.0s -- another 68 loop iterations)
[INFO](slept for another 690.0s -- another 69 loop iterations)
[INFO](slept for another 700.0s -- another 70 loop iterations)
[INFO](slept for another 710.0s -- another 71 loop iterations)
[INFO](slept for another 720.0s -- another 72 loop iterations)
[INFO](slept for another 730.0s -- another 73 loop iterations)
[INFO](slept for another 740.0s -- another 74 loop iterations)
[INFO](slept for another 750.0s -- another 75 loop iterations)
[INFO](slept for another 760.0s -- another 76 loop iterations)
[INFO](slept for another 770.0s -- another 77 loop iterations)
[INFO](slept for another 780.0s -- another 78 loop iterations)
[INFO](slept for another 790.0s -- another 79 loop iterations)
[INFO](slept for another 800.0s -- another 80 loop iterations)
[INFO](slept for another 810.0s -- another 81 loop iterations)
[INFO](slept for another 820.0s -- another 82 loop iterations)
[INFO](slept for another 830.0s -- another 83 loop iterations)
[INFO](slept for another 840.0s -- another 84 loop iterations)
[INFO](slept for another 850.0s -- another 85 loop iterations)
[INFO](slept for another 860.0s -- another 86 loop iterations)
[INFO](slept for another 870.0s -- another 87 loop iterations)
[INFO](slept for another 880.0s -- another 88 loop iterations)
[INFO](slept for another 890.0s -- another 89 loop iterations)
[INFO](slept for another 900.0s -- another 90 loop iterations)
[INFO](slept for another 910.0s -- another 91 loop iterations)
[INFO](slept for another 920.0s -- another 92 loop iterations)
[INFO](slept for another 930.0s -- another 93 loop iterations)
[INFO](slept for another 940.0s -- another 94 loop iterations)
[INFO](slept for another 950.0s -- another 95 loop iterations)
[INFO](slept for another 960.0s -- another 96 loop iterations)
[INFO](slept for another 970.0s -- another 97 loop iterations)
[INFO](slept for another 980.0s -- another 98 loop iterations)
[INFO](slept for another 990.0s -- another 99 loop iterations)
[INFO](slept for another 1000.0s -- another 100 loop iterations)
[INFO](slept for another 1010.0s -- another 101 loop iterations)
[INFO](slept for another 1020.0s -- another 102 loop iterations)
[INFO](slept for another 1030.0s -- another 103 loop iterations)
[INFO](slept for another 1040.0s -- another 104 loop iterations)
[INFO](slept for another 1050.0s -- another 105 loop iterations)
[INFO](slept for another 1060.0s -- another 106 loop iterations)
[INFO](slept for another 1070.0s -- another 107 loop iterations)
[INFO](slept for another 1080.0s -- another 108 loop iterations)
[INFO](slept for another 1090.0s -- another 109 loop iterations)
[INFO](slept for another 1100.0s -- another 110 loop iterations)
[ERROR]Noop. We cannot kill blocked threads. Hopefully, everything will die on SIGTERM.
Traceback (most recent call last):
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/bin/fc_unzip.py", line 33, in
sys.exit(load_entry_point('falcon-unzip==1.3.7', 'console_scripts', 'fc_unzip.py')())
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/mains/start_unzip.py", line 33, in main
unzip.run(**vars(args))
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 195, in run
run_logged(target, config_fn)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 184, in run_logged
unzip_all(config, unzip_config_fn)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/unzip.py", line 27, in unzip_all
tasks_unzip.run_workflow(wf, config, unzip_config_fn)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_unzip/tasks/unzip.py", line 527, in run_workflow
gen_parallel_tasks(
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/falcon_kit/pype.py", line 106, in gen_parallel_tasks
wf.refreshTargets()
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/simple_pwatcher_bridge.py", line 278, in refreshTargets
self._refreshTargets(updateFreq, exitOnFailure)
File "/public/home/zhaoli/software/anaconda3/envs/denovo_asm/lib/python3.8/site-packages/pypeflow/simple_pwatcher_bridge.py", line 340, in _refreshTargets
time.sleep(sleep_time)
KeyboardInterrupt`
I am confused what went wrong.

Hello, how did you solve it on LAF operating system [INFO](slept for another 1100.0s - another 110 loop iterations) ， thank you very much.

l got it, thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pb-cdunn @lizhao007 @wzxie