Feature request: add batch-size to predict command

[Alteroldis]

Dear developers,

If I understood correctly, then the predict command use one thread per contig during ab initio prediction. And for some cases (possibly many processed large contigs at one moment with not enough large RAM size) I have "out of memory" error. For such cases I can decrease number of threads, but this impact to runtime of gapmm and miniprot. I propose to add an option, that affects the thread number only for the ab initio prediction step. Another option is to balance the contig lengths so that at any given time the sum of the contig lengths is roughly equal. Assuming that this is the case and not that the memory consumption of different gene predictors is very different.

[nextgenusfs]

In train each ab initio software is trained separately ie one at a time. In predict it is indeed run in parallel, each predictor is run on a contig, so do you mean you ran out of memory on predict or train?

[Alteroldis]

Sorry, I was wrong. I meant predict command.

[nextgenusfs]

So indeed large contigs could use a lot of RAM, perhaps there is a way to measure RAM usage in the multiprocessing step and only add more jobs if there is space, but I'm not sure how to determine how much RAM a given contig is going to need. Maybe we can start with a profiler built into the code so there is some measurement of RAM usage for each contig. And while you could split contigs into smaller chunks, this makes it really difficult to accurately predict at the areas you split at due to the ab initio predictors using some degree of DNA sequence context.

[Alteroldis]

In general, I suppose it would be easier to add a separate option that allows changing the number of threads for the prediction step. Especially since the pipeline works quite fast, and a fair amount of time is spent on processing gapmm results in one thread. So I don't see any point in running it on nodes with large amount of RAM and CPU threads. Even for large (1-3 Gbps) genomes.

[nextgenusfs]

Most bioinformatics software is not very efficient... often it's simpler to just request way more resources than you need. In this case, I can really only control the python f2 wrapper and the underlying tools are going to do what they do. Running some tests now on some memory profiling to see if we can get an estimate on memory usage for contig length.

[Alteroldis]

Thank you Dr. Palmer, you are absolutely right about the efficiency, unfortunately. It was just a suggestion. In fact, it is not such a big problem to first run with a large number of threads, and then, if an error occurs during the ab initio prediction, reduce the number of threads. Your program is very good at seamlessly restoring the process with almost no loss of progress.
I won't close this discussion, maybe it will be useful for someone to know that launching same command line with reduced number of threads helps if the error occurred right after the line:
[Jul 15 03:45 AM] Running ab initio gene predictions using 24 cpus

[nextgenusfs]

Okay, can you checkout/install the latest in the main branch and then run predict with --monitor-memory. This will flip on some features to track memory usage. It should generate a file in your output_dir/logs/memory-monitoring.jsonl. If you could run it and send me back that file I should be able to determine memory usage on your system. We technically than can use that info to control the ab initio jobs to hopefully not run out of memory. Maybe try to run it with lower number of cpus so that it actually completes properly (if you run on folder that already as results, you can delete predict_misc/predictions*.gff3 and the consensus* files to then re-use everything else and re-run ab initio prediction).

Because of how I run Augustus and genemark (docker wrappers) the RAM usage is not accurate when I calculated locally, so I'm looking for some more data points.

[Alteroldis]

Good evening, I tried predict --cpus 24 --monitor-memory, memory monitoring file is here. Results is similar, during process thread count was decreased to 12. But at the same time, the execution time of ab initio prediction increased several times, may be due memory tracking. And it seems to me that the same (decreased, 12) number of threads would be used during the tRNAscan if its results did already exist for this run.
Also I tried --memory-limit option, but It doesn't seem to limit memory usage to the specified value. And it seems stacked during ab initio prediction. In general, it doesn’t look like the best or easiest path yet. Perfect is the enemy of good.
P.S. I'll update now and check the fix for issue #23.

[nextgenusfs]

Okay, looks like I over-estimated memory requirements on my local data -- I didn't use a lot of different contig sizes (2-8 MB), so this was helpful thank you. Hopefully the latest will scale better, maybe go ahead an try with more cpus (or whatever you would normally run), lets see if it makes it without crashing ;). If this works, I'll just make it the default and flip the opt-in/out so default uses memory aware subprocess calls to the ab initio tools.

[Alteroldis]

Glad to help. Although who is helping whom is the question).
Run with option --monitor-memory for now?
And different question. Is it correct to remove snap and glimmer ab initio predictors with -w snap:0 -w glimmerhmm:0? They have horrendous specificity on my data set. Because of them (and genemark partly), the number of genes is four times higher than expected.

[nextgenusfs]

Yes that should work, but perhaps we should add a cli parameter to skip abinitio predictors a bit more explicitly.

[nextgenusfs]

You also might want to try Helixer, can use the convenience wrapper I wrote here: https://github.com/nextgenusfs/helixerlite