compile_pfb.pl running time issue

[zuyan413]

When I am using 960 split files, it's getting extremely slow (taking around 30 hrs). but when I am reducing the number of split files to 400, it is running fast.
command:
time mpirun -np 72 --oversubscribe perl compile_pfb.pl --listfile path_to_split_file.txt -output output_batch3_2.pfb

[kaichop]

This could be due to memory issues when virtual memory is needed to complete the task with many files, or could be just a file system issue to support simultaneous reading of 960 files. In any case, 400 files are already enough to compile a PFB.

…

On Thu, Jun 6, 2024 at 2:55 AM zuyan413 ***@***.***> wrote: When I am using 960 split files, it's getting extremely slow (taking around 30 hrs). but when I am reducing the number of split files to 400, it is running fast. command: time mpirun -np 72 --oversubscribe perl compile_pfb.pl --listfile path_to_split_file.txt -output output_batch3_2.pfb — Reply to this email directly, view it on GitHub <#120>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNG3ODW6DBSSDYTBCJK7PDZGABWRAVCNFSM6AAAAABI4C4UPWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGMZTONBYGE3DKNA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[zuyan413]

Thankyou. So, if I have 10,000 samples and I run it in batches of 960, then do I have to generate pfb file for each batch separately or one pfb file is okay for all 10k samples? all 10k samples are from same population

[kaichop]

one PFB is okay for all 10k samples.

…

On Tue, Jun 11, 2024 at 2:54 AM zuyan413 ***@***.***> wrote: Thankyou. So, if I have 10,000 samples and I run it in batches of 960, then do I have to generate pfb file for each batch separately or one pfb file is okay for all 10k samples? all 10k samples are from same population — Reply to this email directly, view it on GitHub <#120 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABNG3OFO7QLIHMJGMZCLZPLZG2NLJAVCNFSM6AAAAABI4C4UPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJZHEZTIMBTHE> . You are receiving this because you commented.Message ID: ***@***.***>