Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to interpret this meryl count warning message #1565

Closed
SHuang-Broad opened this issue Nov 19, 2019 · 6 comments
Closed

How to interpret this meryl count warning message #1565

SHuang-Broad opened this issue Nov 19, 2019 · 6 comments

Comments

@SHuang-Broad
Copy link
Contributor

Hi,

My meryl-count out files have the following warning messages. I don't know how to interpret that. Should I allocate more memory for meryl?

Thanks!

WARNING:
WARNING: Cannot fit into 27 GB memory limit.
WARNING: Will split into up to 2 (possibly 3) batches, and merge them at the end.
WARNING:

Full out file


Found perl:
   /usr/bin/perl
   This is perl 5, version 26, subversion 2 (v5.26.2) built for x86_64-linux-gnu-thread-multi

Found java:
   /usr/lib/jvm/java-8-oracle/bin/java
   java version "1.8.0_201"

Found canu:
   /home/shuang/bin/canu/Linux-amd64/bin/canu
   Canu branch v1.9 +383 changes (r9593 5c68cb93ef0cd11fd1802f4c1a9f90d2f87a2ff6)

Running job 1 based on command line options.
Enabling 22 threads.

Counting 7626 (estimated)  million canonical 21-mers from 80 input files:
    sequence-file: ./reads-Father/reads-Father-001.fasta.gz
    sequence-file: ./reads-Father/reads-Father-002.fasta.gz
    sequence-file: ./reads-Father/reads-Father-003.fasta.gz
    sequence-file: ./reads-Father/reads-Father-004.fasta.gz
    sequence-file: ./reads-Father/reads-Father-005.fasta.gz
    sequence-file: ./reads-Father/reads-Father-006.fasta.gz
    sequence-file: ./reads-Father/reads-Father-007.fasta.gz
    sequence-file: ./reads-Father/reads-Father-008.fasta.gz
    sequence-file: ./reads-Father/reads-Father-009.fasta.gz
    sequence-file: ./reads-Father/reads-Father-010.fasta.gz
    sequence-file: ./reads-Father/reads-Father-011.fasta.gz
    sequence-file: ./reads-Father/reads-Father-012.fasta.gz
    sequence-file: ./reads-Father/reads-Father-013.fasta.gz
    sequence-file: ./reads-Father/reads-Father-014.fasta.gz
    sequence-file: ./reads-Father/reads-Father-015.fasta.gz
    sequence-file: ./reads-Father/reads-Father-016.fasta.gz
    sequence-file: ./reads-Father/reads-Father-017.fasta.gz
    sequence-file: ./reads-Father/reads-Father-018.fasta.gz
    sequence-file: ./reads-Father/reads-Father-019.fasta.gz
    sequence-file: ./reads-Father/reads-Father-020.fasta.gz
    sequence-file: ./reads-Father/reads-Father-021.fasta.gz
    sequence-file: ./reads-Father/reads-Father-022.fasta.gz
    sequence-file: ./reads-Father/reads-Father-023.fasta.gz
    sequence-file: ./reads-Father/reads-Father-024.fasta.gz
    sequence-file: ./reads-Father/reads-Father-025.fasta.gz
    sequence-file: ./reads-Father/reads-Father-026.fasta.gz
    sequence-file: ./reads-Father/reads-Father-027.fasta.gz
    sequence-file: ./reads-Father/reads-Father-028.fasta.gz
    sequence-file: ./reads-Father/reads-Father-029.fasta.gz
    sequence-file: ./reads-Father/reads-Father-030.fasta.gz
    sequence-file: ./reads-Father/reads-Father-031.fasta.gz
    sequence-file: ./reads-Father/reads-Father-032.fasta.gz
    sequence-file: ./reads-Father/reads-Father-033.fasta.gz
    sequence-file: ./reads-Father/reads-Father-034.fasta.gz
    sequence-file: ./reads-Father/reads-Father-035.fasta.gz
    sequence-file: ./reads-Father/reads-Father-036.fasta.gz
    sequence-file: ./reads-Father/reads-Father-037.fasta.gz
    sequence-file: ./reads-Father/reads-Father-038.fasta.gz
    sequence-file: ./reads-Father/reads-Father-039.fasta.gz
    sequence-file: ./reads-Father/reads-Father-040.fasta.gz
    sequence-file: ./reads-Father/reads-Father-041.fasta.gz
    sequence-file: ./reads-Father/reads-Father-042.fasta.gz
    sequence-file: ./reads-Father/reads-Father-043.fasta.gz
    sequence-file: ./reads-Father/reads-Father-044.fasta.gz
    sequence-file: ./reads-Father/reads-Father-045.fasta.gz
    sequence-file: ./reads-Father/reads-Father-046.fasta.gz
    sequence-file: ./reads-Father/reads-Father-047.fasta.gz
    sequence-file: ./reads-Father/reads-Father-048.fasta.gz
    sequence-file: ./reads-Father/reads-Father-049.fasta.gz
    sequence-file: ./reads-Father/reads-Father-050.fasta.gz
    sequence-file: ./reads-Father/reads-Father-051.fasta.gz
    sequence-file: ./reads-Father/reads-Father-052.fasta.gz
    sequence-file: ./reads-Father/reads-Father-053.fasta.gz
    sequence-file: ./reads-Father/reads-Father-054.fasta.gz
    sequence-file: ./reads-Father/reads-Father-055.fasta.gz
    sequence-file: ./reads-Father/reads-Father-056.fasta.gz
    sequence-file: ./reads-Father/reads-Father-057.fasta.gz
    sequence-file: ./reads-Father/reads-Father-058.fasta.gz
    sequence-file: ./reads-Father/reads-Father-059.fasta.gz
    sequence-file: ./reads-Father/reads-Father-060.fasta.gz
    sequence-file: ./reads-Father/reads-Father-061.fasta.gz
    sequence-file: ./reads-Father/reads-Father-062.fasta.gz
    sequence-file: ./reads-Father/reads-Father-063.fasta.gz
    sequence-file: ./reads-Father/reads-Father-064.fasta.gz
    sequence-file: ./reads-Father/reads-Father-065.fasta.gz
    sequence-file: ./reads-Father/reads-Father-066.fasta.gz
    sequence-file: ./reads-Father/reads-Father-067.fasta.gz
    sequence-file: ./reads-Father/reads-Father-068.fasta.gz
    sequence-file: ./reads-Father/reads-Father-069.fasta.gz
    sequence-file: ./reads-Father/reads-Father-070.fasta.gz
    sequence-file: ./reads-Father/reads-Father-071.fasta.gz
    sequence-file: ./reads-Father/reads-Father-072.fasta.gz
    sequence-file: ./reads-Father/reads-Father-073.fasta.gz
    sequence-file: ./reads-Father/reads-Father-074.fasta.gz
    sequence-file: ./reads-Father/reads-Father-075.fasta.gz
    sequence-file: ./reads-Father/reads-Father-076.fasta.gz
    sequence-file: ./reads-Father/reads-Father-077.fasta.gz
    sequence-file: ./reads-Father/reads-Father-078.fasta.gz
    sequence-file: ./reads-Father/reads-Father-079.fasta.gz
    sequence-file: ./reads-Father/reads-Father-080.fasta.gz


SIMPLE MODE
-----------

  Disabled for mers larger than 20.


COMPLEX MODE
------------

prefix     # of   struct   kmers/    segs/     data    total
  bits   prefix   memory   prefix   prefix   memory   memory
------  -------  -------  -------  -------  -------  -------
     1     2  P  4886 kB  3813 MM   305 kS    38 GB    38 GB
     2     4  P  4767 kB  1906 MM   148 kS    37 GB    37 GB
     3     8  P  4648 kB   953 MM    72 kS    36 GB    36 GB
     4    16  P  4530 kB   476 MM    35 kS    35 GB    35 GB
     5    32  P  4413 kB   238 MM    17 kS    34 GB    34 GB
     6    64  P  4297 kB   119 MM  8581  S    33 GB    33 GB
     7   128  P  4185 kB    59 MM  4171  S    32 GB    32 GB
     8   256  P  4080 kB    29 MM  2026  S    31 GB    31 GB
     9   512  P  3992 kB    14 MM   984  S    30 GB    30 GB
    10  1024  P  3928 kB  7626 kM   477  S    29 GB    29 GB
    11  2048  P  3920 kB  3813 kM   231  S    28 GB    28 GB
    12  4096  P  4032 kB  1906 kM   112  S    28 GB    28 GB
    13  8192  P  4352 kB   953 kM    54  S    27 GB    27 GB
    14    16 kP  5248 kB   476 kM    27  S    27 GB    27 GB
    15    32 kP  6912 kB   238 kM    13  S    26 GB    26 GB
    16    64 kP    10 MB   119 kM     7  S    28 GB    28 GB
    17   128 kP    17 MB    59 kM     3  S    24 GB    24 GB  Best Value!
    18   256 kP    32 MB    29 kM     2  S    32 GB    32 GB
    19   512 kP    60 MB    14 kM     1  S    32 GB    32 GB
    20  1024 kP   120 MB  7627  M     1  S    64 GB    64 GB
    21  2048 kP   240 MB  3814  M     1  S   128 GB   128 GB
    22  4096 kP   480 MB  1907  M     1  S   256 GB   256 GB
    23  8192 kP   960 MB   954  M     1  S   512 GB   512 GB


FINAL CONFIGURATION
-------------------

WARNING:
WARNING: Cannot fit into 27 GB memory limit.
WARNING: Will split into up to 2 (possibly 3) batches, and merge them at the end.
WARNING:

Configured complex mode for 24.017 GB memory per batch, and up to 2 batches.

kmerCountFileWriter()-- Creating './reads-Father-001.meryl.WORKING' for 21-mers, with prefixSize 17 suffixSize 25 numFiles 64
Loading kmers from './reads-Father/reads-Father-001.fasta.gz' into buckets.
Used 3.289 GB out of 27.640 GB to store          229 kmers.
Used 3.414 GB out of 27.640 GB to store        47582 kmers.
Used 3.539 GB out of 27.640 GB to store       174682 kmers.
Used 3.664 GB out of 27.640 GB to store       637672 kmers.
Used 3.789 GB out of 27.640 GB to store      5417720 kmers.
Used 3.914 GB out of 27.640 GB to store     79236296 kmers.
Loading kmers from './reads-Father/reads-Father-002.fasta.gz' into buckets.
Used 4.039 GB out of 27.640 GB to store    128301483 kmers.
Used 4.164 GB out of 27.640 GB to store    175254779 kmers.
Loading kmers from './reads-Father/reads-Father-003.fasta.gz' into buckets.
Used 4.289 GB out of 27.640 GB to store    221354494 kmers.
Used 4.414 GB out of 27.640 GB to store    267357638 kmers.
Loading kmers from './reads-Father/reads-Father-004.fasta.gz' into buckets.
Used 4.539 GB out of 27.640 GB to store    313038395 kmers.
Used 4.664 GB out of 27.640 GB to store    358210082 kmers.
Loading kmers from './reads-Father/reads-Father-005.fasta.gz' into buckets.
Used 4.789 GB out of 27.640 GB to store    402672281 kmers.
Used 4.914 GB out of 27.640 GB to store    447143722 kmers.
Loading kmers from './reads-Father/reads-Father-006.fasta.gz' into buckets.
Used 5.039 GB out of 27.640 GB to store    491665050 kmers.
Used 5.164 GB out of 27.640 GB to store    535782047 kmers.
Loading kmers from './reads-Father/reads-Father-007.fasta.gz' into buckets.
Used 5.289 GB out of 27.640 GB to store    579384893 kmers.
Used 5.414 GB out of 27.640 GB to store    623266280 kmers.
Loading kmers from './reads-Father/reads-Father-008.fasta.gz' into buckets.
Used 5.539 GB out of 27.640 GB to store    667141191 kmers.
Used 5.664 GB out of 27.640 GB to store    710709680 kmers.
Loading kmers from './reads-Father/reads-Father-009.fasta.gz' into buckets.
Used 5.789 GB out of 27.640 GB to store    754286063 kmers.
Used 5.914 GB out of 27.640 GB to store    797697388 kmers.
Loading kmers from './reads-Father/reads-Father-010.fasta.gz' into buckets.
Used 6.039 GB out of 27.640 GB to store    840890852 kmers.
Used 6.164 GB out of 27.640 GB to store    884301537 kmers.
Loading kmers from './reads-Father/reads-Father-011.fasta.gz' into buckets.
Used 6.289 GB out of 27.640 GB to store    927747893 kmers.
Used 6.414 GB out of 27.640 GB to store    971032118 kmers.
Loading kmers from './reads-Father/reads-Father-012.fasta.gz' into buckets.
Used 6.539 GB out of 27.640 GB to store   1014144240 kmers.
Used 6.664 GB out of 27.640 GB to store   1057274158 kmers.
Used 6.789 GB out of 27.640 GB to store   1100672005 kmers.
Loading kmers from './reads-Father/reads-Father-013.fasta.gz' into buckets.
Used 6.914 GB out of 27.640 GB to store   1143810155 kmers.
Used 7.039 GB out of 27.640 GB to store   1187067543 kmers.
Loading kmers from './reads-Father/reads-Father-014.fasta.gz' into buckets.
Used 7.164 GB out of 27.640 GB to store   1230081267 kmers.
Used 7.289 GB out of 27.640 GB to store   1273162410 kmers.
Loading kmers from './reads-Father/reads-Father-015.fasta.gz' into buckets.
Used 7.414 GB out of 27.640 GB to store   1316258233 kmers.
Used 7.539 GB out of 27.640 GB to store   1359370507 kmers.
Loading kmers from './reads-Father/reads-Father-016.fasta.gz' into buckets.
Used 7.664 GB out of 27.640 GB to store   1402389909 kmers.
Used 7.789 GB out of 27.640 GB to store   1445378567 kmers.
Loading kmers from './reads-Father/reads-Father-017.fasta.gz' into buckets.
Used 7.914 GB out of 27.640 GB to store   1488462625 kmers.
Used 8.039 GB out of 27.640 GB to store   1531711018 kmers.
Loading kmers from './reads-Father/reads-Father-018.fasta.gz' into buckets.
Used 8.164 GB out of 27.640 GB to store   1574603127 kmers.
Used 8.289 GB out of 27.640 GB to store   1617513971 kmers.
Loading kmers from './reads-Father/reads-Father-019.fasta.gz' into buckets.
Used 8.414 GB out of 27.640 GB to store   1660771811 kmers.
Used 8.539 GB out of 27.640 GB to store   1703626048 kmers.
Used 8.664 GB out of 27.640 GB to store   1746794577 kmers.
Loading kmers from './reads-Father/reads-Father-020.fasta.gz' into buckets.
Used 8.789 GB out of 27.640 GB to store   1789592900 kmers.
Used 8.914 GB out of 27.640 GB to store   1832536847 kmers.
Loading kmers from './reads-Father/reads-Father-021.fasta.gz' into buckets.
Used 9.039 GB out of 27.640 GB to store   1875771008 kmers.
Used 9.164 GB out of 27.640 GB to store   1918816325 kmers.
Loading kmers from './reads-Father/reads-Father-022.fasta.gz' into buckets.
Used 9.289 GB out of 27.640 GB to store   1961544649 kmers.
Used 9.414 GB out of 27.640 GB to store   2004762054 kmers.
Loading kmers from './reads-Father/reads-Father-023.fasta.gz' into buckets.
Used 9.539 GB out of 27.640 GB to store   2047667008 kmers.
Used 9.664 GB out of 27.640 GB to store   2090543722 kmers.
Loading kmers from './reads-Father/reads-Father-024.fasta.gz' into buckets.
Used 9.789 GB out of 27.640 GB to store   2133497213 kmers.
Used 9.914 GB out of 27.640 GB to store   2176418002 kmers.
Loading kmers from './reads-Father/reads-Father-025.fasta.gz' into buckets.
Used 10.039 GB out of 27.640 GB to store   2219685820 kmers.
Used 10.164 GB out of 27.640 GB to store   2262607376 kmers.
Loading kmers from './reads-Father/reads-Father-026.fasta.gz' into buckets.
Used 10.289 GB out of 27.640 GB to store   2305739085 kmers.
Used 10.414 GB out of 27.640 GB to store   2348602900 kmers.
Loading kmers from './reads-Father/reads-Father-027.fasta.gz' into buckets.
Used 10.539 GB out of 27.640 GB to store   2391881962 kmers.
Used 10.664 GB out of 27.640 GB to store   2434870172 kmers.
Used 10.789 GB out of 27.640 GB to store   2477781291 kmers.
Loading kmers from './reads-Father/reads-Father-028.fasta.gz' into buckets.
Used 10.914 GB out of 27.640 GB to store   2520801385 kmers.
Used 11.039 GB out of 27.640 GB to store   2563792608 kmers.
Loading kmers from './reads-Father/reads-Father-029.fasta.gz' into buckets.
Used 11.164 GB out of 27.640 GB to store   2606706489 kmers.
Used 11.289 GB out of 27.640 GB to store   2649987340 kmers.
Loading kmers from './reads-Father/reads-Father-030.fasta.gz' into buckets.
Used 11.414 GB out of 27.640 GB to store   2692884056 kmers.
Used 11.539 GB out of 27.640 GB to store   2735747738 kmers.
Loading kmers from './reads-Father/reads-Father-031.fasta.gz' into buckets.
Used 11.664 GB out of 27.640 GB to store   2778785854 kmers.
Used 11.789 GB out of 27.640 GB to store   2821872926 kmers.
Loading kmers from './reads-Father/reads-Father-032.fasta.gz' into buckets.
Used 11.914 GB out of 27.640 GB to store   2864839930 kmers.
Used 12.039 GB out of 27.640 GB to store   2907607191 kmers.
Loading kmers from './reads-Father/reads-Father-033.fasta.gz' into buckets.
Used 12.164 GB out of 27.640 GB to store   2950600021 kmers.
Used 12.289 GB out of 27.640 GB to store   2993742687 kmers.
Loading kmers from './reads-Father/reads-Father-034.fasta.gz' into buckets.
Used 12.414 GB out of 27.640 GB to store   3036726209 kmers.
Used 12.539 GB out of 27.640 GB to store   3079625637 kmers.
Used 12.664 GB out of 27.640 GB to store   3122774989 kmers.
Loading kmers from './reads-Father/reads-Father-035.fasta.gz' into buckets.
Used 12.789 GB out of 27.640 GB to store   3165528321 kmers.
Used 12.914 GB out of 27.640 GB to store   3208358754 kmers.
Loading kmers from './reads-Father/reads-Father-036.fasta.gz' into buckets.
Used 13.039 GB out of 27.640 GB to store   3251508715 kmers.
Used 13.164 GB out of 27.640 GB to store   3294794114 kmers.
Loading kmers from './reads-Father/reads-Father-037.fasta.gz' into buckets.
Used 13.289 GB out of 27.640 GB to store   3337739942 kmers.
Used 13.414 GB out of 27.640 GB to store   3380901945 kmers.
Loading kmers from './reads-Father/reads-Father-038.fasta.gz' into buckets.
Used 13.539 GB out of 27.640 GB to store   3423830327 kmers.
Used 13.664 GB out of 27.640 GB to store   3466630944 kmers.
Loading kmers from './reads-Father/reads-Father-039.fasta.gz' into buckets.
Used 13.789 GB out of 27.640 GB to store   3509770732 kmers.
Used 13.914 GB out of 27.640 GB to store   3552637872 kmers.
Loading kmers from './reads-Father/reads-Father-040.fasta.gz' into buckets.
Used 14.039 GB out of 27.640 GB to store   3595517085 kmers.
Used 14.164 GB out of 27.640 GB to store   3638451564 kmers.
Loading kmers from './reads-Father/reads-Father-041.fasta.gz' into buckets.
Used 14.289 GB out of 27.640 GB to store   3681389429 kmers.
Used 14.414 GB out of 27.640 GB to store   3724666659 kmers.
Used 14.539 GB out of 27.640 GB to store   3767558446 kmers.
Loading kmers from './reads-Father/reads-Father-042.fasta.gz' into buckets.
Used 14.664 GB out of 27.640 GB to store   3810563341 kmers.
Used 14.789 GB out of 27.640 GB to store   3853514560 kmers.
Loading kmers from './reads-Father/reads-Father-043.fasta.gz' into buckets.
Used 14.914 GB out of 27.640 GB to store   3896284309 kmers.
Used 15.039 GB out of 27.640 GB to store   3939285848 kmers.
Loading kmers from './reads-Father/reads-Father-044.fasta.gz' into buckets.
Used 15.164 GB out of 27.640 GB to store   3982519344 kmers.
Used 15.289 GB out of 27.640 GB to store   4025354907 kmers.
Loading kmers from './reads-Father/reads-Father-045.fasta.gz' into buckets.
Used 15.414 GB out of 27.640 GB to store   4068104045 kmers.
Used 15.539 GB out of 27.640 GB to store   4111107604 kmers.
Loading kmers from './reads-Father/reads-Father-046.fasta.gz' into buckets.
Used 15.664 GB out of 27.640 GB to store   4154107775 kmers.
Used 15.789 GB out of 27.640 GB to store   4196939688 kmers.
Loading kmers from './reads-Father/reads-Father-047.fasta.gz' into buckets.
Used 15.914 GB out of 27.640 GB to store   4239871177 kmers.
Used 16.039 GB out of 27.640 GB to store   4282901625 kmers.
Loading kmers from './reads-Father/reads-Father-048.fasta.gz' into buckets.
Used 16.164 GB out of 27.640 GB to store   4325966450 kmers.
Used 16.289 GB out of 27.640 GB to store   4368849347 kmers.
Used 16.414 GB out of 27.640 GB to store   4411900956 kmers.
Loading kmers from './reads-Father/reads-Father-049.fasta.gz' into buckets.
Used 16.539 GB out of 27.640 GB to store   4454726912 kmers.
Used 16.664 GB out of 27.640 GB to store   4497979441 kmers.
Loading kmers from './reads-Father/reads-Father-050.fasta.gz' into buckets.
Used 16.789 GB out of 27.640 GB to store   4540642146 kmers.
Used 16.914 GB out of 27.640 GB to store   4583537603 kmers.
Loading kmers from './reads-Father/reads-Father-051.fasta.gz' into buckets.
Used 17.039 GB out of 27.640 GB to store   4626623562 kmers.
Used 17.164 GB out of 27.640 GB to store   4669582423 kmers.
Loading kmers from './reads-Father/reads-Father-052.fasta.gz' into buckets.
Used 17.289 GB out of 27.640 GB to store   4712432168 kmers.
Used 17.414 GB out of 27.640 GB to store   4755631021 kmers.
Loading kmers from './reads-Father/reads-Father-053.fasta.gz' into buckets.
Used 17.539 GB out of 27.640 GB to store   4798381110 kmers.
Used 17.664 GB out of 27.640 GB to store   4841482674 kmers.
Loading kmers from './reads-Father/reads-Father-054.fasta.gz' into buckets.
Used 17.789 GB out of 27.640 GB to store   4884389943 kmers.
Used 17.914 GB out of 27.640 GB to store   4927357883 kmers.
Loading kmers from './reads-Father/reads-Father-055.fasta.gz' into buckets.
Used 18.039 GB out of 27.640 GB to store   4970258476 kmers.
Used 18.164 GB out of 27.640 GB to store   5013101807 kmers.
Used 18.289 GB out of 27.640 GB to store   5056229664 kmers.
Loading kmers from './reads-Father/reads-Father-056.fasta.gz' into buckets.
Used 18.414 GB out of 27.640 GB to store   5099181119 kmers.
Used 18.539 GB out of 27.640 GB to store   5142339165 kmers.
Loading kmers from './reads-Father/reads-Father-057.fasta.gz' into buckets.
Used 18.664 GB out of 27.640 GB to store   5185047406 kmers.
Used 18.789 GB out of 27.640 GB to store   5228150749 kmers.
Loading kmers from './reads-Father/reads-Father-058.fasta.gz' into buckets.
Used 18.914 GB out of 27.640 GB to store   5271035223 kmers.
Used 19.039 GB out of 27.640 GB to store   5314095673 kmers.
Loading kmers from './reads-Father/reads-Father-059.fasta.gz' into buckets.
Used 19.164 GB out of 27.640 GB to store   5356852633 kmers.
Used 19.289 GB out of 27.640 GB to store   5399819393 kmers.
Loading kmers from './reads-Father/reads-Father-060.fasta.gz' into buckets.
Used 19.414 GB out of 27.640 GB to store   5442778289 kmers.
Used 19.539 GB out of 27.640 GB to store   5485846554 kmers.
Loading kmers from './reads-Father/reads-Father-061.fasta.gz' into buckets.
Used 19.664 GB out of 27.640 GB to store   5528608757 kmers.
Used 19.789 GB out of 27.640 GB to store   5571483353 kmers.
Loading kmers from './reads-Father/reads-Father-062.fasta.gz' into buckets.
Used 19.914 GB out of 27.640 GB to store   5614436453 kmers.
Used 20.039 GB out of 27.640 GB to store   5657398613 kmers.
Used 20.164 GB out of 27.640 GB to store   5700499463 kmers.
Loading kmers from './reads-Father/reads-Father-063.fasta.gz' into buckets.
Used 20.289 GB out of 27.640 GB to store   5743492673 kmers.
Used 20.414 GB out of 27.640 GB to store   5786489979 kmers.
Loading kmers from './reads-Father/reads-Father-064.fasta.gz' into buckets.
Used 20.539 GB out of 27.640 GB to store   5829418789 kmers.
Used 20.664 GB out of 27.640 GB to store   5872130165 kmers.
Loading kmers from './reads-Father/reads-Father-065.fasta.gz' into buckets.
Used 20.789 GB out of 27.640 GB to store   5915120058 kmers.
Used 20.915 GB out of 27.640 GB to store   5958153506 kmers.
Loading kmers from './reads-Father/reads-Father-066.fasta.gz' into buckets.
Used 21.040 GB out of 27.640 GB to store   6001209090 kmers.
Used 21.165 GB out of 27.640 GB to store   6044068584 kmers.
Loading kmers from './reads-Father/reads-Father-067.fasta.gz' into buckets.
Used 21.290 GB out of 27.640 GB to store   6086854440 kmers.
Used 21.415 GB out of 27.640 GB to store   6129910043 kmers.
Loading kmers from './reads-Father/reads-Father-068.fasta.gz' into buckets.
Used 21.540 GB out of 27.640 GB to store   6172810859 kmers.
Used 21.665 GB out of 27.640 GB to store   6215762083 kmers.
Loading kmers from './reads-Father/reads-Father-069.fasta.gz' into buckets.
Used 21.790 GB out of 27.640 GB to store   6258781952 kmers.
Used 21.915 GB out of 27.640 GB to store   6301799542 kmers.
Used 22.040 GB out of 27.640 GB to store   6344427052 kmers.
Loading kmers from './reads-Father/reads-Father-070.fasta.gz' into buckets.
Used 22.165 GB out of 27.640 GB to store   6387571832 kmers.
Used 22.290 GB out of 27.640 GB to store   6430499262 kmers.
Loading kmers from './reads-Father/reads-Father-071.fasta.gz' into buckets.
Used 22.415 GB out of 27.640 GB to store   6473408982 kmers.
Used 22.540 GB out of 27.640 GB to store   6516484072 kmers.
Loading kmers from './reads-Father/reads-Father-072.fasta.gz' into buckets.
Used 22.665 GB out of 27.640 GB to store   6559652772 kmers.
Used 22.790 GB out of 27.640 GB to store   6602376307 kmers.
Loading kmers from './reads-Father/reads-Father-073.fasta.gz' into buckets.
Used 22.915 GB out of 27.640 GB to store   6645168858 kmers.
Used 23.040 GB out of 27.640 GB to store   6688218265 kmers.
Loading kmers from './reads-Father/reads-Father-074.fasta.gz' into buckets.
Used 23.165 GB out of 27.640 GB to store   6731256534 kmers.
Used 23.290 GB out of 27.640 GB to store   6774030473 kmers.
Loading kmers from './reads-Father/reads-Father-075.fasta.gz' into buckets.
Used 23.415 GB out of 27.640 GB to store   6817042151 kmers.
Used 23.540 GB out of 27.640 GB to store   6859975913 kmers.
Loading kmers from './reads-Father/reads-Father-076.fasta.gz' into buckets.
Used 23.665 GB out of 27.640 GB to store   6902946793 kmers.
Used 23.790 GB out of 27.640 GB to store   6945849598 kmers.
Used 23.915 GB out of 27.640 GB to store   6988786412 kmers.
Loading kmers from './reads-Father/reads-Father-077.fasta.gz' into buckets.
Used 24.040 GB out of 27.640 GB to store   7031883331 kmers.
Used 24.165 GB out of 27.640 GB to store   7074665873 kmers.
Loading kmers from './reads-Father/reads-Father-078.fasta.gz' into buckets.
Used 24.290 GB out of 27.640 GB to store   7117765501 kmers.
Used 24.415 GB out of 27.640 GB to store   7160373509 kmers.
Loading kmers from './reads-Father/reads-Father-079.fasta.gz' into buckets.
Used 24.540 GB out of 27.640 GB to store   7203539979 kmers.
Used 24.665 GB out of 27.640 GB to store   7246419719 kmers.
Loading kmers from './reads-Father/reads-Father-080.fasta.gz' into buckets.
Used 24.790 GB out of 27.640 GB to store   7289195731 kmers.
Used 24.915 GB out of 27.640 GB to store   7332065494 kmers.

Writing results to './reads-Father-001.meryl.WORKING', using 22 threads.
finishIteration()--

Finished counting.
Bye.
@brianwalenz
Copy link
Member

You can leave it as is.

There is only a small penalty for using too little memory; meryl will compute counts in batches, then merge the batches at the end. Here, it's warning that there might be 2 or 3 batches -- perfectly reasonable; the merge is quite fast -- but in fact, it fit everything into one batch.

For this specific step in canu, there is a benefit to using even smaller memory sizes. Canu will then be able to run multiple meryl jobs at once, and merge the results at the end. So, instead of running one big 30gb job, you can run 6 jobs at 6gb each (so 1/6th the wall time), then do a quick merge at the end.

@SHuang-Broad
Copy link
Contributor Author

Hmm... Interesting.
This is actually an out file from one of the 52 batches.

canu repartitioned my parental short reads into approximately 4000 (4k) files.
Based on what you are suggesting, I can specify merylMemory to be an even smaller number, so that canu will configure the count process into even smaller batches (less files per batch, higher batch count).
Is that interpretation correct?

@SHuang-Broad
Copy link
Contributor Author

And one follow up question:

my understanding is that the meryl steps in trio binning the child reads are configured to be using the same amount of memory, i.e. count, merge, and subtract.
If I were to specify merylMemory to a lower number, say 6G, how would that affect the merge and subtract step?
Are they memory intensive (i.e. having to load all the intermediate files to memory)?

@brianwalenz
Copy link
Member

That's correct.

I'm sure there's some sweet spot where the benefit of multiple parallel jobs balances the overhead of merging those results. I have no idea where it is. Running one gigantic batch will be very slow, but fast to merge. Running 4000 batches will be pretty quick (assuming 4000 cpus) but need enormous I/O bandwidth.

Also, recall that this step doesn't listen to merylMemory. It's picking a partitioning based on the number of files - https://github.com/marbl/canu/blob/master/src/pipelines/canu/HaplotypeReads.pm#L368 (and about 100 lines before there too).

@SHuang-Broad
Copy link
Contributor Author

Thanks for the answers Brian!

I remember that now.

I think I do have a bit of luxury for better IO for our use case (scattering the count operations across one hundred google VM seems to be acceptable to us).

@SHuang-Broad SHuang-Broad mentioned this issue Nov 19, 2019
Open
@SHuang-Broad
Copy link
Contributor Author

Closing as Brian answered the question perfectly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@brianwalenz @SHuang-Broad