[Error] std::bad_alloc

[yassineS]

Hiya,

I am trying to run dystruct on a fairly large dataset, a 1200 individuals with 800k SNPs after LD pruning. I'm running this on a large maching 240G or RAM and with OMP_NUM_THREADS=62. After running for 2h or so, I got the following core dump:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

[tyjo]

Thanks for opening this issue. This looks like you a running out of memory somewhere. Does the program output anything besides the error? Can you include your system specifications (OS/compiler)? I wonder if you are hitting a resource limit set by the OS.

One thing to consider - you won't see a benefit in performance after the number of threads increases beyond the number of populations (K). Does decreasing the number of threads solve the issue?

[yassineS]

Thanks for the quick feedback. I tried with OMP_NUM_THREADS=K, but it still failed the same way, and this is the only output I'm getting:

loading genotype matrix...
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Aborted (core dumped)

I feel like I need to be more aggressive with my prunning (use a lower R^2). I will also try to 'thin' the dataset a little. Unless you can think of an alternative approach.

[tyjo]

It looks like it is running out of memory when loading the data. However, if each SNP takes 4 bytes then 800k SNPs in 1200 individuals should only use approximately (4 x 800k x 1200) / (1024^3) = 3.6 GB of memory, so I wonder if something else is going on here.

Do you mind uploading the header of your input file (just the generation times, not genotypes)? I will try to replicate using dummy data. If Dystruct is allocating more memory than it should, I will upload a fix.

Depending on the number of time points in your data, 800K SNPs might be a bit large to run in a reasonable amount of time.

[yassineS]

Hi Tyler, Thank you for your help again. I have attached here the header of my file.  I was wondering what would a more aggressive LD pruning do to the results? Also, does an unequal number of samples in each time-bin affects the results?

On Wed, 16 May 2018 at 3:49 am Tyler < Tyler ( Tyler ***@***.***> ) > wrote: It looks like it is running out of memory when loading the data. However, if each SNP takes 4 bytes then 800k SNPs in 1200 individuals should only use approximately 4 800k 1200 / (1024^3) = 3.6 GB of memory, so I wonder if something else is going on here. Do you mind uploading the header of your input file (just the generation times, not genotypes)? I will try to replicate using dummy data. If Dystruct is allocating more memory than it should, I will upload a fix. Depending on the number of time points in your data, 800K SNPs might be a bit large to run in a reasonable amount of time. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub ( #6 (comment) ) , or mute the thread ( https://github.com/notifications/unsubscribe-auth/ABr4IYZO2fIGtFw6quhnOApbG1Ff1fgvks5tyxw5gaJpZM4T9tCQ ).

GEN.FROM.MAX.BIN 0 450 500 600 600 600 600 650 650 700 700 850 1100 1150 1200 1200 1250 1250 1250 1250 1250 1300 1300 1300 1300 1300 1300 1300 1300 1300 1350 1350 1350 1350 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 1400 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1450 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1500 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1550 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1600 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1650 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1700 1750 1750 1750 1750 1750 1750 1750 1750 1800 1800 1800 1800 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825 1825

[tyjo]

Both our model and ADMIXTURE assume unlinked sites, so you want to do some amount of LD pruning. I think similar LD thresholds for thinning SNPs for ADMIXTURE should work with our model too. Although, there is a trade-off because removing sites with high LD disproportionately affects individuals with missing data. Section 6 in the supplementary information of this paper briefly discussions LD pruning for ADMIXTURE and ancient DNA.

I generated dummy data using your headers, but I couldn't replicate the memory problem. On my machine 800,000 SNPs took up 1768MB while loading the data, which increased to 3051MB during the inference algorithm. Just in case there is a problem with the input file, I uploaded code that runs a few more checks in the input file.

[yassineS]

Thank you Tyler for your help. I will investigate my setup and input file.