Segmentation fault even for small file and huge amount of memory

[vicbp1]

Hi Ying
I am trying to run IBDkin in a data set of 85K individuals (2.8M SNPs). The average size of each compressed hapibd output is 50GB. I am running the following line in a machine with 500Gb

for i in $(seq 1 5); do ${IBDkin} --ibdfile ibd.txt --map geneticmap.txt --ind complete_list_individuals_freeze1 --range ranges.txt --nthreads 60 --out kinship_analysis_part${i} --outmask --outcoverage --kinship 0.03125 --part 5 ${i} ; done

Even when I run just for the small chr (~27Gb) I got the Segmentation fault error when ibdkin is reading the hapibd file.

First pass ...
reading /21_split1.ibd.gz
Segmentation fault (core dumped)

Do you have any recommendation?

Another question, does ibdkin have any bias related to the kinship coeffient inferences for recent admixed individuals?

[YingZhou001]

Hi Victor,

Thanks for reaching.
Based on your situation, I will try to add option "--part 10 1" to your command. If the problem is still there, I will check the input files.

Best,
Ying

[vicbp1]

Thank for your answer!
I double checked the inputs and they look ok. For values like --part 10, 20, or 50 the result is the same even when I increase the memory availability (700G). Just for one chr that I had a different error : Error!, maker ranges for chromosome do not cover the position 21138926

Even when my data have that region

Any thoughts?

[YingZhou001]

Hi Victor, I don't think this is a memory problem.
Did you check the range file? it should cover all your markers.
And also check the chromosome identifier, it should be in the same codes for all input files.
If the range file is good, I will cehck 1) ibd file, compare the format with the example to look difference. 2) for each IBD segment, the end position should be larger than the start position. 3) the genetic map, both the physical and genetic positions should be in increasing order.

Best,
Ying

[vicbp1]

Hi Ying,

I found that there was something wrong during the compression of the files and now is working
Thank you so much for your time!

[vicbp1]

Hi Ying,

After checking all my files and running again I have found this error:

Error!, maker ranges for chromosome do not cover the position 4160868

What it means exactly? Do I have to modify my range sfile?

[YingZhou001]

Hi Victor, Yes, please try to modify the range file. The range file is used for allocate memory so that it can save time for ibdkin for reading ibd file. Best, Ying

…

On Mon, Oct 18, 2021 at 12:57 PM Victor Borda ***@***.***> wrote: Hi Ying, After checking all my files and running again I have found this error: *Error!, maker ranges for chromosome do not cover the position 4160868* What it means exactly? Do I have to modify my range sfile? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHO6E2ENQOMKYDTYAHNQEA3UHR34LANCNFSM5F6PIAYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[bguo068]

Hi @YingZhou001,

It has been a while since we met via Zoom in the MCMC workshop this summer :).
Victor and I are from the same group. We looked at this issue together this afternoon. We figured that the issue was caused by the memory overflow from str1 or str2 to chr.

IBDkin/src-v2.8.7.7/parallel.c

Lines 623 to 625 in 61ccecb

	char str1[BUFF1], str2[BUFF1];
	int tmpid;
	int chr, id1, id2;

IBDkin/src-v2.8.7.7/parallel.c

Lines 634 to 636 in 61ccecb

	for(i = 0; i < buffi; i++){
	if(i % N != th_index)continue;
	sscanf(strbuff[i], "%s %*s %s %*s %*s %f %f %f", str1, str2, &p1, &p2, &l);

It would not be an issue if the sample names are short. But we happened to have long sample names in our IBD input files (as long as 58 bytes, longer than buffer size (BUFF1 or 32) .

IBDkin/src-v2.8.7.7/head.h

Lines 16 to 17 in 61ccecb

	#define BUFF 128
	#define BUFF1 32

After we edited #define BUFF1 32 to #define BUFF1 128, #define BUFF 128 to #define BUFF 512 and recompiled the code, it runs without crashing immediately. Hope it runs smoothly to the end. @vicbp1

[YingZhou001]

Thanks Bing, that is a super catch! Welcome to modify my scripts in any ways! Best, Ying

…

On Tue, Oct 19, 2021 at 5:14 PM Bing Guo ***@***.***> wrote: Hi @YingZhou001 <https://github.com/YingZhou001>, It has been a while since we met via Zoom in the MCMC workshop this summer :). Victor and I are from the same group. We looked at this issue together this afternoon. We figured that the issue was caused by the memory overflow from str1 to chr. https://github.com/YingZhou001/IBDkin/blob/61ccecbf3cae8b5dd4b42716aeed099e35a3c3ed/src-v2.8.7.7/parallel.c#L623-L625 https://github.com/YingZhou001/IBDkin/blob/61ccecbf3cae8b5dd4b42716aeed099e35a3c3ed/src-v2.8.7.7/parallel.c#L634-L636 It would not be an issue if the sample names are short. But we happened to have long sample names in our IBD input files (as long as 58 bytes, longer than buffer size (BUFF1 or 32) . https://github.com/YingZhou001/IBDkin/blob/61ccecbf3cae8b5dd4b42716aeed099e35a3c3ed/src-v2.8.7.7/head.h#L17 After we edited #define BUFF1 32 to #define BUFF1 128 and recompiled the code, it runs without crashing immediately. Hope it runs smoothly to the end. @vicbp1 <https://github.com/vicbp1> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHO6E2FPKM4VDCOTJ5S54CLUHYCWDANCNFSM5F6PIAYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.