Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Segmentation fault?? Can not run rvtest!! #23

Closed
zishang30 opened this issue Apr 24, 2017 · 13 comments
Closed

Segmentation fault?? Can not run rvtest!! #23

zishang30 opened this issue Apr 24, 2017 · 13 comments

Comments

@zishang30
Copy link

Hi, Xiaowei:

We have prepared tabixed vcf.gz file and PED file and kinship matrix and prepare to run the rvtest. But when we run the rvtest it response the mistake about Segmentation fault as follows:
[INFO] Program version: 20170210
[INFO] Analysis started at: Mon Apr 24 10:14:36 2017
[INFO] Loaded [ 3788 ] samples from VCF files
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 602 [ 922000077 922000077 0 0 NA NA ] ...
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 1243 [ 922003426 922003426 0 0 NA NA ] ...
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 1456 [ 922003311 922003311 0 0 NA NA ] ...
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 1624 [ 922001212 922001212 0 0 NA NA ] ...
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 1625 [ 922000100 922000100 0 0 NA NA ] ...
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 1996 [ 922000199 922000199 0 0 NA NA ] ...
[INFO] Loaded [ 3782 ] sample pheontypes
[WARN] Total [ 6 ] samples are dropped from VCF file due to missing phenotype.
[WARN] Drop sample [ 922000077 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922003426 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922003311 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922001212 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922000100 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922000199 ] from VCF file due to missing phenotype
[INFO] Loaded 1850 male, 1932 female and 0 sex-unknonw samples from phenotypes_all_f4.ped
[INFO] Analysis begins with [ 3782 ] samples...
[INFO] load 1 parameters.
[INFO] Meta analysis uses window size 500,000 to produce covariance statistics under additive model
[INFO] Family-based model specified. Loading kinship file...
[INFO] DONE: Loaded kinship file [ S4F4all_kinship_matrix.kinship ] successfully in [ 20.8 ] seconds.
[INFO] DONE: Spectral decomposition of the kinship matrix succeeded in [ 268.2 ] seconds.
[WARN] Autosomal kinship loaded, but no hemizygote region kinship provided, some sex chromosome tests will be skipped.
[INFO] Impute missing genotype to mean (by default)
[INFO] Analysis started
Segmentation fault

Why this happen ? and How can we fix it ?
This is part of GIANT project and the deadline is approaching.

Thank you very much!
Best regards

Zishan
@zhanxw
Copy link
Owner

zhanxw commented Apr 24, 2017 via email

@zishang30
Copy link
Author

The full command line are:
rvtest --inVcf HRC_N3788_chr22.vcf.gz --pheno phenotypes_all.ped --pheno-name HT_INV --kinship kinship_matrix.kinship --meta score,cov[windowsize=500000] --out HRC_N3788_chr22

We already enlarged the memory to 200GB, and try to run it in an large disk space(5TB), and update the rvtest to newest version, but we still have this problem...

Now I try to run the rvtest without the kinshipmatrix command, the full command as follow:
rvtest --inVcf HRC_N3788_chr21.vcf.gz --pheno phenotypes_all.ped --pheno-name HT_INV --meta score,cov[windowsize=500000] --out HRC_N3788_chr21

And the analysis start:
[INFO] Program version: 20170210
[INFO] Analysis started at: Mon Apr 24 15:59:55 2017
[INFO] Loaded [ 3788 ] samples from VCF files
[INFO] Loaded [ 3782 ] sample pheontypes
[WARN] Total [ 6 ] samples are dropped from VCF file due to missing phenotype.
[WARN] Drop sample [ 922000077 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922003426 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922003311 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922001212 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922000100 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922000199 ] from VCF file due to missing phenotype
[INFO] Loaded 1850 male, 1932 female and 0 sex-unknonw samples from phenotypes_all.ped
[INFO] Analysis begins with [ 3782 ] samples...
[INFO] load 1 parameters.
[INFO] Meta analysis uses window size 500,000 to produce covariance statistics under additive model
[INFO] Impute missing genotype to mean (by default)
[INFO] Analysis started

So this fault is because of kinshipmatrix?
But we do need to run rvtest with kinshipmatrix.
How could we run rvtest with kinshipmatrix and avoid Segmentation fault?

Thank you very much for your reply!

Best regards

Zishan

@zhanxw
Copy link
Owner

zhanxw commented Apr 24, 2017

When you used rvtests without the kinship option, did you get analysis outputs successfully?
I guess that the generated kinship matrix did not work well with the VCF file.
Maybe one of them is corrupt.
Can you first make sure the input VCF file is correct (I am referring to #22)?

@zishang30
Copy link
Author

Dear Xiaowei

I suddenly remembered one point may be important to this segmentation fault:
Kinship matrix is used to calculate for family study and relatedness.
Actually our cohort do not have population stratification (only European ancestry) and related individuals(we already exclude related individuals after PCA analysis) , should we still need to generate kinship matrix to run it with RVTEST?
May be this is the problem?
But the statistic plan ask us to run rvtest with kinshipmatrix....
If we don't need kinshipmatrix, could we just run rvtest without kinshipmatrix?
PS I already check the kinship matrix coefficient, and most them are very small(<0.01).

@zhanxw
Copy link
Owner

zhanxw commented Apr 25, 2017

I think you kinship matrix can still be helpful after you excluding related individuals and using only Europeans.
Is there any possibility that I can use your data for debugging purpose?

@zishang30
Copy link
Author

Dear Xiaowei

Thank you very much for your reply.
I have consult this to my supervisor, but he said we can not sent you the data because of data protect privacy in our institute, I am sorry.

What I means yesterday is: Is there any possibility that the generated kinship matrix are not work well with VCF file because our study actually don't need kinship matrix?

@zishang30
Copy link
Author

zishang30 commented Apr 26, 2017 •
edited
Loading

Dear Xiaowei

I guess this segmentation fault may not only due to the kinship matrix itself but also to the software...?
Because I found I have successfully run rvtest with kinshipmatrix one month ago:
INFO] Program version: 20170210
[INFO] Git Version: 584dea4
[INFO] Parameters BEGIN

ParameterList created at Mon Mar 13 18:19:13 2017

--inVcf "chr1_notmonomorph.vcf.gz" --out HRC_Chr1" --pheno "phenotypes_all.ped" --pheno-name "HT_INV" --meta "score,cov[Windowsize=500000]" --kinship "1_kinship_matrix.kinship"
[INFO] Parameters END
[INFO] Analysis started at: Mon Mar 13 18:19:14 2017
[INFO] Loaded [ 3788 ] samples from VCF files
[WARN] Skip: Missing or invalid phenotype type, skipping line 602 [ 922000077 922000077 0 0 NA NA ] ...
[WARN] Skip: Missing or invalid phenotype type, skipping line 1243 [ 922003426 922003426 0 0 NA NA ] ...
[WARN] Skip: Missing or invalid phenotype type, skipping line 1456 [ 922003311 922003311 0 0 NA NA ] ...
[WARN] Skip: Missing or invalid phenotype type, skipping line 1624 [ 922001212 922001212 0 0 NA NA ] ...
[WARN] Skip: Missing or invalid phenotype type, skipping line 1625 [ 922000100 922000100 0 0 NA NA ] ...
[WARN] Skip: Missing or invalid phenotype type, skipping line 1996 [ 922000199 922000199 0 0 NA NA ] ...
[INFO] Loaded [ 3782 ] sample pheontypes
[WARN] Total [ 6 ] samples are dropped from VCF file due to missing phenotype.
[WARN] Drop sample [ 922000077 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922003426 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922003311 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922001212 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922000100 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922000199 ] from VCF file due to missing phenotype
[INFO] Loaded 1850 male, 1932 female and 0 sex-unknonw samples from phenotypes_all_f4.ped
[INFO] Analysis begins with [ 3782 ] samples...
[INFO] load 1 parameters.
[INFO] Meta analysis uses window size 500,000 to produce covariance statistics under additive model
[INFO] Family-based model specified. Loading kinship file...
[INFO] DONE: Loaded kinship file [ S4F41_kinship_matrix.kinship ] successfully in [ 326.0 ] seconds.
[INFO] DONE: Spectral decomposition of the kinship matrix succeeded in [ 2085.4 ] seconds.
[WARN] Autosomal kinship loaded, but no hemizygote region kinship provided, some sex chromosome tests will be skipped.
[INFO] Impute missing genotype to mean (by default)
[INFO] Analysis started
[INFO] Analyzed [ 1562110 ] variants
[INFO] Analysis ends at: Tue Mar 21 16:41:05 2017
[INFO] Analysis took 685311 seconds

...........
And I try to run rvtest use the same vcf.gz and the same kinshipmatrix we made before, all the parameters are the same and we run the command again:
New version of rvtests is available:New version (20170228) is availabe. Feel free to upgrade.
[INFO] Program version: 20170210
[INFO] Analysis started at: Wed Apr 26 15:26:41 2017
[INFO] Loaded [ 3788 ] samples from VCF files
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 602 [ 922000077 922000077 0 0 NA NA ] ...
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 1243 [ 922003426 922003426 0 0 NA NA ] ...
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 1456 [ 922003311 922003311 0 0 NA NA ] ...
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 1624 [ 922001212 922001212 0 0 NA NA ] ...
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 1625 [ 922000100 922000100 0 0 NA NA ] ...
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 1996 [ 922000199 922000199 0 0 NA NA ] ...
[INFO] Loaded [ 3782 ] sample pheontypes
[WARN] Total [ 6 ] samples are dropped from VCF file due to missing phenotype.
[WARN] Drop sample [ 922000077 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922003426 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922003311 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922001212 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922000100 ] from VCF file due to missing phenotype
[WARN] Drop sample [ 922000199 ] from VCF file due to missing phenotype
[INFO] Loaded 1850 male, 1932 female and 0 sex-unknonw samples from phenotypes_all_f4.ped
[INFO] Analysis begins with [ 3782 ] samples...
[INFO] load 1 parameters.
[INFO] Meta analysis uses window size 500,000 to produce covariance statistics under additive model
[INFO] Family-based model specified. Loading kinship file...
[INFO] DONE: Loaded kinship file [ 1_kinship_matrix.kinship ] successfully in [ 20.1 ] seconds.
[INFO] DONE: Spectral decomposition of the kinship matrix succeeded in [ 159.4 ] seconds.
[WARN] Autosomal kinship loaded, but no hemizygote region kinship provided, some sex chromosome tests will be skipped.
[INFO] Impute missing genotype to mean (by default)
[INFO] Analysis started
Segmentation fault

The same input and the same command but this time we still got the Segmentation fault....
So this may not only due to the kinship matrix ......
苍天啊！！！难道是因为水星逆行吗T0T......

Could you just show me how to debug this segmentation fault in my computer ?

Sorry to bring you so much trouble.....
Thank you very much!

Best regards

Zishan

@zhanxw
Copy link
Owner

zhanxw commented Apr 26, 2017

Can you please list the files under the analysis directory?
Please paste the results of "ls -l".
I doubt some kinship files are empty and that caused the problem.

@zishang30
Copy link
Author

zishang30 commented Apr 28, 2017 •
edited
Loading

Dear xiaowei
I just move all empty file in the analysis directory, move data to new empty disk and make "data directory" and "result directory", and make sure there are no empty files in this 2 directory and run rvtest again......

still failed with segmentation fault......

BY the way, I consult this problem with the computer scientist in our center, he give me 3 possible reason:

  1. Hardware RAM problem (we already use largest RAM so this may not be a problem)
  2. Renew of needed library
    ---what installed and which library are used
    ---missing library_path due to update the software version. (we download the full software so this may not be a problem)
  3. Problem of Compilation (programming) --- gcc -03 may casue this problem, should be gcc -02

And then the scientist who responsible for our server try to download the source code of rvtest and try to compile rvtest on our server, and found some error:
#################################
rapidjson-1.0.2/thirdparty/gtest/
rapidjson-1.0.2/travis-doxygen.sh
ln -s -f rapidjson-1.0.2 rapidjson
sh -c '# no need to compile, just need its header'
make[2]: Leaving directory /opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third' g++ -MMD -O2 -D__STDC_LIMIT_MACROS -std=c++0x -Wall -Wno-unused-function -fopenmp -DGIT_VERSION="\"Unknown\"" -D_USE_KNETFILE -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/samtools/bcftools -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/samtools -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/tabix -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/eigen -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/pcre/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/gsl/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/zlib -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/bzip2 -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/nlopt/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/rapidjson/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/regression -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/libVcf -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/base -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/libsrc -I. -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2 -c Argument.cpp -o Argument.o g++ -MMD -O2 -D__STDC_LIMIT_MACROS -std=c++0x -Wall -Wno-unused-function -fopenmp -DGIT_VERSION="\"Unknown\"" -D_USE_KNETFILE -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/samtools/bcftools -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/samtools -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/tabix -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/eigen -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/pcre/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/gsl/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/zlib -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/bzip2 -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/nlopt/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/rapidjson/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/regression -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/libVcf -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/base -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/libsrc -I. -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2 -c Exception.cpp -o Exception.o g++ -MMD -O2 -D__STDC_LIMIT_MACROS -std=c++0x -Wall -Wno-unused-function -fopenmp -DGIT_VERSION="\"Unknown\"" -D_USE_KNETFILE -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/samtools/bcftools -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/samtools -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/tabix -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/eigen -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/pcre/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/gsl/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/zlib -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/bzip2 -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/nlopt/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/third/rapidjson/include -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/regression -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/libVcf -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/base -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/libsrc -I. -I/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2 -c IO.cpp -o IO.o IO.cpp: In function 'int removeEmptyField(std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >*)': IO.cpp:863: error: expected primary-expression before '[' token IO.cpp:863: error: expected primary-expression before ']' token IO.cpp:863: error: expected primary-expression before 'const' make[1]: *** [IO.o] Error 1 make[1]: Leaving directory /opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/base'
make: *** [/opt/rvtests/rvtests-2.0.2-src/rvtests-2.0.2/base/lib-base.a] Error 2
[root@055 rvtests-2.0.2]#
##############################################

Does this help you to fix the problem?
另外，我的表哥在国内也是电子行业的，他建议看看编译器地址段设置，再看看有没有浮点运算没处理好，或者是溢出。会不会是kinshipmatrix 里面有一些数字太小比如说-5.28E-05,6.63E-05
这样导致了问题?

@zhanxw
Copy link
Owner

zhanxw commented Jun 6, 2017 •
edited
Loading

没准编译器版本太老？能贴一下运行这个的结果么： gcc --version? 谢谢！

@zhanxw
Copy link
Owner

zhanxw commented Jun 13, 2017

Close it for now as we have not heard back from the issue submitter @zishang30
This issue can be re-opened later.

@zhanxw zhanxw closed this as completed Jun 13, 2017
@azrakurbasic
Copy link

Hi,
I've downloaded binary from https://github.com/zhanxw/rvtests/releases/tag/v2.0.2. I get massage
"strtod: Invalid argument" in the script error file, i.e. :

[INFO] Loaded [ 5098 ] samples from VCF files
strtod: Invalid argument
[WARN] Skip: Missing or invalid phenotype type, skipping line 2 [ 1 2401910120 0 0 2 NA
0.492007349143316 0.492007349143316 21.3153420398514 NA NA NA NA NA
NA NA NA NA 0.41940866606156 0.41940866606156 0.133433671542592 NA
NA NA NA NA -0.349571569933184 -0.349571569933184 NA 0.583089320041346
0.583089320041346 ] ...
strtod: Invalid argument

This is however nor present in the log file obtained from retest. Any explanation?
Thanks in advance/Azra

@zhanxw
Copy link
Owner

zhanxw commented Jun 16, 2017

You can safely ignore this error "strtod: Invalid argument".
The new version of RVTESTS has removed this warning message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zhanxw @azrakurbasic @zishang30