Error after executing test run

[tkamath1]

Hi, I am receiving the following error after executing the test run for polyfun. I ran the following command in the polyfun conda environment:

python test_polyfun.py --python3 python --ldstore ldstore_v1.1_x86_6
4/ldstore

The following was the error message:
Writing fine-mapping results to /tmp/tmp4xe7wax0/finemap.1.46000001.49000001.gz Traceback (most recent call last): File "test_polyfun.py", line 226, in <module> test_finemapper(temp_dir, args.ldstore, args.python3) File "test_polyfun.py", line 208, in test_finemapper compare_dfs(tmpdir, gold_dir, outfile) File "test_polyfun.py", line 35, in compare_dfs assert np.all(df1[c] == df2[c]), 'found mismatch between %s and %s in column %s'%(file1, file2, c) AssertionError: found mismatch between /tmp/tmp4xe7wax0/finemap.1.46000001.49000001.gz and /home/tkamath/polyfun/gold/finemap.1.46000001.49000001.gz in column SNP

[omerwe]

Hi Tushar, I'm sorry to hear about your error. Can you please send me the file /tmp/tmp4xe7wax0/finemap.1.46000001.49000001.gz to oweissbrod@hsph.harvard.edu? If it was already deleted, please rerun the script and send me the temp file mentioned in the error message. I will take a look and try to figure out what went wrong. Thanks, Omer

…

On Tue, Feb 4, 2020, 3:03 PM Tushar Kamath ***@***.***> wrote: Hi, I am receiving the following error after executing the test run for polyfun. I ran the following command in the polyfun conda environment: python test_polyfun.py --python3 python --ldstore ldstore_v1.1_x86_6 4/ldstore The following was the error message: Writing fine-mapping results to /tmp/tmp4xe7wax0/finemap.1.46000001.49000001.gz Traceback (most recent call last): File "test_polyfun.py", line 226, in <module> test_finemapper(temp_dir, args.ldstore, args.python3) File "test_polyfun.py", line 208, in test_finemapper compare_dfs(tmpdir, gold_dir, outfile) File "test_polyfun.py", line 35, in compare_dfs assert np.all(df1[c] == df2[c]), 'found mismatch between %s and %s in column %s'%(file1, file2, c) AssertionError: found mismatch between /tmp/tmp4xe7wax0/finemap.1.46000001.49000001.gz and /home/tkamath/polyfun/gold/finemap.1.46000001.49000001.gz in column SNP — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#19?email_source=notifications&email_token=ACNCB4YJ2SZDJZYEUQKWGMDRBHCZRA5CNFSM4KP5DR7KYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4ILAEKMA>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACNCB46COUWBC4YOQFVHDK3RBHCZRANCNFSM4KP5DR7A> .

[davidaknowles]

I'm getting the same error. The content of the temp result file is:
temp.txt

[omerwe]

Hi David,

My apologies for this. Some users apparently get different fine-mapping results with the same input but I still haven't figured out why. First, are you sure you're using SuSiE v0.8.0 and the latest Polyfun code from Github?
If yes, can you please help me resolve this by invoking run_finemapper.py with the flag "--debug-dir" and sending me the list of intermediate files that were created? Specifically, please type the following command from the polyfun directory:

mkdir -p output
python run_finemapper.py \
    --geno example_data/chr1 \
    --sumstats example_data/chr1.finemap_sumstats.txt.gz \
    --n 383290 \
    --chr 1 \
    --start 46000001 \
    --end 49000001 \
    --method susie \
    --max-num-causal 5 \
    --cache-dir LD_cache \
    --out output/finemap.1.46000001.49000001.gz \
    --debug-dir <DEBUG_DIR> \
    --ldstore <PATH_TO_LDSTORE_EXECUTABLE>

specifying the correct DEBUG_DIR and LDstore executables.

Afterwards please send me the following two files:
The file called debug.zip that should have been created in the DEUBG_DIR
The file called output/finemap.1.46000001.49000001.gz (i.e. the regular output file)

Thanks again and hopefully I'll resolve this soon!

[davidaknowles]

Hmm so I'm getting
[ERROR] Error: File 'LD_cache/chr1.1.46000001.49000001.bcor_1' cannot be opened!
when I use this instead of test_polyfun.py!

[omerwe]

Hi, I sent the text below over email two weeks ago. I just noticed it's not here for some reason, so am re-posing this in case you never got the email:

Just to make sure: are you getting this error when specifying the "--debug-dir" flag, and not getting it when omitting this flag? The only effect of this flag is to create temp input files, everything else should stay exactly the same. Can you please confirm if this is the case? On my system the command works with and without specifying this flag...

As a side note, you can run test_polyfun.py without specifying "--ldstore". In this case the fine-mapping tests will not run but all the other ones will. If the other tests succeed, then you can be reassured that the only problems are related to the fine-mapping code, not the to the prior causal probabilities related code.

[hsomineni]

Hi Omer. I was trying to run polyfun.py to compute per SNP h ^ 2, and am getting the following error (shown in the attachment). Would be great if you can take a look at it and see if anything catches your attention.

First, I performed the test run with example_data, and it ran just fine. Then I made my own input file (sum stats file with about 5.6 million variants from across 22 chrs), and tried to rerun polyfun.py on this, using the annotations and weights from the example_data just to see how that works.

Looks like it's computing per-SNP h^2 for variants on almost half of those chromosomes before breaking, at around chr13.

Please let me know if you need any more details.
Thanks a lot for your time!!

[omerwe]

Thanks for the bug report. I suspect that the intersection of SNPs with sumstats and with annotations data does not span all 22 human chromosomes . This breaks down some of the assumptions of PolyFun, leading to this bug. The most likely reason is that the example annotations files only include a small random sample of genome-wide SNPs. I updated the code to give a clearer error message if this happens (please git pull and try again). If this is not the cause for this error please reopen this issue and I'll look into it further.

[hsomineni]

That makes so much sense. It's possible that none of those SNPs from my chr13 sum stats are in the example_data (annotation/weights). Will look more into this and update you shortly. Thanks a lot for the prompt response.