Error: Estimating residual variance failed: the estimated value is negative

[jcm6t]

Thanks for making the PolyFun software available, and supporting it with great documentation.

During a finemapper run, we got the following warning which turned into an exception. I'm guessing that we are on the low end of sample size for using this approach (n=11,800) and this represents a sample size issue. We are using the provided UK BB LD-files.

python3 $POLYFUNDIR/finemapper.py
--ld LD_files/chr7_26000001_29000001
--sumstats chr7.res.prior.Z
--n 11802
--chr 7
--start 26000001
--end 29000001
--method susie
--max-num-causal 5
--out chr7.finemap1.gz

• Is this assumption correct or is there possibly something else happening ?
• Probably be helpful to have a more graceful handling of this situation

thanks, Joe

[WARNING] R[write to console]: Error in (function (bhat, shat, R, n, var_y, XtX, Xty, yty, X_colmeans = NA, :
Estimating residual variance failed: the estimated value is negative

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/mnt/t1/projects/polyfun/finemapper.py", line 1267, in
df_finemap = finemap_obj.finemap(locus_start=args.start, locus_end=args.end, num_causal_snps=args.max_num_causal,
File "/mnt/t1/projects/polyfun/finemapper.py", line 851, in finemap
susie_obj = self.susieR.susie_bhat(
AttributeError: module 'susieR' has no attribute 'susie_bhat'

[jdblischak]

I doubt this is an issue with your sample size. This is likely due to a mismatch between the study population and the LD reference panel. PolyFun runs susie_suff_stat() directly because it assumes you are using insample LD, e.g. how the PolyFun paper used UKBB summary statistics with UKBB LD.

I recommend passing the flag --susie-resvar 1 to finemapper.py. This results in susie_suff_stat() getting called with the arguments residual_variance = 1 and estimate_residual_variance = FALSE, which are the settings used by susie_rss().

polyfun/finemapper.py

Lines 844 to 845 in 594df4f

	residual_variance=(self.R_null if (residual_var_init is None) else residual_var_init),
	estimate_residual_variance=(residual_var is None),

https://github.com/stephenslab/susieR/blob/508fbfce737809aba691507b76cabf8c6a96d655/R/susie_rss.R#L200

[jcm6t]

John,
Thanks for the pointer to the susie interface issues. There is also a discussion happening here:
stephenslab/susieR#162