Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

estimator is apparently biased for large SNR, large n case #1

Closed
shabbychef opened this issue Mar 6, 2018 · 13 comments

Comments

Projects
None yet
2 participants
@shabbychef
Copy link

commented Mar 6, 2018

Hi;

I didn't want to catch you by surprise, but I have been analyzing your estimator via the sharpeRratio package. I think it is worth filing an issue for the apparent bias in the case when the signal noise ratio and number of samples is large. You can see my write up here: http://www.sharperat.io/sharper-estimator.html ; The 'bias' is inferred from in the second plot, on the lower right, which I reproduce here. It is the RMSE versus sample size:

sharper_rmses-1

Since the drawdown estimator has lower standard error than the Sharpe for these experiments, the worse RMSE must be due to bias. A later blog post confirms this.

Again, I don't mean to surprise you with all this. I have other issues with the drawdown estimator (see the followup posts: http://www.sharperat.io/tag/drawdown-estimator.html ), but I think this one is a limitation of the software and not the method itself, and could easily be corrected; as far as I understand it, you could make the fix by expanding the range of simulations used to train the spline function that gives SNR estimates as a function of the number of highs and lows.

@damienchallet

This comment has been minimized.

Copy link
Owner

commented Mar 6, 2018

@shabbychef

This comment has been minimized.

Copy link
Author

commented Mar 6, 2018

I am interested in the skewed returns case as well, but I think it will be a bigger theoretical project.

@shabbychef

This comment has been minimized.

Copy link
Author

commented May 9, 2018

Again, I don't want to catch you by surprise, but it looks like merely shrinking the moment-based Sharpe by a factor of 0.75 gives reduced standard errors (and bias) similar to that seen by the drawdown estimator, as I write here: http://www.sharperat.io/shrinkage-sharpe.html . Perhaps retraining your spline function on a very wide range of population Sharpe will reduce the bias and shrinkage?

@damienchallet

This comment has been minimized.

Copy link
Owner

commented May 17, 2018

@shabbychef

This comment has been minimized.

Copy link
Author

commented May 22, 2018

hi Damien.

If I understand the text around Eq. (8), it is based on extensive simulations. My worry is that the simulations are hiding the shrinkage caused by the limited range used to train the spline function. Put another way, taking the moment-based Sharpe and multiplying it by 0.8 seems to give the same bias-variance tradeoff as the drawdown-based estimator, but I don't think we want to claim Eq (8) applies to that method. (I agree there is no universal shrinkage for the moment-based estimator; the 0.8 is simply a straw man.) I am not sure about how Eq. 8 is being used to estimate the bias of the vanilla Sharpe estimator. However, that bias can be estimated directly by a Taylor expansion, and seems to be fairly small for 'reasonable' population skews.

Regarding the Cramer-Rao bound, the problem I considered was rather basic: the estimator 'knows' the df of the t-distribution of returns ('nu'), as well as the standard deviation (fixed at 1), and has to estimate the mean. However, an estimator could freely ignore all that information, pass the returns to a Sharpe estimator (like the drawdown estimator) as a black box, then use its estimate of the Sharpe as an estimate of the mean (knowing that sigma=1). Using an estimator of Sharpe in this problem should put one at a strict disadvantage, and one should see higher mean square errors. However, to the contrary, we see the drawdown estimator giving MSE below the Cramer Rao 'floor'. The inescapable conclusion is that the assumptions of CRLB are violated, most probably that errors are independent of the unknown.

@shabbychef

This comment has been minimized.

Copy link
Author

commented Jun 4, 2018

hi again; As I did not understand your comments about the bias of the Sharpe, I did some research, and found Yong Bao's 2009 paper, which I write up here: http://www.sharperat.io/sharpe-is-biased.html . I would be curious to see how it corresponds to the bias in terms of nu that you hint at above, for the case of t-distributed returns.

@damienchallet

This comment has been minimized.

Copy link
Owner

commented Jun 4, 2018

@shabbychef

This comment has been minimized.

Copy link
Author

commented Feb 26, 2019

Hi Damien,

I am curious about the status of the spline function. I see that there is a repo for the R package which is described as version 1.2, and the code accepts a nu parameter. Is that code ready for general use?

thanks,

@damienchallet

This comment has been minimized.

Copy link
Owner

commented Mar 12, 2019

@shabbychef

This comment has been minimized.

Copy link
Author

commented Mar 14, 2019

The following code illustrates the problem with bias in the current implementation:

# draw from a shifted rescaled t-distribution, then feed to the 
# drawdown estimator
simit <- function(nsim=1000,zeta=0.1,nday=100,nu=4) {
  require(sharpeRratio,quietly=TRUE)
  retv <- replicate(nsim,{
    x <- zeta + sqrt((nu-2)/nu) * rt(nday,df=nu)
    # the moment-based sample sharpe ratio
    ssr <- mean(x) / sd(x)
    # the drawdown estimator; potentially feed in 'nu' here for testing.
    ddr <- sharpeRratio::estimateSNR(x)$SNR
    c(ssr,ddr)
  })
  rownames(retv) <- c('moment_based','drawdown')
  t(retv)
}
set.seed(1234)
pzeta <- 0.1
colMeans(simit(nsim=1e3,zeta=pzeta)) / pzeta

I get the following, indicating the Sharpe ratio is slightly positively biased (a known phenomenon, shrinking with sample size), but the drawdown estimator is very shrunken, with mean values typically around 70-80% of the population value. Since the pzeta is in "daily" units, values around 0-0.2 should be typical for an equities strategy.

moment_based     drawdown 
      1.0210       0.7421 
@damienchallet

This comment has been minimized.

Copy link
Owner

commented Mar 19, 2019

Good news, the spline is fine. Bad news, the approximate relationship in my paper on which the problematic package version does not hold for every timeseries length and number of records. Good news: thanks to large scale simulations, I have found a way to fix the problem. See my sharpeRratio repository. Please test it and feel free to report any issues on the latter repo.

@shabbychef

This comment has been minimized.

Copy link
Author

commented Mar 20, 2019

That is good news. For my simple test above I now get the results

moment_based     drawdown 
      1.0210       0.9658 

I am curious, however, whether this change will affect the apparent RMSE improvements of the drawdown estimator over the moment estimator?

@damienchallet

This comment has been minimized.

Copy link
Owner

commented Mar 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.