max.sharpe() OptimizationError

[alienss]

Hey,

Trying the basic tutorial using the test prices seems to work fine, but inputting crypto prices leads to errors when calculating the max Sharpe value:

df_merged.head()

Date	BTC	ETH	LTC
2018-01-01	13657.2	772.641	229.033
2018-01-02	14982.1	884.444	255.684
2018-01-03	15201	962.72	245.368
2018-01-04	15599.2	980.922	241.37
2018-01-05	17429.5	997.72	249.271

mu = expected_returns.mean_historical_return(df_merged)

BTC   -0.654099
ETH   -1.011251
LTC   -1.016013
dtype: float64

S = risk_models.sample_cov(df_merged)

	BTC	ETH	LTC
BTC	0.447307	0.467077	0.487518
ETH	0.467077	0.756588	0.623595
LTC	0.487518	0.623595	0.803425

ef = EfficientFrontier(mu, S)
raw_weights = ef.max_sharpe()

OptimizationError                         Traceback (most recent call last)
<ipython-input-96-c11009779024> in <module>()
      1 ef = EfficientFrontier(mu, S)
----> 2 raw_weights = ef.max_sharpe()

1 frames
/usr/local/lib/python3.6/dist-packages/pypfopt/base_optimizer.py in _solve_cvxpy_opt_problem(self)
    198             raise exceptions.OptimizationError
    199         if opt.status != "optimal":
--> 200             raise exceptions.OptimizationError
    201         self.weights = self._w.value.round(16) + 0.0  # +0.0 removes signed zero
    202 

OptimizationError: Please check your objectives/constraints or use a different solver.

Weirdly enough, if I limit the price interval to just a couple of months, max_sharpe works fine, but then min_volatility breaks down with the same error.

[robertmartin8]

Hi @alienss

Thanks for reaching out! I suspect the problem is that with the expected returns are negative (to be honest I don't quite understand how you are getting a value of more than -1... ). This is quite unusual from a portfolio optimisation perspective, so I'm not fully surprised that the optimiser is complaining. Here are some suggestions:

1. Use a different estimate of expected returns. In this case, I suggest you look at ema_historical_return with a lower span (maybe 100). I think you are much more likely to get better results if most of your expected returns are positive.
2. Use exp_cov. Again, because you are dealing with an asset that has had massive historical swings, it is worth paying more attention to recent data.

Let me know how it goes!

Best,
Robert

[alienss]

Hmm, interesting observation, seems that I might've been creating my dataframe a bit weird.

start = "2018-01-01"
end = "2018-12-31"

results = []
coins = ['BTC', 'ETH', 'LTC']

for coin in coins:
  result = pdr.DataReader(coin + '-USD','yahoo',start,end)
  result = result.drop(columns=['High', 'Low', 'Open', 'Close', 'Volume'])
  result.rename(columns={'Adj Close' : coin}, inplace=True)
  results.append(result)

I was then doing

df_merged = reduce(lambda left,right: pd.merge(left,right,on=['Date'], how='outer'), results)

but I now switched to

df_merged = reduce(lambda left,right: pd.merge(left,right, left_index=True, right_index=True), results)

and it seems to give me better results for mu:

BTC   -0.632080
ETH   -0.750600
LTC   -0.932068
dtype: float64

Unfortunately, I'm still getting the same error for max.sharpe().

I'll try your suggestions and see how it works.

Thanks!

[CescFF]

Hello,
I had a similar issue:

"Try another solver, or solve with verbose=True for more "
cvxpy.error.SolverError: Solver 'OSQP' failed. Try another solver, or solve with verbose=True for more information.

But only when I updated to pyportfolioopt 1.1.0 from pyportfolioopt 0.5.5.
It's true that there are most of the assets with negative expected return at the moment of failure. Already tried ema and exp_cov and nothing changes.

Regards,

[robertmartin8]

@CescFF @alienss

The reason things broke in 1.1.0 is that I rebuilt the backend to make it do the optimisation 'properly', using an actual convex optimiser rather than a generic nonlinear optimiser. So if it is currently raising an error, it is because the optimisation problem is not convex.

max_sharpe is possible to formulate as a convex opt problem (which is what I do on the backend), but it requires a variable substitution, which is why it doesn't play so well with other constraints. I'm afraid I can't help you debug more unless you send me your actual dataset so I can investigate why it's not convex.

If you want to access the scipy optimiser from v0.5.5, here's how you can do that:

import objective_functions

ef = EfficientFrontier(mu, S)
weights = ef.nonconvex_objective(
        objective_functions.sharpe_ratio,
        objective_args=(ef.expected_returns, ef.cov_matrix),
        weights_sum_to_one=True,
    )

I should warn you that this ignores your other constraints. If you would like to include them, you must re-build them using the scipy syntax. An example is below:

    min_weight, max_weight = 0.01, 0.3
    weights = ef.nonconvex_objective(
        objective_functions.sharpe_ratio,
        objective_args=(ef.expected_returns, ef.cov_matrix),
        constraints=[
            {"type": "eq", "fun": lambda w: np.sum(w) - 1},  # sum to 1
            {"type": "ineq", "fun": lambda w: w - min_weight},  # greater than min_weight
            {"type": "ineq", "fun": lambda w: max_weight - w},  # less than max_weight
        ],
    )

Hope this helps!

Best,
Robert

[alienss]

@robertmartin8 the scipy optimiser seems to work for my case.

Here's what I use for my dataset now:

start = "2018-01-01"
end = "2018-12-31"

stocks = ['BTC', 'ETH', 'LTC', 'XRP']
stock_dfs = []

for stock in stocks:
  stock_price = pdr.DataReader(stock + '-USD','yahoo',start,end)
  stock_price = stock_price.drop(columns=['High', 'Low', 'Open', 'Close', 'Volume'])
  stock_dfs.append(stock_price)

all_stock_prices = []
for stock_df in stock_dfs:
  all_stock_prices.append(stock_df['Adj Close'])

prices_df = pd.concat(all_stock_prices, axis=1)
prices_df.columns = stocks

I noticed one thing though, that's not really related to the optimizer. The annualized returns can sometime go below -1, since it's basically taking the mean returns and multiplying them with the number of trading days (eg. in a down market like for the above dataset, where the mean returns are negative, so multiplying those with 365 for example gets you -1.3 or something similar).

Don't know if there's a solution to this though (or if there is need for one, the scipy optimizer seems to deal fine in this case; on the other hand, I get "Expected annual return: -103.5%", which is... odd).

[alienss]

PS: Fixed the expected annual return issue by using compounding=True on mean_historical_return() (shouldn't that always be the correct way of calculating annualized returns?)

[robertmartin8]

@alienss

I'm glad you managed to get it to work! The compounding parameter is discussed in issue #85.

One thing you should think about is whether your expected returns are a good estimate of future returns – if they aren't, then the max_sharpe portfolio will be pretty useless!

Let me know if you have any other issues :)

Best,
Robert

[alienss]

@robertmartin8

In terms of expected returns being a good estimate of future returns, that's a tricky one, especially with cryptocurrency :) I guess I could either choose a lower frequency, such as monthly returns instead of annualized ones, or use the EMA returns with a short span.

To note though, while negative returns are above -1 now, the cvx optimizer still has issues solving it (the scipy one works ok though). I also tried manually adding this cvxopt implementation as described here, and that seems to solve the problem with an optimal status, so I'm unsure why the max_sharpe function cannot find an optimal solution.

[robertmartin8]

@alienss

The method used in the Quantopian thread is very different to ef.max_sharpe(). They are finding the efficient risk portfolios for a bunch of different risk levels, then returning the highest Sharpe ratio. If you want something like that, you might have better luck with the Critical Line Algorithm.

I am quite familiar with optimising crypto portfolios – I spent some time at a fund doing it professionally. In my experience using any simple model of expected returns leads to pretty bad performance – better to just use HRP or min_variance. Just my two cents

[alienss]

@robertmartin8

Thanks for the advice! Just curious, is min_variance similar to what ef.min_volatility() does?

[robertmartin8]

@alienss

whoops, I meant min_volatility 😂

[franceslius]

@robertmartin8 max.shape() optimation error can be solved by ur way, but max_quadratic_utility() also has the same issue.