Can we use two reward functions jointly?

[bhaskatripathi]

I want to optimize a portfolio by Maximizing the Sharpe ratio and at the same time maximizing Sortino ratio. Is it possible to use two rewards functions jointly in FinRL ? The portfolio will be both Risk-Return and Drawdown adjusted in this case.

[Athe-kunal]

The reward is extracted at each time when the agent takes action, so the daily return is an appropriate reward function as you can compute for each action. But the Sharpe ratio and Sortino ratio are generally formulated for yearly return. So in my opinion you can do hyperparameter optimization using Sharpe and Sortino ratio. Here in this tutorial we are doing HPO optimization for the validation dataset using Sharpe ratio only. You can also use the Sortino and Sharpe ratio as multi-objective optimization. Optuna has support for multi-objective function, presented here
Please try out using Sharpe and Sortino as multi-objective optimization parameters from Optuna.

[bhaskatripathi]

Thank you so much. Appreciate your response.
I was able to optimize the hyperparams using your optuna code and it makes sense to use returns as reward as it has a daily value.
However, I had a question :

Theoretically, the DRL model is trained using the train dataframe with return as a reward. This means the return will be maximized. However, why do you apply the objective only on validation set? The model has learned based on rewards but it is being tested based on Sharpe/Sortino. I am not able to understand this part. Ideally the you train a model on a given condition and see how it works on the unseen data given the same conditions. Am I missing something ?

[bhaskatripathi]

I think it will lead to overfitting if you tune the hyperparameters on the test set. Do you mean you have divided your data into Train, validation and test. Any tuning hyperparams on validation ? If that is what you are suggesting, then it is correct.

[Athe-kunal]

Yes it is like you train your model for the specified number of epochs with a hyperparameter configuration and based on that back trade through validation set. Based on validation sharpe ratio using Bayesian techniques pick the next best set of hyperparameters. Also, as you said it is divided into train-- validation-- test set.

[bhaskatripathi]

Thank you. I was able to do the multi-objective optimization and it is giving much improved results on the backtrade now.
Appreciate your timely help. God bless you!