强化学习：资产配置

[ChenglongChen]

1. Framekwork

• 开发RL框架，包括RLTradeStrategy（集成state/action interpreter，reward，policy，backtest），simulator等
• 开发topkdropout strategy，学习n drop参数
• 开发weight strategy，直接学习仓位管理
• 开发topk strategy，直接学习topk持仓
• 开发TopkDropoutSelectionStrategy，直接学习n drop的sell和buy
• Only tradable集成到weightstrategy：https://github.com/microsoft/qlib/blob/main/qlib/contrib/strategy/signal_strategy.py#L150
• Only tradable集成到topkstrategy：https://github.com/microsoft/qlib/blob/main/qlib/contrib/strategy/signal_strategy.py#L150
• fix start time issue, when start time is not tradable

2. Feature

• add historical features
• add position features
• add n_step historical features
• support Alpha158

3. Model

• 使用PPO学习n drop参数，action空间太小，效果不显著
• 使用DDPG/TD3学习rerank & weighting
• 使用MultiBinary & PPO直接输出topk，权重使用均仓
• 使用imitation learning，学习lgbm的输出，同时学习actor：https://github.com/thu-ml/tianshou/blob/master/tianshou/policy/imitation/td3_bc.py
• add transformer in MetaNet
• add different rewards

4. Trade

• 优化模型效果，超越目前的baseline
• 重构tradestrategy，以支持实盘

[ChenglongChen]

1. Overall Performance

Reward	Strategy	Baseline	MetaPPO	MetaDDPG	MetaTD3	MetaSAC
InformationRatioReward	TopkDropoutSignalStrategy	1.656972	1.971290		1.929209	1.179975
InformationRatioReward	WeightStrategy	0.785682	1.338792		0.867902
ExessReturnReward	TopkDropoutSignalStrategy	0.263561	0.228271		0.191836
ExessReturnReward	WeightStrategy	0.110891	0.186833		0.167790

2. Learning Curve

2.1 InformationRatioReward

a) TopkDropoutSignalStrategy+MetaPPO

b) TopkDropoutSignalStrategy+MetaTD3

c) TopkDropoutSignalStrategy+MetaSAC

d) WeightStrategy+MetaPPO

e) WeightStrategy+MetaTD3

f) WeightStrategy+MetaSAC

2.2 ExessReturnReward

a) TopkDropoutSignalStrategy+MetaPPO

b) TopkDropoutSignalStrategy+MetaTD3

c) TopkDropoutSignalStrategy+MetaSAC

d) WeightStrategy+MetaPPO

e) WeightStrategy+MetaTD3

f) WeightStrategy+MetaSAC

[ChenglongChen]

Overall Performance

Reward	Strategy	Baseline	MetaPPO	MetaSAC
EpisodeInformationRatioReward	TopkDropoutSelectionStrategy	1.644155	1.884990	1.415276
EpisodeInformationRatioReward	TopkDropoutDynamicStrategy	1.644155	2.143160

Learning Curve