Some Qustions about Data Prepared and Backtest

[prevailings]

❓ Questions and Help

I downloaded cn_data v2 (.bin data, update to 2021-06-11), and I arranged my .csv data by akshare from sina.
我下载了v2版本的数据（更新到2021-06-11）, 另外我通过akshare准备了2021-06-12到今天的csv数据。

1. The example data use "close" and "adjclose"
   通过示例代码下载的数据包含close和adjclose这两项
   (python scripts/get_data.py csv_data_cn --target_dir ~/.qlib/csv_data/cn_data)

while the doc suggests adjusted price and factor
文档里建议使用调整后的价格和复权因子。
(https://qlib.readthedocs.io/en/latest/component/data.html?highlight=adjusted#converting-csv-format-into-qlib-format).

which format should I use? The adjusted price I guess it should be "qfq", Not "hfq" and "bfq", and the factor should also be "qfq-factor"?
我应该使用哪种格式比较好？调整的价格应该是前复权，而不是后复权或者不复权， 复权因子也同样是前复权因子？

2. Since I have downed the cn_data, I just arranged "2021-06-12" to today .csv data, after transformed to .bin data, how to merge the two .bin data? The rested columns, like dividends, splits, just let it alone, do not deal with them?
   自己准备的数据转换成bin格式后(2021-06-12到今天)，如何和下载的bin格式数据（2021-06-11之前）合并到一起呢？
   另外下载的数据含有其他一些数据列，例如dividends, splits, 这些多余的列的bin数据不需要处理吧？

3. Jointquant used "Dynamic restoration mode" in backtest, which is to use the real price at that time buy and sell, and use the qfq price at that time. Does qlib also use this backtest mode?
   像其他家平台，回测使用的动态复权(真实价格)模式：
   回测时，使用当天的真实价格进行买卖；
   回测时，如果需要使用前复权价格，会以当天作为起始（当天复权因子为1），而不是以今天2021-10-10复权因子为1。
   我想知道qlib的回测也是这种模式吗？如果不是，可以具体说说吗？

英文表述可能不准确，请以中文为准，谢谢。
Thank you in advance.

[zhupr]

@prevailings Hi,

English:

bin data in get_data.py qlib_data Two-step operation on get_data.py csv_data_cn:

• converting data to "It does this by anchoring the current price and working backward":https://github.com/microsoft/qlib/blob/main/scripts/data_collector/yahoo/collector.py#L396
• divide the data from the "previous step" by the close of the first day: https://github.com/microsoft/qlib/blob/main/scripts/data_collector/yahoo/collector.py#L432

1. The data provided by qlib (get_data.py csv_data_cn) is the csv of the data sample from YahooFinance: close is the raw ata , adjclose is the "It does this by anchoring the current price and working backward" data, factor is similar to adjclose.
2. It is not recommended to combine data for the same stock from two different data sources, as even the raw data from different data sources may differ, even if the original data is the same, different data sources have different adjusted algorithms, resulting in different adjusted factors; fields that are not used, such as `splits', can be ignored for conversion
   • If the user wants to merge, the following steps are required:
     • The bin data (get_data.py qlib_data) provided by qlib is a two-step conversion of the raw data, as above
     • If the user's data is to be merged with qlib:
       • need to convert qlib's bin data (get_data.py qlib_data) to raw data, and then merge it with the user's raw data
       • the merged data are calculated together as "adjusted data"(need to ensure that the adjusted factor is the same for different data sources, this step is difficult to do if the adjusted algorithm of the data sources is not known)
3. qlib backtesting steps

   If a stock has close=1, adj_close=2, factor=adj_close/close=2; there is a signal at backtesting time to buy this stock deal_amount=151 shares (adjusted): https://github.com/microsoft/qlib/blob/main/qlib/backtest/exchange.py#L620

   • If the stock to be traded has a factor data field: the deal_amount to be bought is specified according to the real stock price:
     • recalculate the number of shares adjusted based on the original price of this stock (Chinese stock trading unit trade_unit = 100 shares): deal_amount = (deal_amount * factor + 0.1) // trade_unit * trade_unit / factor = 151 * 2 + 0.1 // 100 * 100 / 2 = 150；the final backtest will use adj_close to trade deal_amount=150
     • if there is no factor field, adj_close will eventually be used to trade deal_amount=151

中文:

get_data.py qlib_data中的bin数据 对get_data.py csv_data_cn做了两步操作:

• 把数据转换为前复权: https://github.com/microsoft/qlib/blob/main/scripts/data_collector/yahoo/collector.py#L396
• 把前复权的数据 除以 第一天的close: https://github.com/microsoft/qlib/blob/main/scripts/data_collector/yahoo/collector.py#L432

如果能获取到后复权数据, 建议使用后复权数据

1. qlib提供的数据(get_data.py csv_data_cn)是从YahooFinance获取数据样例的csv的: close是原始数据, adjclose是前复权数据, 由于是adjclose是前复权, 所以复权因子factor是前复权因子.

2. 不建议把两个不同的数据源的同一个股票的数据合并, 因为不同数据源即使是原始数据也会有差异; 即使原始数据相同, 不同数据源的复权算法不同, 导致复权因子也不同；使用不到的字段如splits, 可以忽略转换

   • 如果需要合并, 需要做以下步骤:
     • qlib提供的bin数据(get_data.py qlib_data)是对原始数据做了两次转换, 如上;
     • 如果用户的数据要和qlib合并在一起:
       • 需要把qlib的bin数据(get_data.py qlib_data) 转换为未复权数据, 然后和用户的未复权数据进行合并
       • 合并后的数据一起计算复权(需要保证不同数据源的复权因子相同, 如果不知道数据源的复权算法, 这步很难做)

3. qlib回测大致逻辑：

   如果某个股票的close=1, adj_close=2, factor=adj_close/close=2; 回测时有个信号, 购买这个股票deal_amount=151股(复权后): https://github.com/microsoft/qlib/blob/main/qlib/backtest/exchange.py#L620

   • 如果需要交易的股票有factor数据字段: 会根据真实的股票价格规范需要买入的deal_amount, 示例:
     • 根据这个股票的未复权价格重新计算复权后的股数(A股交易单位trade_unit=100股): deal_amount = (deal_amount * factor + 0.1) // trade_unit * trade_unit / factor = 151 * 2 + 0.1 // 100 * 100 / 2 = 150；最后回测时会使用adj_close交易deal_amount=150股
   • 如果没有factor字段, 最终会使用adj_close交易deal_amount=151股

[prevailings]

Busy recently, forgot to reply.
Thank you very much!