示例中的数据是否复权？

[JessieJie1998]

❓ Questions and Help

（抱歉我怕用英文不能描述清楚我的问题。）
我用下面的命令分别对比你们样例中给的数据和我在万德量化数据库WINDDF中提取的数据，发现有很大的不同。进一步，基于不同的数据训练模型回测的效果相差很多。

from qlib.data import D
D.features()

1.这个是你们的数据（YAHOO Finance）：

instrument	datetime	$open	$high	$low	$close	$volume	$money	$factor	$vwap	$change
SH600010	2020-01-02	4.700388	4.735730	4.665047	4.700388	57784524.0	NaN	3.534127	NaN	0.007576
SH600010	2020-01-03	4.735730	4.771071	4.700388	4.735730	69716760.0	NaN	3.534127	NaN	0.007519
SH600010	2020-01-06	4.735730	4.806412	4.700388	4.771071	81217512.0	NaN	3.534127	NaN	0.007463
SH600010	2020-01-07	4.771071	4.912436	4.735730	4.841753	99735016.0	NaN	3.534127	NaN	0.014815
SH600010	2020-01-08	4.735730	4.735730	4.735730	4.735730	81830424.0	NaN	3.534127	NaN	-0.021898

2.这是我提取的数据，其中$open、$high 、 $low 、 $close都是没有复权的数据

instrument	datetime	$open	$high	$low	$close	$volume	$money	$factor	$vwap	$change
SH600010	2020-01-02	1.33	1.34	1.32	1.33	2042178.25	272334.00000	8.332732	1.3335	0.007576
SH600010	2020-01-03	1.34	1.35	1.33	1.34	2463878.50	330344.46875	8.332732	1.3407	0.007519
SH600010	2020-01-06	1.34	1.36	1.33	1.35	2870329.75	386598.53125	8.332732	1.3469	0.007463
SH600010	2020-01-07	1.35	1.39	1.34	1.37	3524761.75	482354.53125	8.332732	1.3685	0.014815
SH600010	2020-01-08	1.36	1.37	1.33	1.34	2891990.75	390556.37500	8.332732	1.3505	-0.021898

2.这是我提取的数据，其中$open、$high 、 $low 、 $close均为复权调整后的数据

instrument	datetime	$open	$high	$low	$close	$volume	$money	$factor	$vwap	$change
SH600010	2020-01-02	11.08	11.17	11.00	11.08	2042178.25	272334.00000	8.332732	1.3335	0.007576
SH600010	2020-01-03	11.17	11.25	11.08	11.17	2463878.50	330344.46875	8.332732	1.3407	0.007519
SH600010	2020-01-06	11.17	11.33	11.08	11.25	2870329.75	386598.53125	8.332732	1.3469	0.007463
SH600010	2020-01-07	11.25	11.58	11.17	11.42	3524761.75	482354.53125	8.332732	1.3685	0.014815
SH600010	2020-01-08	11.33	11.42	11.08	11.17	2891990.75	390556.37500	8.332732	1.3505	-0.021898

然后我有如下问题：

1. 如何在https://finance.yahoo.com/lookup找到cn_data数据爬虫对应的地方呢？
2. 你们的$open、$high 、 $low 、 $close和$change是用的复权后的还是前的呢？我不知道如何在https://finance.yahoo.com/lookup上找到对应的数据，所以没有办法确认。
3. 你们觉得$open、$high 、 $low 、 $close和$change应该用复权后的还是前的呢？
   后续( qlib.contrib.data.handler)直接采用 close 或 vwap 计算股票未来收益率，且回测(qlid.contrib.evaluate.backtest)使用 close、open 或 vwap 进行交易，我觉得似乎应该用复权后的数据？
   4.似乎上述数据从某种程度上通过复权因子调整后差不多相同，我看到因子库(qlib.contrib.data.handler)中、对价格相关的因子做了“归一化”（比如除了close或者open），所以就算这些数据相差了一定的倍数，理论上训练和回测的结果应该非常相似，但是我用上述不同数据回测结果差别特别大，请问是为什么呢？

谢谢！

[zhupr]

1. cn_data from: https://query2.finance.yahoo.com/v8/finance/chart/{symbol} same as https://finance.yahoo.com/quote/600010.SS/history?p=600010.SS. Use yahooquery to get
2. cn_data is already adjusted: docs
   • cn_data adjusted code: adjusted_price
3. The difference with the WIND data is because cn_data has done a manual adjusted:
   • call adjusted_price and _manual_adj_data
4. The backtest results are different because some stock data in WIND and Yahoo are quite different, reference: https://github.com/microsoft/qlib/blob/main/scripts/data_collector/yahoo/README.md#collect-data-from-yahoo-finance

[JessieJie1998]

Thanks! Your reply really helps a lot. I have another question. @zhupr

I want to verify that the csv data and data generated by "python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --region cn" can work in the same way and have the same effect, becasue I will use my own csv data in later tests.

1. I use the function D.features() to get the data generated in the Qlib example and save them in csv format stock by stock.
2. Then I dump all the csv data and initialize another worklfow by dumped data.
3. Next, I use the same parameters as in the workflow_by_code example to train the model and do backtest.

However, there are differences between the results of csv data and the data generated by command. I know there is randomness in the LightGBM model, but I don't know wether such a big difference is normal.

[zhupr]

@JessieJie1998 Hi,
You can use datacompy to compare the differences between the two data sets, you can put out the screenshots. Reference:

# pip install datacompy
import datacompy

import qlib
from qlib.data import D

MARKET = "all"
FIELDS = ["$close", "$open", "$high", "$low", "$volume", "$factor"]

# get data set 01
qlib.init(provider_uri="your data set dir_01")
df1 = D.features(D.instruments(MARKET), fields=FIELDS)

# get data set 02
qlib.init(provider_uri="your data set dir_02")
df2 = D.features(D.instruments(MARKET), fields=FIELDS)

compare = datacompy.Compare(
    df1,
    df2,
    on_index=True,
    # TODO: Adjust these two parameters appropriately
    abs_tol=1e-08,  # Optional, defaults to 0
    rel_tol=1e-05,  # Optional, defaults to 0
)

res = compare.matches(ignore_extra_columns=True)
print(f"compare result: {res}")
if not res:
    print(compare.report())

[JessieJie1998]

@zhupr Thanks! It seems that dataframes initialized in different ways are the same.

But I am still confused why do the prediction result and backtest change a lot when I change provider_uri? The result should also stay unchanged when I change init way because the underlying data are the same.

[you-n-g]

@JessieJie1998
I think you can try to change the random seed of model on the same dataset.
By comparing the randomness from different random seed, you can check if the randomness of result is abnormal.

[github-actions]

This issue is stale because it has been open for three months with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days