public access error when running get_data script

[icannotnamemyself]

❓ Questions and Help

We sincerely suggest you to carefully read the documentation of our library as well as the official paper. After that, if you still feel puzzled, please describe the question clearly under this issue.

running the following command

python get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn

I get this error:

requests.exceptions.HTTPError: 409 Client Error: Public access is not permitted on this storage account. for url: https://qlibpublic.blob.core.windows.net/data/default/stock_data/v2/qlib_data_cn_1d_0.9.1.zip

not seeing any expaination of how this data can be accessed publicly or privately in this documentation?

[icannotnamemyself]

for now , google the error , nothing can be found related to qlib...

[cgwu1999]

same problem!!!!

[cgwu1999]

and it works 2 days ago

[SunsetWolf]

Hello, @PiKaChu-wcg @wayne155 , we checked this issue, you can try to change the code in qlib.tests.data at line 163 to make it like this.
file_name = _get_file_name("latest")
After that, it will work fine.
If you want, you can pull the request, fix the issue, and become a contributor to qlib.

[icannotnamemyself]

Hello, @PiKaChu-wcg @wayne155 , we checked this issue, you can try to change the code in qlib.tests.data at line 163 to make it like this. file_name = _get_file_name("latest") After that, it will work fine. If you want, you can pull the request, fix the issue, and become a contributor to qlib.

thanks for reply,
I've changed the code ,and seems the problem still exist?

python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn

code:

        file_name = _get_file_name('latest')
        if not self.check_dataset(file_name, version):
            file_name = _get_file_name("latest")
        self._download_data(file_name.lower(), target_dir, delete_old, dataset_version=version)

requests.exceptions.HTTPError: 409 Client Error: Public access is not permitted on this storage account. for url: https://qlibpublic.blob.core.windows.net/data/default/stock_data/v2/qlib_data_cn_1d_latest.zip

[Karllzy]

Hello, @PiKaChu-wcg @wayne155 , we checked this issue, you can try to change the code in qlib.tests.data at line 163 to make it like this. file_name = _get_file_name("latest") After that, it will work fine. If you want, you can pull the request, fix the issue, and become a contributor to qlib.

thanks for reply, I've changed the code ,and seems the problem still exist?

python scripts/get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn

code:

        file_name = _get_file_name('latest')
        if not self.check_dataset(file_name, version):
            file_name = _get_file_name("latest")
        self._download_data(file_name.lower(), target_dir, delete_old, dataset_version=version)

requests.exceptions.HTTPError: 409 Client Error: Public access is not permitted on this storage account. for url: https://qlibpublic.blob.core.windows.net/data/default/stock_data/v2/qlib_data_cn_1d_latest.zip

I have the same problem and I tried to visit the given URL with other tools, but the result is the same.

PublicAccessNotPermitted Public access is not permitted on this storage account. RequestId:a6f970f9-e01e-001c-273b- 9f9aff000000 Time:2023-06-15T03:44:40.9912199Z

It seems that the server of the given url is not public now.

[SunsetWolf]

Public access to the data is currently blocked, and we are working on a solution to this issue.

[icannotnamemyself]

Public access to the data is currently blocked, and we are working on a solution to this issue.

no offense, when can this be fixed? any other data sources we can use to workaround?

[SunsetWolf]

This is a problem we still need to consider. There is currently a solution.
Change line 19 in qlib.tests.data to.
REMOTE_URL = "http://fintech.msra.cn/stock_data/downloads"
and then execute it on the command line.
python get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn
There will be no problem.
Here are the results of my execution.

[icannotnamemyself]

"http://fintech.msra.cn/stock_data/downloads"

thank you so much , it finally worked!!!

[Karllzy]

This is a problem we still need to consider. There is currently a solution. Change line 19 in qlib.tests.data to. REMOTE_URL = "http://fintech.msra.cn/stock_data/downloads" and then execute it on the command line. python get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn There will be no problem. Here are the results of my execution.

Thank you, I changed the file in the site-packages and it worked.

The path of it in a python38 virtual environment is venv/lib/python3.8/site-packages/qlib/tests/data.py

[davidt922]

In my case line 65 in qlib.tests.data a fp.close() have to be added to not have a BadZipFile error

[lixinzju]

This is a problem we still need to consider. There is currently a solution. Change line 19 in qlib.tests.data to. REMOTE_URL = "http://fintech.msra.cn/stock_data/downloads" and then execute it on the command line. python get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn There will be no problem. Here are the results of my execution.

so cool，thank you！

[szeyao]

REMOTE_URL = "http://fintech.msra.cn/stock_data/downloads"

Is this link still working? I'm getting the same error above now.

[yangjichen]

got same error

REMOTE_URL = "http://fintech.msra.cn/stock_data/downloads"

[yangjichen]

Can any body help me…😭😭😭same error

[SunsetWolf]

Hi, @yangjichen @szeyao @davidt922 The above proposed method is a temporary method, the server is located in China, so the experience of foreign users is very bad, now this problem has been fixed in PR 1558, using the source installation, can solve this problem. Thank you for your feedback, and sorry for the bad experience.

[yiyione]

I got another ERROR (TypeError: token must be bytes) after #1558
Can anybody help ?

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/qlib/qlib/run/get_data.py", line 9, in <module>
    fire.Fire(GetData)
  File "/usr/local/lib/python3.9/dist-packages/fire/core.py", line 141, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.9/dist-packages/fire/core.py", line 475, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
  File "/usr/local/lib/python3.9/dist-packages/fire/core.py", line 691, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/qlib/qlib/tests/data.py", line 191, in qlib_data
    if not self.check_dataset(file_name):
  File "/qlib/qlib/tests/data.py", line 96, in check_dataset
    url = self.merge_remote_url(file_name)
  File "/qlib/qlib/tests/data.py", line 39, in merge_remote_url
    token = fernet.decrypt(self.TOKEN).decode()
  File "/usr/lib/python3/dist-packages/cryptography/fernet.py", line 75, in decrypt
    timestamp, data = Fernet._get_unverified_token_data(token)
  File "/usr/lib/python3/dist-packages/cryptography/fernet.py", line 100, in _get_unverified_token_data
    utils._check_bytes("token", token)
  File "/usr/lib/python3/dist-packages/cryptography/utils.py", line 30, in _check_bytes
    raise TypeError("{} must be bytes".format(name))
TypeError: token must be bytes

[SunsetWolf]

Hi @yiyione , a contributor has submitted PR1577, which fixes this issue.

[yifei-zhhhh]

Hi @yiyione , a contributor has submitted PR1577, which fixes this issue.

Hi, it seems there still exists this problem. I've checked that it's already updated.

File "scripts/get_data.py", line 9, in
fire.Fire(GetData)
File "/root/miniconda3/lib/python3.8/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/root/miniconda3/lib/python3.8/site-packages/fire/core.py", line 475, in _Fire
component, remaining_args = _CallAndUpdateTrace(
File "/root/miniconda3/lib/python3.8/site-packages/fire/core.py", line 691, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "/root/miniconda3/lib/python3.8/site-packages/qlib/tests/data.py", line 189, in qlib_data
if not self.check_dataset(file_name):
File "/root/miniconda3/lib/python3.8/site-packages/qlib/tests/data.py", line 94, in check_dataset
url = self.merge_remote_url(file_name)
File "/root/miniconda3/lib/python3.8/site-packages/qlib/tests/data.py", line 37, in merge_remote_url
token = fernet.decrypt(self.TOKEN).decode()
File "/root/miniconda3/lib/python3.8/site-packages/cryptography/fernet.py", line 75, in decrypt
timestamp, data = Fernet._get_unverified_token_data(token)
File "/root/miniconda3/lib/python3.8/site-packages/cryptography/fernet.py", line 100, in _get_unverified_token_data
utils._check_bytes("token", token)
File "/root/miniconda3/lib/python3.8/site-packages/cryptography/utils.py", line 29, in _check_bytes
raise TypeError("{} must be bytes".format(name))
TypeError: token must be bytes

[SunsetWolf]

I installed the latest version of qlib on my ubuntu system, using python 3.8, using the source installation method, and ran the python get_data.py qlib_data --target_dir ~/.qlib/qlib_data/cn_data --interval 1d --region cn command, and did not find this problem.

[yiyione]

Hi @yiyione , a contributor has submitted PR1577, which fixes this issue.

Thanks, it works for me!

[github-actions]

This issue is stale because it has been open for three months with no activity. Remove the stale label or comment on the issue otherwise this will be closed in 5 days