Fix loading from HF GCP cache #5321
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
| builder_instance._download_and_prepare = None | ||
| builder_instance.download_and_prepare() | ||
| ds = builder_instance.as_dataset() | ||
| assert ds is not None |
There was a problem hiding this comment.
Are you sure this is a regression test?
Before your fix, this test was not raising the reported MissingBeamOptions error, but a TypeError:
TypeError: 'NoneType' object is not callable
in relation with builder_instance._download_and_prepare = None in your test.
There was a problem hiding this comment.
It should not call builder_instance._download_and_prepare since it should download from the HF cache.
If it raises TypeError, it means it tried to download the dataset normally instead of using the HF cache
It comes from the changes in https://github.com/huggingface/datasets/pull/5107/files#diff-355ae5c229f95f86895404b72378ecd6e966c41cbeebb674af6fe6e9611bc126 |
As reported in https://discuss.huggingface.co/t/error-loading-wikipedia-dataset/26599/4 it's not possible to download a cached version of Wikipedia from the HF GCP cache
I fixed it and added an integration test (runs in 10sec)