Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The Keras examples should load data with allow_pickle=True #28102

Closed
vamcily opened this issue Apr 24, 2019 · 14 comments

Comments

Projects
None yet
10 participants
@vamcily
Copy link

commented Apr 24, 2019

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow):No, I was trying the tutorials/keras/basic_text_classification
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS High Sierra 10.13.5
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): 1.13.1
  • Python version: 3.6.2
  • Numpy version: 1.16.3

Describe the current behavior
When I tried the following code from the basic_text_classification sample:

(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)

It failed with error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/tensorflow/python/keras/datasets/imdb.py", line 86, in load_data
    x_train, labels_train = f['x_train'], f['y_train']
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/lib/npyio.py", line 262, in __getitem__
    pickle_kwargs=self.pickle_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/numpy/lib/format.py", line 692, in read_array
    raise ValueError("Object arrays cannot be loaded when "
ValueError: Object arrays cannot be loaded when allow_pickle=False

According to the change of numpy in numpy/numpy#13359, the default value of allow_pickle was changed from True to False.

Please update the sample code accordingly (may pass in allow_pickle=True explicitly to np.load).

Describe the expected behavior
The basic_text_classification example could be followed without error.

Other info / logs
When I downgrade numpy from 1.16.3 to 1.16.1 (whose default value was still allow_pickle=True), I could finish the basic_text_classification example successfully.

@bwalsh0

This comment has been minimized.

Copy link

commented Apr 24, 2019

You need to use np.load(path) with the new parameters np.load(path, allow_pickle), where allow_pickle is a boolean.

This is a recent change in numpy which appears to be for improved source integrity. I encountered the same error as you listed, updating the method call in the respective .py files solves it.

In your case, find imdb.py and change np.load(path) in line 86 to np.load(path, allow_pickle=True)

@vamcily

This comment has been minimized.

Copy link
Author

commented Apr 24, 2019

You need to use np.load(path) with the new parameters np.load(path, allow_pickle), where allow_pickle is a boolean.

This is a recent change in numpy which appears to be for improved source integrity. I encountered the same error as you listed, updating the method signature in the respective .py files solves it.

Good suggestion, I could make this change in my local environment first.
I still suggest the official code could be updated thus other developers could avoid the issue.

@yongtang

This comment has been minimized.

Copy link
Member

commented Apr 25, 2019

@vamcily Think the issue has been fixed in 79a8d5c

@vamcily

This comment has been minimized.

Copy link
Author

commented Apr 26, 2019

@vamcily Think the issue has been fixed in 79a8d5c

Nice! Which release would include this fix?

@gadagashwini gadagashwini assigned ymodak and unassigned gadagashwini Apr 26, 2019

@Alsanabani

This comment has been minimized.

Copy link

commented Apr 26, 2019

I tried the suggested solution but error still there!!

You need to use np.load(path) with the new parameters np.load(path, allow_pickle), where allow_pickle is a boolean.

This is a recent change in numpy which appears to be for improved source integrity. I encountered the same error as you listed, updating the method call in the respective .py files solves it.

In your case, find imdb.py and change np.load(path) in line 86 to np.load(path, allow_pickle=True)

I tried the suggested code change, but the error still there!! what is the problem in your opinion?

@cswangchen

This comment has been minimized.

Copy link

commented Apr 28, 2019

hi,there i use the solution,before i solved it my numpy version is 1.16.3 and then

Downgrate numpy to fix a problem

!pip install numpy==1.16.2
import numpy as np
print(np.version)
it fixed my problem the same as yours!
best luck!

@xiaozeizei

This comment has been minimized.

Copy link

commented Apr 28, 2019

I tried the suggested solution but error still there!!

You need to use np.load(path) with the new parameters np.load(path, allow_pickle), where allow_pickle is a boolean.
This is a recent change in numpy which appears to be for improved source integrity. I encountered the same error as you listed, updating the method call in the respective .py files solves it.
In your case, find imdb.py and change np.load(path) in line 86 to np.load(path, allow_pickle=True)

I tried the suggested code change, but the error still there!! what is the problem in your opinion?

This method does work, however, I meet the same problem as yours.
If you are using Jupyter Notebook ,you need to restart Jupyter Notebook after some changes made in local package file. Running code like "from keras.datasets import imdb" again is useless.

The interesting point is that, in Jupyter, the codes showed as part of errors are real-time when the loaded files are not , then it may make us confused.

@colinwu

This comment has been minimized.

Copy link

commented Apr 28, 2019

@Alsanabani

This comment has been minimized.

Copy link

commented Apr 28, 2019

hi,there i use the solution,before i solved it my numpy version is 1.16.3 and then

Downgrate numpy to fix a problem

!pip install numpy==1.16.2
import numpy as np
print(np.version)
it fixed my problem the same as yours!
best luck!

that finally solved for me, I downgraded numpy version from 1.16.3 to 1.16.1 which default value is still allow_pickle=True.
just remove current version then install the 1.16.1 version via:

pip uninstall numpy
pip install --upgrade numpy==1.16.1

enjoy

@subhobrata

This comment has been minimized.

Copy link

commented Apr 29, 2019

I am still getting the same error in Google Colab

I tried both
pip uninstall numpy
pip install --upgrade numpy==1.16.1

and

!pip install numpy==1.16.2
import numpy as np
print(np.version)

also but still not working, for Google Colab, do we need to do something extra.

(x_train,y_train),(x_valid,y_valid) = imdb.load_data(num_words=n_unique_words)

/usr/local/lib/python3.6/dist-packages/numpy/lib/format.py in read_array(fp, allow_pickle, pickle_kwargs)
690 if pickle_kwargs is None:
691 pickle_kwargs = {}
--> 692 try:
693 array = pickle.load(fp, **pickle_kwargs)
694 except UnicodeError as err:

ValueError: Object arrays cannot be loaded when allow_pickle=False

@colinwu

This comment has been minimized.

Copy link

commented Apr 29, 2019

@subhobrata

This comment has been minimized.

Copy link

commented Apr 29, 2019

yes I restarted the runtime.
For full code, I created a notebook in Github.

Below is the link

https://github.com/subhobrata/Keras_issues/blob/master/Keras_issue.ipynb

@ymodak

This comment has been minimized.

Copy link
Contributor

commented May 1, 2019

keras.datasets.imdb is broken in 1.13 and 1.14, by np 1.16.3. Its fixed in tf-nightly. Thanks!

@ymodak ymodak closed this May 1, 2019

@tensorflow-bot

This comment has been minimized.

Copy link

commented May 1, 2019

Are you satisfied with the resolution of your issue?
Yes
No

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.