-
Notifications
You must be signed in to change notification settings - Fork 19.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fit_generator Segmentation fault #8225
Comments
Did you solve this? I have the same problem (also using fit_generator) but during the epoch, consistently within the first or first 5 epochs.
|
I still got the problem on this version of the script (using tensorflow==1.3.0 and Keras==2.0.8). The solution I ended up with is to stop using |
I just had the exact same problem. The first epoch ends with a segmentation fault : It doesn't happen when I remove the validation_data. With keras 1.2.2 and tensorflow 1.4.0 |
Same problem |
Same problem. @fchollet is there any way this could be fixed? I'm happy to help provide debug info and potentially contribute as needed. Using fit_generator, with generator arguments for both training and validation data. I've tried it with my full dataset (237K rows) and a sample subset of the full dataset (1000 rows), both with ~1K columns, and in both cases the segmentation fault happens right after the first epoch finishes. Like others, if I remove the validation data it doesn't occur. I'm using a train/test split of 85/15 and a batch size of 64 for both the full and sample datasets (so I'm only reading 64 rows from the HDF5 file at any given time, in the generator). Output from Versions: Unlike @dgorissen, I'm experiencing this issue on 2.0.8 as well as 2.1.6. |
I believe I actually just figured out what was causing my personal issue. Not sure if this will apply to others, but in my generator I was using I solved this problem by passing |
I had the same problem with the following code
I fixed the issue with Edit:Actually it does not work every time
|
I experience the same problem (segmentation fault during the first epochs when using The segmentation fault occurs when I run fit_generator on CPUs with a batch size of 40 4/10 [===========>..................] - ETA: 10:59 - loss: 0.4438Segmentation fault (core dumped) Here is a small standalone script that produces the segmentation fault Any idea what is going wrong? I am running:
|
Hi, I just met this symptom in my Docker environment with Keras 2.2.4 and Tensorflow 1.12 (GPU). For me, the issue disappeared when I changed Tensorflow to Environment
Mon Jun 17 17:23:44 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.67 Driver Version: 418.67 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce RTX 208... Off | 00000000:01:00.0 Off | N/A |
| 40% 39C P8 21W / 250W | 403MiB / 10989MiB | 6% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce RTX 208... Off | 00000000:02:00.0 Off | N/A |
| 37% 34C P8 2W / 250W | 1MiB / 10989MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1114 G /usr/lib/xorg/Xorg 224MiB |
| 0 2511 G compiz 79MiB |
| 0 10650 G ...-token=9E760AB97E59CC5C02D0AFC5D37FE54E 98MiB |
+-----------------------------------------------------------------------------+
Installation
FROM tensorflow/tensorflow:1.13.1-gpu-py3 as ship
LABEL maintainer="luncliff@gmail.com"
RUN pip install -qqq --upgrade pip && pip install -qqq keras
RUN pip install -qqq pillow
# ... |
For the larger datasets with Keras multithreading, users needs to adopt a threadsafe generator method to deal with the issue. There is a brief introdcution by Anand Chitipothu as well as the explaination of composed functions by Mathieu Larose. The threadsafe method has been adopted in the library of Faster RCNN by RGB and Kaiming He. threadsafe_code: http://anandology.com/blog/using-iterators-and-generators/ |
Hi,
I use
model.fit_generator
to handle a large dataset.I want to read data by batch from a source file, which I did successfully using a CSV file.
When I want to use
pandas.read_hdf
function, kerasfit_generator
ends-up with a segmentation fault:I already noticed that if I do not use
validation_data
, I don't get the segmentation fault but I don't understand why.Here is a link to the small example I'm running: wine-example
Any help would be very much appreciated.
Cheers,
Ben
The text was updated successfully, but these errors were encountered: