Add multiprocessing for fit generator

[tdeboissiere]

Implement fit_generator with python multiprocessing (#2730).

Sorry for the spam, I closed my previous PR by accident...
Anyway, this implementation now passes all the checks on Travis, has improved documentation and unit tests.

[fchollet]

maxproc should be nb_worker, for consistency.

[tdeboissiere]

Clarification for lazy-evaluated array:

From what I understand, non-picklable objects can't be passed to children processes easily. This includes lazy evaluated arrays.

So for instance, if you define a generator this way:

arr_data = np.random.randint(0,256, (1000, 3, 224, 224))
arr_labels = np.random.randint(0, 2, 1000)
np.savez("data_dummy", **{"data": arr_data, "labels": arr_labels})

arr = np.load("data_dummy.npz") # arr is non picklable and lazy evaluated 
# (the data is not loaded into memory at this stage).

def myGenerator():

    while True:
        X = arr["data"]
        y = arr["labels"]
        yield (X, y)`

Then arr won't be passed correctly to all the children processes and the code will throw an error (typically the generator will return None).

However, if you load arr in memory before defining the generator or if you define your generator as:

def myGenerator():

    while True:
        arr = np.load("data_dummy.npz")
        X = arr["data"]
        y = arr["labels"]
        yield (X, y)

you won't have this issue.

Catch-all:
I don't need the catch-all, it was merely a convenience when debugging.

Edit: Pushed a new version with your indications

[fchollet]

If providing code examples in a docstring, it should use the MarkDown syntax for code snippets (as used elsewhere in docstring code snippets in the codebase) and the code should follow PEP8 syntax.

Alternatively, you could simply remove the code example, remove the mention of "lazy evaluated arrays", and simply state that the data should be picklable.

[fchollet]

Overall I think this would break a lot of people's code. For instance it would immediately break the cifar10_cnn.py example we have in Keras. It may well break a large majority of all data generators that are currently in use.

I'll think about it some more.

[tdeboissiere]

Sure. In the meantime I could push an example of how to build a generator with multiprocessing (along the line of the gist I posted in the original issue) if you think that is relevant

[carlthome]

I'd like something like this. It's not straightforward how to multi-thread fit_generator, evaluate_generator and predict_generator from outside without introducing an additional queue.

[fchollet]

I think we should consider making it possible to have both the threading and the processing implementation (essentially we just need to change the imports) and have an argument for switching from one to the other (such as pickle_safe, which would default to False for backwards compatibility). How this works should be clearly explained in the documentation.

That way we can benefit from this PR while avoiding breaking everyone's code.

[tboquet]

I'm using Celery and it forks processes on workers (it's possible to control this behavior though). When the model are small enough it's possible to have multiple models running on the same GPU. It would maybe break if multiprocessing is enabled by default. +1 to have both threading and multithreading 👍 .

[tdeboissiere]

I have added multiprocessing as an option with the keyword pickle_safe as mentioned above.

Note that I have reintroduced the deprecated flag when using nb_worker > 1 and pickle_safe = False.

[fchollet]

Note that I have reintroduced the deprecated flag when using nb_worker > 1 and pickle_safe = False.

Sounds good, it's much better to have a unified API.

LGTM, merging. Thanks for the contribution!

[mdeff]

@tdeboissiere thanks for your contribution. Shouldn't we make the pickle_safe limitation clearer in the fit_generator documentation ? I think the example you give above would help a lot. Maybe also warn people that external libraries used for pre-processing may use non-pickable objects.

[alphamupsiomega]

Keras 1.2.2 does have this multiprocessing included, but my fit_generator() is still about 4x slower than fit(). Can this be further sped up?