AttributeError: Can't pickle local object 'GeneratorEnqueuer.start.<locals>.data_generator_task'

[yf704475209]

Hi, I meet a problem when I run the train_shapes.ipynb. In the Training block, when I run model.train(dataset_train, dataset_val, learning_rate=config.LEARNING_RATE, epochs=1, layers='heads')
it reports a error, this is it's detail:


I have run the demo successfully, but it seems I can not train my own model. please help if you know anything about my error! Thanks a lot!

[PengTedShen]

@waleedka I encountered this issue too, I know this is the duplicated issue similar as #13
but I am still not quite understood how to solve such issue on windows 10 platform.Thanks

[MatinHz]

You can get rid of this error by changing use_multiprocessing value in line 2202 of model.py to False.

[JPlin]

Does anyone know how to solve this issue?

[erhwenkuo]

@JPlin in order to run in my Windows 10 machine, I have manually modified "model.py" of Mask_RCNN source code at lline#2192。
The origin code block looks like below:

self.keras_model.fit_generator(
            train_generator,
            initial_epoch=self.epoch,
            epochs=epochs,
            steps_per_epoch=self.config.STEPS_PER_EPOCH,
            callbacks=callbacks,
            validation_data=next(val_generator),
            validation_steps=self.config.VALIDATION_STEPS,
            max_queue_size=100,
            workers=max(self.config.BATCH_SIZE // 2, 2),
            use_multiprocessing=True,
        )

The modified code block looks like:

self.keras_model.fit_generator(
            train_generator,
            initial_epoch=self.epoch,
            epochs=epochs,
            steps_per_epoch=self.config.STEPS_PER_EPOCH,
            callbacks=callbacks,
            validation_data=next(val_generator),
            validation_steps=self.config.VALIDATION_STEPS,
            max_queue_size=100,
            workers=1,
            use_multiprocessing=False,
        )

Just like @MatinHz mentioned, I have modifiy "workers=1" and "use_multiprocessing=False"!

Now, you should be able to do the "train_shapes" tutorial.

Please noted that, this is just a temporal fix!

[JPlin]

@erhwenkuo Thanks for your answer.I want to train my own dataset ,and it seems important to use multiprocessing to speed up training.So i am looking forward a more solid solution.

[wjd92]

@JPlin Did you find any other solution?

[xjock]

@erhwenkuo Thanks for your solution, it works!

[philferriere]

@yf704475209 @PengTedShen @waleedka @MatinHz @JPlin @erhwenkuo @wjdhuster2018 @xjock, I have submitted this PR to Keras to get the issue fixed. You may wan to add your vote/voice to the conversation here to get this long-standing Windows issue finally adressed.

[DarylWM]

Thank you for your work on this @philferriere . I read through the discussion but it wasn't clear whether multi-GPU training on Windows will be possible now the PR has been merged?

[philferriere]

@DarylWM The PR address an orthogonal issue (i.e., generating input data/labels on the CPU), not multi-GPU training. I do not know how well multi-GPU training works on Windows. If you have the hardware setup and get to play with it, I would love to know what kind of performance you manage to get. Good luck!

[waleedka]

@philferriere I saw that Keras has merged your PR. Thank you for the fix. Is it accurate to say that this model will now work on Windows without needing any changes?