adding expert to hp ignore due to pickeling issue

[bonham79]

Fixes: #192

TQDM in the transducer's expert module doesn't pickle currently. So when doing multi-gpu training, pytorch gets annoyed. Since it's not really needed beyond training, I'm dropping it from the model.

Note, when running prediction with the model, the lack of the hyperparameter means a new expert will have to be created. This is dumb, so need to add a dummy class to the transducer model to avoid this. This portion I'm leaving in draft atm.

[Adamits]

Multiprocess + pickle has given me so many issues recently across interdependent libraries...

One edge case: how about continued training from a checkpoint? Then we need the expert right, which I assume has some relevant state?

[kylebgorman]

@Adamits I think this would pose exactly the same issue as the multi-GPU case.

[Adamits]

@kylebgorman Right, I meant an edge case where this solution would break something. If I understand correctly, Travis wants to just not save the expert state to avoid the pickle issue with TQDM. But if we want to continue form a checkpoint, i think we need the expert state.

[tbartley94]

One edge case: how about continued training from a checkpoint? Then we need the expert right, which I assume has some relevant state?

you'd have to train the expert again. thankfully your np fix makes it less annoying. but yeah, that's why i was curious if there were better ideas.

fwiw, the entire point behind the expert transducer is for single gpu setup. (Actually it works better on CPU for most cases due to the expert queries). You don't get really parallelism speed ups since there's so many gpu-> cpu transfers.

[kylebgorman]

Yeah I also find the transducer is better on CPU. (The same was true with the original code in DyNet.)

[bonham79]

Yeah I also find the transducer is better on CPU. (The same was true with the original code in DyNet.)

Yeah it's the main motivation for its use so not surprised.