Postpost compilation in DatasetEvaluator

[rizar]

For debugging it's much better if the compilation in DatasetEvaluator happens when it's first called, not when it's created.

[dmitriy-serdyuk]

Hmm. As for me, it is better to crash sooner (if something is wrong with evaluation) rather than later.

[rizar]

You can then trigger DataStreamMonitoring on before_training. This way you will first do the monitoring and only then compile the training algorithm.

…

On Thu, 30 Mar 2017 at 11:12 dmitriy-serdyuk ***@***.***> wrote: Hmm. As for me, it is better to crash sooner (if something is wrong with evaluation) rather than later. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1185 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAn8YrhA6Gi83gorGUOc0LOMKIw9NFb3ks5rq8ZngaJpZM4Mucvr> .

[dmitriy-serdyuk]

Yes, but evaluation might be very slow. It is always frustrating when model compiles for several minutes, then evaluates for half an hour and then crashes with a first batch of training. And if you interrupt it during evaluation you'll have to wait again for compilation.

I would explicitly compile the evaluator on before_training callback.

[rizar]

Let's discuss this offline. On Thu, 30 Mar 2017 at 11:24 dmitriy-serdyuk <notifications@github.com> wrote: Yes, but evaluation might be very slow. It is always frustrating when model compiles for several minutes, then evaluates for half an hour and then crashes with a first batch of training. And if you interrupt it during evaluation you'll have to wait again for compilation. I would explicitly compile the evaluator on before_training callback. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1185 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAn8YgGDofc4RnlinPR5vSan3GGNe86dks5rq8lAgaJpZM4Mucvr> .

[dmitriy-serdyuk]

LGTM then