con not save mode

[cyqluowang]

with tf.train.MonitoredTrainingSession:

            if sess.should_stop():
                tf_feed.terminate()

            if is_chief:
                signatures = {
                    tf.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY: {
                        'inputs': {'image': input_x},
                        'outputs': {'prediction': pred},
                        'method_name': tf.saved_model.signature_constants.PREDICT_METHOD_NAME
                    }
                }
                TFNode.export_saved_model(sess, args.serving,
                                          tf.saved_model.tag_constants.SERVING,
                                          signatures)

no error

log:
2018-07-02 11:53:54,523 INFO (MainThread-31539) No assets to save.
2018-07-02 11:53:54,523 INFO (MainThread-31539) No assets to write.

The program worked well and the training was complete

My guess is that the program ended too quickly to write data

[cyqluowang]

MonitoredTrainingSession
cluster = TFCluster.run(sc, soft_dist.map_fun, args, args.cluster_size, num_ps, args.tensorboard, TFCluster.InputMode.SPARK)

[leewyang]

My guess is that the program ended too quickly to write data

Yes, that might be the issue. There is a slight race condition at the end (for InputMode.SPARK), where the data feeding job is done (and therefore Spark starts the shutdown process), but the chief worker is still exporting the model. A simple workaround is to add a sleep statement just before the TFCluster.shutdown() method in your driver.

[cyqluowang]

I tried, but I couldn't achieve my goal

[cyqluowang]

Am I using it somewhere wrong?

[leewyang]

This should work... although I'm seeing some weird behavior in local my tests w.r.t. step condition. Can you try setting your max steps to be less than your expected number of steps?

[cyqluowang]

Thank you for your reply. I'm actually running less than expected, every time will run

            if sess.should_stop():
                tf_feed.terminate()

The first line of the log above has the call log。
I think you might want to try num-executors >2. may be 4 or 6
thk

[leewyang]

@cyqluowang please take a look at #312 to see if it resolves your issue.

[cyqluowang]

it works.

Thank you very much for your advice and reply

[leewyang]

Glad it worked! It's now merged into master.