@fchollet @yifeif May be related to #14504

@yifeif could you please take a look.

@ispirmustafa any idea? The run_config is passed in directly when creating the keras version of the Estimator. Do we need to pass these configurations anywhere else?

It should not be related with Keras code. It should work since it is handled within Estimator.
@droidicus could you please print est_keras.config and est_keras.config.cluster_spec.as_dict?

Sure thing, here is the output:

*********** est_keras.config *************************************
<tensorflow.python.estimator.run_config.RunConfig object at 0x7f7694423fd0>
*********** est_keras.config.cluster_spec.as_dict()  *************
{}
******************************************************************

Code to reproduce (same as the source in the origional issue above, but with print statements after the creation of the keras estimator), and full log output are avaliable here: https://gist.github.com/droidicus/146532eacf88ed57538bb41a8fc7da4b

It has been 14 days with no activity and this issue has an assignee.Please update the label and/or status accordingly.

Gentle ping, this is still an issue for me.

Hi @shivaniag,
Estimator sends the given session_config directly to the Session constructor. Could you please assign somebody who is more familiar with tf.Session and it's handling of GPU settings?

FYI, I've checked the keras.model_to_estimator. It's sending the config properly to tf.estimator.Estimator.
tf.estimator.Estimator sends that config to tf.train.SessionManager calls which uses it as a constructor argument to tf.Session.

Just as an FYI, while this is still a problem in TFv1.5rc0 we were able to do the following as a workaround for now, by setting the default session manually the memory fraction is respected:

import os
import numpy as np
import tensorflow as tf

tf.logging.set_verbosity(tf.logging.INFO)

gpu_options = tf.GPUOptions(per_process_gpu_memory_fraction=0.5)
sess_config = tf.ConfigProto(gpu_options=gpu_options)
# Manually set the default session instead
tf.Session(config=sess_config).as_default()
#run_config = tf.estimator.RunConfig(session_config=sess_config)

inputs = tf.keras.layers.Input(shape=(10,))
outputs = tf.keras.layers.Dense(10)(inputs)
model = tf.keras.models.Model(inputs, outputs)
model.compile(optimizer='sgd', loss='mse')
est_keras = tf.keras.estimator.model_to_estimator(keras_model=model)#, config=run_config)

input_name = model.input_names[0]
data = np.random.rand(1000,10).astype(np.float32)
train_input_fn = tf.estimator.inputs.numpy_input_fn({input_name:data}, data, batch_size=10, num_epochs=None, shuffle=False)

train_spec = tf.estimator.TrainSpec(input_fn=train_input_fn, max_steps=100000)
eval_spec = tf.estimator.EvalSpec(input_fn=train_input_fn, steps=10)
tf.estimator.train_and_evaluate(est_keras, train_spec, eval_spec)

@yifeif, @fchollet is there any place within underlying Keras code that uses default session while building the graph, train_op, ...?

A member of the TensorFlow organization has replied after the stat:awaiting tensorflower label was applied.

Nagging Assignee: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Sorry for the delay. This is how model_to_estimator create its model_fn. @ispirmustafa anything you see with session that should be done differently? Thanks!

Also tried print out estimator.config.__dict__ and estimator.config.cluster_spec.__dict__ for canned estimator, custom estimator and keras converted estimator, and I'm seeing the same results:
`
*********** est_keras.config.dict *************************************
{'_save_checkpoints_secs': 600, '_session_config': gpu_options {
per_process_gpu_memory_fraction: 0.5
}
, '_keep_checkpoint_max': 5, '_task_type': 'worker', '_is_chief': True, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f8e3ebcabd0>, '_save_checkpoints_steps': None, '_keep_checkpoint_every_n_hours': 10000, '_service': None, '_num_ps_replicas': 0, '_tf_random_seed': None, '_master': '', '_num_worker_replicas': 1, '_task_id': 0, '_log_step_count_steps': 100, '_model_dir': '/tmp/tmp8jvdNt', '_save_summary_steps': 100}
*********** est_keras.config.cluster_spec.dict *************
{'_cluster_def': , '_cluster_spec': {}}

`

Nagging Assignees @yifeif, @shivaniag, @ispirmustafa: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

A fix has been submitted internally and should make to master tomorrow. Thanks!

Fantastic, thanks!