Eval gets stuck forever in the Trainer Component #113

benjamintanweihao · 2019-05-13T09:52:13Z

When I'm in the Trainer component, eval gets stuck forever:

[2019-05-13 17:40:53,499] {logging_mixin.py:95} INFO - [2019-05-13 17:40:53,499] {saver.py:1270} INFO - Restoring parameters from /home/benjamintan/workspace/darkrai/logs/shapes_768_1024_20190513T1740/model.ckpt-100
[2019-05-13 17:40:54,151] {logging_mixin.py:95} INFO - [2019-05-13 17:40:54,151] {session_manager.py:491} INFO - Running local_init_op.
[2019-05-13 17:40:54,198] {logging_mixin.py:95} INFO - [2019-05-13 17:40:54,198] {session_manager.py:493} INFO - Done running local_init_op.

Strangely, if I replace the path eval example path with the training example path, it manages to make progress to the model validator (though it fails model validation).

Any pointers on how to debug this?

The text was updated successfully, but these errors were encountered:

1025KB · 2019-05-13T18:07:27Z

Hi, are you running the taxi example or your own one?

benjamintanweihao · 2019-05-13T23:56:45Z

I’m using my own one.

…

On 14 May 2019, at 2:07 AM, Jiayi Zhao ***@***.***> wrote: Hi, are you running the taxi example or your own one? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

1025KB · 2019-05-14T17:48:33Z

SchemaGen generate schema based on train split and ExampleValidator validates eval split, could you check schemagen and examplevalidator's output to see if there is anything wrong in the data?

benjamintanweihao · 2019-05-15T07:11:56Z

Thanks for the pointer! It seems like the data was generated correctly:

ruoyu90 · 2019-05-15T22:15:24Z

@benjamintanweihao Can you attach the log for the trainer component? Assuming you're using Airflow (from your last reply), you can find the log under $AIRFLOW_HOME/logs/<your_pipeline_name>.trainer

benjamintanweihao · 2019-05-16T05:42:16Z

@ruoyu90 Yes, I'm using Airflow. Here you go:

[2019-05-15 15:41:02,007] {__init__.py:1139} INFO - Dependencies all met for <TaskInstance: darkrai.Trainer.darkrai.Trainer.exec 2019-05-15T07:32:22.181045+00:00 [queued]>
[2019-05-15 15:41:02,016] {__init__.py:1139} INFO - Dependencies all met for <TaskInstance: darkrai.Trainer.darkrai.Trainer.exec 2019-05-15T07:32:22.181045+00:00 [queued]>
[2019-05-15 15:41:02,016] {__init__.py:1353} INFO - 
--------------------------------------------------------------------------------
[2019-05-15 15:41:02,016] {__init__.py:1354} INFO - Starting attempt 1 of 1
[2019-05-15 15:41:02,016] {__init__.py:1355} INFO - 
--------------------------------------------------------------------------------
[2019-05-15 15:41:02,035] {__init__.py:1374} INFO - Executing <Task(PythonOperator): darkrai.Trainer.exec> on 2019-05-15T07:32:22.181045+00:00
[2019-05-15 15:41:02,035] {base_task_runner.py:119} INFO - Running: ['airflow', 'run', 'darkrai.Trainer', 'darkrai.Trainer.exec', '2019-05-15T07:32:22.181045+00:00', '--job_id', '1566', '--raw', '-sd', 'DAGS_FOLDER/darkrai_pipeline.py', '--cfg_path', '/tmp/tmp0sn_gsdf']
[2019-05-15 15:41:03,116] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [2019-05-15 15:41:03,115] {__init__.py:51} INFO - Using executor SequentialExecutor
[2019-05-15 15:41:03,517] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [2019-05-15 15:41:03,515] {__init__.py:305} INFO - Filling up the DagBag from /home/benjamintan/workspace/darkrai/pipeline/dags/darkrai_pipeline.py
[2019-05-15 15:41:08,891] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [2019-05-15 15:41:08,890] {settings.py:50} WARNING - Path for coco pretrained h5 could not be found...
[2019-05-15 15:41:08,891] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [2019-05-15 15:41:08,891] {settings.py:53} WARNING - DATA dirpath PATH_TO_THE_DATA_DIR could not be found !
[2019-05-15 15:41:08,891] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [2019-05-15 15:41:08,891] {settings.py:56} WARNING - DATA_OTHER dirpath PATH_TO_THE_DATA_DIR could not be found !
[2019-05-15 15:41:08,897] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [2019-05-15 15:41:08,896] {deprecation.py:323} WARNING - From /home/benjamintan/workspace/darkrai/pipeline/dags/darkrai_pipeline.py:48: PipelineDecorator (from tfx.orchestration.pipeline) is deprecated and will be removed in a future version.
[2019-05-15 15:41:08,897] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec Instructions for updating:
[2019-05-15 15:41:08,897] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec PipelineDecorator is no longer needed. Please construct a pipeline directly from a list of components  using the constructor call to pipeline.Pipeline.
[2019-05-15 15:41:08,923] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec <tfx.components.base.base_component.ComponentOutputs object at 0x7f79a3ae3f28>
[2019-05-15 15:41:08,924] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [2019-05-15 15:41:08,923] {cli.py:517} INFO - Running <TaskInstance: darkrai.Trainer.darkrai.Trainer.exec 2019-05-15T07:32:22.181045+00:00 [running]> on host dnnserver2
[2019-05-15 15:41:08,935] {python_operator.py:104} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_ID=darkrai.Trainer
AIRFLOW_CTX_TASK_ID=darkrai.Trainer.exec
AIRFLOW_CTX_EXECUTION_DATE=2019-05-15T07:32:22.181045+00:00
AIRFLOW_CTX_DAG_RUN_ID=backfill_2019-05-15T07:32:22.181045+00:00
[2019-05-15 15:41:08,949] {logging_mixin.py:95} INFO - [2019-05-15 15:41:08,949] {base_executor.py:72} INFO - Starting Executor execution.
[2019-05-15 15:41:08,951] {logging_mixin.py:95} INFO - [2019-05-15 15:41:08,951] {base_executor.py:74} INFO - Inputs for Executor is: {"transformed_examples": [{"artifact": {"id": "11", "typeId": "2", "uri": "/home/benjamintan/tfx/pipelines/darkrai/Transform/transformed_examples/11/train/", "properties": {"split": {"stringValue": "train"}, "state": {"stringValue": "published"}, "span": {"intValue": "0"}, "type_name": {"stringValue": "ExamplesPath"}}}, "artifact_type": {"name": "ExamplesPath", "properties": {"span": "INT", "name": "STRING", "type_name": "STRING", "split": "STRING", "state": "STRING"}}}, {"artifact": {"id": "12", "typeId": "2", "uri": "/home/benjamintan/tfx/pipelines/darkrai/Transform/transformed_examples/11/eval/", "properties": {"type_name": {"stringValue": "ExamplesPath"}, "split": {"stringValue": "eval"}, "state": {"stringValue": "published"}, "span": {"intValue": "0"}}}, "artifact_type": {"name": "ExamplesPath", "properties": {"state": "STRING", "span": "INT", "name": "STRING", "type_name": "STRING", "split": "STRING"}}}], "transform_output": [{"artifact": {"id": "10", "typeId": "8", "uri": "/home/benjamintan/tfx/pipelines/darkrai/Transform/transform_output/11/", "properties": {"type_name": {"stringValue": "TransformPath"}, "split": {"stringValue": ""}, "state": {"stringValue": "published"}, "span": {"intValue": "0"}}}, "artifact_type": {"name": "TransformPath", "properties": {"state": "STRING", "span": "INT", "name": "STRING", "type_name": "STRING", "split": "STRING"}}}], "schema": [{"artifact": {"id": "9", "typeId": "6", "uri": "/home/benjamintan/tfx/pipelines/darkrai/SchemaGen/output/10/", "properties": {"type_name": {"stringValue": "SchemaPath"}, "split": {"stringValue": ""}, "state": {"stringValue": "published"}, "span": {"intValue": "0"}}}, "artifact_type": {"name": "SchemaPath", "properties": {"name": "STRING", "type_name": "STRING", "split": "STRING", "state": "STRING", "span": "INT"}}}]}
[2019-05-15 15:41:08,951] {logging_mixin.py:95} INFO - [2019-05-15 15:41:08,951] {base_executor.py:76} INFO - Outputs for Executor is: {"output": [{"artifact": {"uri": "/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/32/", "properties": {"span": {"intValue": "0"}, "type_name": {"stringValue": "ModelExportPath"}, "split": {"stringValue": ""}}}, "artifact_type": {"name": "ModelExportPath", "properties": {"split": "STRING", "state": "STRING", "span": "INT", "name": "STRING", "type_name": "STRING"}}}]}
[2019-05-15 15:41:08,951] {logging_mixin.py:95} INFO - [2019-05-15 15:41:08,951] {base_executor.py:78} INFO - Execution properties for Executor is: {"train_args": "{\n  \"numSteps\": 10\n}", "eval_args": "{\n  \"numSteps\": 2\n}", "module_file": "/home/benjamintan/workspace/darkrai/pipeline/components/trainer/trainer.py", "custom_config": null}
[2019-05-15 15:41:10,363] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec 2019-05-15 15:41:10.363436: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
[2019-05-15 15:41:10,363] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec 2019-05-15 15:41:10.363519: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:161] retrieving CUDA diagnostic information for host: dnnserver2
[2019-05-15 15:41:10,364] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec 2019-05-15 15:41:10.363542: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:168] hostname: dnnserver2
[2019-05-15 15:41:10,364] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec 2019-05-15 15:41:10.363648: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:192] libcuda reported version is: 410.104.0
[2019-05-15 15:41:10,364] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec 2019-05-15 15:41:10.363701: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:196] kernel reported version is: 410.104.0
[2019-05-15 15:41:10,364] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec 2019-05-15 15:41:10.363716: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:303] kernel version seems to match DSO: 410.104.0
[2019-05-15 15:41:10,400] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec 2019-05-15 15:41:10.400318: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 1995350000 Hz
[2019-05-15 15:41:10,405] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec 2019-05-15 15:41:10.405300: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x55b6488b18e0 executing computations on platform Host. Devices:
[2019-05-15 15:41:10,405] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec 2019-05-15 15:41:10.405337: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
[2019-05-15 15:41:10,406] {logging_mixin.py:95} INFO - 
Configurations:
[2019-05-15 15:41:10,407] {logging_mixin.py:95} INFO - AUGMENTATOR                    None
[2019-05-15 15:41:10,407] {logging_mixin.py:95} INFO - BACKBONE                       BackboneType.RESNET_50
[2019-05-15 15:41:10,407] {logging_mixin.py:95} INFO - BATCH_SIZE                     1
[2019-05-15 15:41:10,407] {logging_mixin.py:95} INFO - DATASET_FILTER_TYPES           DatasetFilterType.NONE
[2019-05-15 15:41:10,407] {logging_mixin.py:95} INFO - DECODERS                       {'fcn': <libs.perception.head.segmentation_config.SegmentationConfig object at 0x7f79a3a68eb8>}
[2019-05-15 15:41:10,407] {logging_mixin.py:95} INFO - GPU_COUNT                      1
[2019-05-15 15:41:10,407] {logging_mixin.py:95} INFO - GRADIENT_CLIP_NORM             5.0
[2019-05-15 15:41:10,407] {logging_mixin.py:95} INFO - IMAGES_PER_GPU                 1
[2019-05-15 15:41:10,407] {logging_mixin.py:95} INFO - IMAGE_MAX_DIM                  1024
[2019-05-15 15:41:10,407] {logging_mixin.py:95} INFO - IMAGE_MIN_DIM                  768
[2019-05-15 15:41:10,408] {logging_mixin.py:95} INFO - IMAGE_MIN_SCALE                0
[2019-05-15 15:41:10,408] {logging_mixin.py:95} INFO - IMAGE_RESIZE_MODE              custom
[2019-05-15 15:41:10,408] {logging_mixin.py:95} INFO - IMAGE_SHAPE                    [ 768 1024    3]
[2019-05-15 15:41:10,408] {logging_mixin.py:95} INFO - INIT_WITH                      imagenet
[2019-05-15 15:41:10,408] {logging_mixin.py:95} INFO - LEARNING_MOMENTUM              0.9
[2019-05-15 15:41:10,408] {logging_mixin.py:95} INFO - LEARNING_RATE                  0.001
[2019-05-15 15:41:10,408] {logging_mixin.py:95} INFO - MAPPING                        {<Label.BACKGROUND: ('BG', [0, 0, 0], 'BG')>: [<Label.BACKGROUND: ('BG', [0, 0, 0], 'BG')>], <Label.TWO_WHEELERS: ('vehicle--two-wheelers', [0, 0, 230], '2-wh')>: [<Label.BICYCLE: ('vehicle--bicycle', [119, 11, 32], 'bike')>, <Label.MOTORCYCLE: ('vehicle--motorcycle', [0, 0, 230], 'moto')>, <Label.RIDER: ('human--rider', [255, 0, 0], 'rider')>], <Label.CAR: ('vehicle--car', [0, 0, 142], 'car')>: [<Label.CAR: ('vehicle--car', [0, 0, 142], 'car')>], <Label.LARGE_VEHICLE: ('vehicle--large', [0, 0, 70], 'big')>: [<Label.BUS: ('vehicle--bus', [0, 60, 100], 'bus')>, <Label.TRUCK: ('vehicle--truck', [0, 0, 70], 'truck')>, <Label.OTHER_VEHICLE: ('vehicle--other_vehicle', [128, 64, 64], 'other-vhc')>], <Label.PEDESTRIAN: ('human--pedestrian', [220, 20, 60], 'ped')>: [<Label.PEDESTRIAN: ('human--pedestrian', [220, 20, 60], 'ped')>]}
[2019-05-15 15:41:10,408] {logging_mixin.py:95} INFO - MAPPING_SEGMENTATION           {<Label.BACKGROUND: ('BG', [0, 0, 0], 'BG')>: [<Label.BACKGROUND: ('BG', [0, 0, 0], 'BG')>], <Label.TWO_WHEELERS: ('vehicle--two-wheelers', [0, 0, 230], '2-wh')>: [<Label.BICYCLE: ('vehicle--bicycle', [119, 11, 32], 'bike')>, <Label.MOTORCYCLE: ('vehicle--motorcycle', [0, 0, 230], 'moto')>, <Label.RIDER: ('human--rider', [255, 0, 0], 'rider')>], <Label.CAR: ('vehicle--car', [0, 0, 142], 'car')>: [<Label.CAR: ('vehicle--car', [0, 0, 142], 'car')>], <Label.LARGE_VEHICLE: ('vehicle--large', [0, 0, 70], 'big')>: [<Label.BUS: ('vehicle--bus', [0, 60, 100], 'bus')>, <Label.TRUCK: ('vehicle--truck', [0, 0, 70], 'truck')>, <Label.OTHER_VEHICLE: ('vehicle--other_vehicle', [128, 64, 64], 'other-vhc')>], <Label.PEDESTRIAN: ('human--pedestrian', [220, 20, 60], 'ped')>: [<Label.PEDESTRIAN: ('human--pedestrian', [220, 20, 60], 'ped')>], <Label.FREESPACE: ('freespace', [128, 64, 128], 'FS')>: [<Label.FREESPACE: ('freespace', [128, 64, 128], 'FS')>]}
[2019-05-15 15:41:10,409] {logging_mixin.py:95} INFO - MAX_GT_INSTANCES               100
[2019-05-15 15:41:10,409] {logging_mixin.py:95} INFO - MEAN_PIXEL                     [123.7 116.8 103.9]
[2019-05-15 15:41:10,409] {logging_mixin.py:95} INFO - MINI_MASK_SHAPE                (56, 56)
[2019-05-15 15:41:10,409] {logging_mixin.py:95} INFO - NAME                           shapes_768_1024_
[2019-05-15 15:41:10,409] {logging_mixin.py:95} INFO - NAME_PREFIX                    shapes
[2019-05-15 15:41:10,409] {logging_mixin.py:95} INFO - NORMALIZATION_TYPE             NormalizationType.BATCH
[2019-05-15 15:41:10,409] {logging_mixin.py:95} INFO - PRE_PROCESS_IMAGES             PreProcessImage.NONE
[2019-05-15 15:41:10,410] {logging_mixin.py:95} INFO - SCHEDULE                       [(15, <OptimizerType.SGD: 'sgd'>, 0.001, 'heads'), (25, <OptimizerType.SGD: 'sgd'>, 0.0002, '4+'), (35, <OptimizerType.SGD: 'sgd'>, 5e-05, 'all')]
[2019-05-15 15:41:10,410] {logging_mixin.py:95} INFO - STEPS_PER_EPOCH                10
[2019-05-15 15:41:10,410] {logging_mixin.py:95} INFO - TRAIN_BN                       False
[2019-05-15 15:41:10,410] {logging_mixin.py:95} INFO - UNCERTAINTY_WEIGHTED_LOSSES    False
[2019-05-15 15:41:10,410] {logging_mixin.py:95} INFO - USE_MINI_MASK                  True
[2019-05-15 15:41:10,410] {logging_mixin.py:95} INFO - USE_TF_PROFILER                False
[2019-05-15 15:41:10,410] {logging_mixin.py:95} INFO - VALIDATION_STEPS               2
[2019-05-15 15:41:10,410] {logging_mixin.py:95} INFO - WEIGHT_DECAY                   0.0001
[2019-05-15 15:41:10,410] {logging_mixin.py:95} INFO - 
[2019-05-15 15:41:10,410] {logging_mixin.py:95} INFO - 
[2019-05-15 15:41:10,420] {logging_mixin.py:95} INFO - [('eval_files', '/home/benjamintan/tfx/pipelines/darkrai/Transform/transformed_examples/11/eval/*'), ('eval_steps', 2), ('output_dir', '/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/32/'), ('schema_file', '/home/benjamintan/tfx/pipelines/darkrai/SchemaGen/output/10/schema.pbtxt'), ('serving_model_dir', '/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/32/serving_model_dir'), ('train_files', ['/home/benjamintan/tfx/pipelines/darkrai/Transform/transformed_examples/11/train/*']), ('train_steps', 10), ('transform_output', '/home/benjamintan/tfx/pipelines/darkrai/Transform/transform_output/11/'), ('warm_start_from', None)]
[2019-05-15 15:41:10,529] {logging_mixin.py:95} INFO - [2019-05-15 15:41:10,529] {deprecation.py:323} WARNING - From /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/ops/resource_variable_ops.py:435: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
[2019-05-15 15:41:19,935] {logging_mixin.py:95} INFO - **** HYPERPARAMETERS ****
[2019-05-15 15:41:19,936] {logging_mixin.py:95} INFO - [('eval_files', '/home/benjamintan/tfx/pipelines/darkrai/Transform/transformed_examples/11/eval/*'), ('eval_steps', 2), ('output_dir', '/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/32/'), ('schema_file', '/home/benjamintan/tfx/pipelines/darkrai/SchemaGen/output/10/schema.pbtxt'), ('serving_model_dir', '/home/benjamintan/tfx/pipelines/darkrai/Trainer/output/32/serving_model_dir'), ('train_files', ['/home/benjamintan/tfx/pipelines/darkrai/Transform/transformed_examples/11/train/*']), ('train_steps', 10), ('transform_output', '/home/benjamintan/tfx/pipelines/darkrai/Transform/transform_output/11/'), ('warm_start_from', None)]
[2019-05-15 15:41:19,937] {logging_mixin.py:95} INFO - [2019-05-15 15:41:19,937] {run_config.py:532} INFO - Initializing RunConfig with distribution strategies.
[2019-05-15 15:41:19,937] {logging_mixin.py:95} INFO - [2019-05-15 15:41:19,937] {estimator_training.py:166} INFO - Not using Distribute Coordinator.
[2019-05-15 15:41:19,938] {logging_mixin.py:95} INFO - [2019-05-15 15:41:19,938] {estimator.py:201} INFO - Using config: {'_model_dir': '/home/benjamintan/workspace/darkrai/logs/shapes_768_1024_20190515T1541', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': allow_soft_placement: true
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 50, '_train_distribute': <tensorflow.contrib.distribute.python.one_device_strategy.OneDeviceStrategy object at 0x7f7a3f60f048>, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f799d185b38>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1, '_distribute_coordinator_mode': None}
[2019-05-15 15:41:19,939] {logging_mixin.py:95} INFO - [2019-05-15 15:41:19,939] {executor.py:139} INFO - Training model.
[2019-05-15 15:41:19,940] {logging_mixin.py:95} INFO - [2019-05-15 15:41:19,940] {estimator_training.py:185} INFO - Not using Distribute Coordinator.
[2019-05-15 15:41:19,940] {logging_mixin.py:95} INFO - [2019-05-15 15:41:19,940] {training.py:610} INFO - Running training and evaluation locally (non-distributed).
[2019-05-15 15:41:19,941] {logging_mixin.py:95} INFO - [2019-05-15 15:41:19,941] {training.py:698} INFO - Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs None.
[2019-05-15 15:41:19,942] {logging_mixin.py:95} INFO - **** TRAIN INPUT FN ****
[2019-05-15 15:41:19,942] {logging_mixin.py:95} INFO - ['/home/benjamintan/tfx/pipelines/darkrai/Transform/transformed_examples/11/train/*']
[2019-05-15 15:41:19,942] {logging_mixin.py:95} INFO - <tensorflow_transform.output_wrapper.TFTransformOutput object at 0x7f79c84c2e80>
[2019-05-15 15:41:19,944] {logging_mixin.py:95} INFO - Before build_gt_data_dataset
[2019-05-15 15:41:20,593] {logging_mixin.py:95} INFO - Before filter
[2019-05-15 15:41:20,608] {logging_mixin.py:95} INFO - After filter
[2019-05-15 15:41:20,610] {logging_mixin.py:95} INFO - <DatasetV1Adapter shapes: {images: (?, ?, 3), image_meta: (12,), image_rel_path: (), gt_class_ids: (?,), gt_boxes: (?, 4), loss_activation: (2,), gt_instance_masks: (?, ?, ?), gt_seg_masks: (?, ?, 6)}, types: {images: tf.uint8, image_meta: tf.float32, image_rel_path: tf.string, gt_class_ids: tf.int32, gt_boxes: tf.int32, loss_activation: tf.bool, gt_instance_masks: tf.bool, gt_seg_masks: tf.bool}>
[2019-05-15 15:41:20,663] {logging_mixin.py:95} INFO - BATCH SIZE
[2019-05-15 15:41:20,663] {logging_mixin.py:95} INFO - 1
[2019-05-15 15:41:20,667] {logging_mixin.py:95} INFO - Before Map
[2019-05-15 15:41:20,673] {logging_mixin.py:95} INFO - [2019-05-15 15:41:20,673] {deprecation.py:323} WARNING - From /home/benjamintan/workspace/darkrai/libs/perception/dataset/tf_dataset.py:170: Print (from tensorflow.python.ops.logging_ops) is deprecated and will be removed after 2018-08-20.
Instructions for updating:
Use tf.print instead of tf.Print. Note that tf.print returns a no-output operator that directly prints the output. Outside of defuns or eager mode, this operator will not be executed unless it is directly specified in session.run or used as a control dependency for other operators. This is only a concern in graph mode. Below is an example of how to ensure tf.print executes in graph mode:
```python
    sess = tf.Session()
    with sess.as_default():
        tensor = tf.range(10)
        print_op = tf.print(tensor)
        with tf.control_dependencies([print_op]):
          out = tf.add(tensor, tensor)
        sess.run(out)
    ```
Additionally, to use tf.print in python 2.7, users must make sure to import
the following:

  `from __future__ import print_function`
[2019-05-15 15:41:20,673] {logging_mixin.py:95} INFO - 
[2019-05-15 15:41:20,685] {logging_mixin.py:95} INFO - After Map
[2019-05-15 15:41:20,686] {logging_mixin.py:95} INFO - <DatasetV1Adapter shapes: ({images: (?, ?, ?, 3), image_meta: (?, 12), gt_seg_masks: (?, ?, ?, 6)}, {images: (?, ?, ?, 3), image_meta: (?, 12), image_rel_path: (?,), loss_activation: (?, 2), gt_seg_masks: (?, ?, ?, 6)}), types: ({images: tf.uint8, image_meta: tf.float32, gt_seg_masks: tf.bool}, {images: tf.uint8, image_meta: tf.float32, image_rel_path: tf.string, loss_activation: tf.bool, gt_seg_masks: tf.bool})>
[2019-05-15 15:41:20,840] {logging_mixin.py:95} INFO - [2019-05-15 15:41:20,839] {estimator.py:1111} INFO - Calling model_fn.
[2019-05-15 15:41:24,914] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,914] {estimator.py:93} INFO - Variables to be trained:
[2019-05-15 15:41:24,914] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,914] {estimator.py:95} INFO -     scene_model/model/fcn_score_fr/kernel:0
[2019-05-15 15:41:24,915] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,915] {estimator.py:95} INFO -     scene_model/model/fcn_score_fr/bias:0
[2019-05-15 15:41:24,915] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,915] {estimator.py:95} INFO -     scene_model/model/fcn_upsampling_upscore2/fcn_upsampling_upscore2/kernel:0
[2019-05-15 15:41:24,915] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,915] {estimator.py:95} INFO -     scene_model/model/fcn_upsampling_upscore2/fcn_upsampling_upscore2/bias:0
[2019-05-15 15:41:24,915] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,915] {estimator.py:95} INFO -     scene_model/model/fcn_score_feed2/kernel:0
[2019-05-15 15:41:24,915] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,915] {estimator.py:95} INFO -     scene_model/model/fcn_score_feed2/bias:0
[2019-05-15 15:41:24,915] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,915] {estimator.py:95} INFO -     scene_model/model/fcn_upsampling_upscore4/fcn_upsampling_upscore4/kernel:0
[2019-05-15 15:41:24,915] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,915] {estimator.py:95} INFO -     scene_model/model/fcn_upsampling_upscore4/fcn_upsampling_upscore4/bias:0
[2019-05-15 15:41:24,916] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,916] {estimator.py:95} INFO -     scene_model/model/fcn_score_feed4/kernel:0
[2019-05-15 15:41:24,916] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,916] {estimator.py:95} INFO -     scene_model/model/fcn_score_feed4/bias:0
[2019-05-15 15:41:24,916] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,916] {estimator.py:95} INFO -     scene_model/model/fcn_upsampling_upscore32/fcn_upsampling_upscore32/kernel:0
[2019-05-15 15:41:24,916] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,916] {estimator.py:95} INFO -     scene_model/model/fcn_upsampling_upscore32/fcn_upsampling_upscore32/bias:0
[2019-05-15 15:41:24,980] {logging_mixin.py:95} INFO - [2019-05-15 15:41:24,980] {deprecation.py:323} WARNING - From /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
[2019-05-15 15:41:25,628] {logging_mixin.py:95} INFO - [2019-05-15 15:41:25,628] {basic_session_run_hooks.py:527} INFO - Create CheckpointSaverHook.
[2019-05-15 15:41:25,630] {logging_mixin.py:95} INFO - [2019-05-15 15:41:25,630] {estimator.py:1113} INFO - Done calling model_fn.
[2019-05-15 15:41:26,157] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,156] {estimator.py:1294} INFO - Warm-starting with WarmStartSettings: WarmStartSettings(ckpt_to_initialize_from='/home/benjamintan/workspace/darkrai/logs/shapes_768_1024_20190515T1541/init/', vars_to_warm_start=['scene_model/res_net/conv1/kernel:0', 'scene_model/res_net/conv1/bias:0', 'scene_model/res_net/bn_conv1/gamma:0', 'scene_model/res_net/bn_conv1/beta:0', 'scene_model/res_net/private__conv_block/res2a_branch2a/kernel:0', 'scene_model/res_net/private__conv_block/res2a_branch2a/bias:0', 'scene_model/res_net/private__conv_block/bn2a_branch2a/gamma:0', 'scene_model/res_net/private__conv_block/bn2a_branch2a/beta:0', 'scene_model/res_net/private__conv_block/res2a_branch2b/kernel:0', 'scene_model/res_net/private__conv_block/res2a_branch2b/bias:0', 'scene_model/res_net/private__conv_block/bn2a_branch2b/gamma:0', 'scene_model/res_net/private__conv_block/bn2a_branch2b/beta:0', 'scene_model/res_net/private__conv_block/res2a_branch2c/kernel:0', 'scene_model/res_net/private__conv_block/res2a_branch2c/bias:0', 'scene_model/res_net/private__conv_block/bn2a_branch2c/gamma:0', 'scene_model/res_net/private__conv_block/bn2a_branch2c/beta:0', 'scene_model/res_net/private__conv_block/res2a_branch1/kernel:0', 'scene_model/res_net/private__conv_block/res2a_branch1/bias:0', 'scene_model/res_net/private__conv_block/bn2a_branch1/gamma:0', 'scene_model/res_net/private__conv_block/bn2a_branch1/beta:0', 'scene_model/res_net/private__identity_block/res2b_branch2a/kernel:0', 'scene_model/res_net/private__identity_block/res2b_branch2a/bias:0', 'scene_model/res_net/private__identity_block/bn2b_branch2a/gamma:0', 'scene_model/res_net/private__identity_block/bn2b_branch2a/beta:0', 'scene_model/res_net/private__identity_block/res2b_branch2b/kernel:0', 'scene_model/res_net/private__identity_block/res2b_branch2b/bias:0', 'scene_model/res_net/private__identity_block/bn2b_branch2b/gamma:0', 'scene_model/res_net/private__identity_block/bn2b_branch2b/beta:0', 'scene_model/res_net/private__identity_block/res2b_branch2c/kernel:0', 'scene_model/res_net/private__identity_block/res2b_branch2c/bias:0', 'scene_model/res_net/private__identity_block/bn2b_branch2c/gamma:0', 'scene_model/res_net/private__identity_block/bn2b_branch2c/beta:0', 'scene_model/res_net/private__identity_block_1/res2c_branch2a/kernel:0', 'scene_model/res_net/private__identity_block_1/res2c_branch2a/bias:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2a/gamma:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2a/beta:0', 'scene_model/res_net/private__identity_block_1/res2c_branch2b/kernel:0', 'scene_model/res_net/private__identity_block_1/res2c_branch2b/bias:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2b/gamma:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2b/beta:0', 'scene_model/res_net/private__identity_block_1/res2c_branch2c/kernel:0', 'scene_model/res_net/private__identity_block_1/res2c_branch2c/bias:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2c/gamma:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2c/beta:0', 'scene_model/res_net/private__conv_block_1/res3a_branch2a/kernel:0', 'scene_model/res_net/private__conv_block_1/res3a_branch2a/bias:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2a/gamma:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2a/beta:0', 'scene_model/res_net/private__conv_block_1/res3a_branch2b/kernel:0', 'scene_model/res_net/private__conv_block_1/res3a_branch2b/bias:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2b/gamma:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2b/beta:0', 'scene_model/res_net/private__conv_block_1/res3a_branch2c/kernel:0', 'scene_model/res_net/private__conv_block_1/res3a_branch2c/bias:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2c/gamma:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2c/beta:0', 'scene_model/res_net/private__conv_block_1/res3a_branch1/kernel:0', 'scene_model/res_net/private__conv_block_1/res3a_branch1/bias:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch1/gamma:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch1/beta:0', 'scene_model/res_net/private__identity_block_2/res3b_branch2a/kernel:0', 'scene_model/res_net/private__identity_block_2/res3b_branch2a/bias:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2a/gamma:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2a/beta:0', 'scene_model/res_net/private__identity_block_2/res3b_branch2b/kernel:0', 'scene_model/res_net/private__identity_block_2/res3b_branch2b/bias:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2b/gamma:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2b/beta:0', 'scene_model/res_net/private__identity_block_2/res3b_branch2c/kernel:0', 'scene_model/res_net/private__identity_block_2/res3b_branch2c/bias:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2c/gamma:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2c/beta:0', 'scene_model/res_net/private__identity_block_3/res3c_branch2a/kernel:0', 'scene_model/res_net/private__identity_block_3/res3c_branch2a/bias:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2a/gamma:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2a/beta:0', 'scene_model/res_net/private__identity_block_3/res3c_branch2b/kernel:0', 'scene_model/res_net/private__identity_block_3/res3c_branch2b/bias:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2b/gamma:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2b/beta:0', 'scene_model/res_net/private__identity_block_3/res3c_branch2c/kernel:0', 'scene_model/res_net/private__identity_block_3/res3c_branch2c/bias:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2c/gamma:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2c/beta:0', 'scene_model/res_net/private__identity_block_4/res3d_branch2a/kernel:0', 'scene_model/res_net/private__identity_block_4/res3d_branch2a/bias:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2a/gamma:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2a/beta:0', 'scene_model/res_net/private__identity_block_4/res3d_branch2b/kernel:0', 'scene_model/res_net/private__identity_block_4/res3d_branch2b/bias:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2b/gamma:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2b/beta:0', 'scene_model/res_net/private__identity_block_4/res3d_branch2c/kernel:0', 'scene_model/res_net/private__identity_block_4/res3d_branch2c/bias:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2c/gamma:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2c/beta:0', 'scene_model/res_net/private__conv_block_2/res4a_branch2a/kernel:0', 'scene_model/res_net/private__conv_block_2/res4a_branch2a/bias:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2a/gamma:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2a/beta:0', 'scene_model/res_net/private__conv_block_2/res4a_branch2b/kernel:0', 'scene_model/res_net/private__conv_block_2/res4a_branch2b/bias:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2b/gamma:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2b/beta:0', 'scene_model/res_net/private__conv_block_2/res4a_branch2c/kernel:0', 'scene_model/res_net/private__conv_block_2/res4a_branch2c/bias:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2c/gamma:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2c/beta:0', 'scene_model/res_net/private__conv_block_2/res4a_branch1/kernel:0', 'scene_model/res_net/private__conv_block_2/res4a_branch1/bias:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch1/gamma:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch1/beta:0', 'scene_model/res_net/private__identity_block_5/res4b_branch2a/kernel:0', 'scene_model/res_net/private__identity_block_5/res4b_branch2a/bias:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2a/gamma:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2a/beta:0', 'scene_model/res_net/private__identity_block_5/res4b_branch2b/kernel:0', 'scene_model/res_net/private__identity_block_5/res4b_branch2b/bias:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2b/gamma:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2b/beta:0', 'scene_model/res_net/private__identity_block_5/res4b_branch2c/kernel:0', 'scene_model/res_net/private__identity_block_5/res4b_branch2c/bias:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2c/gamma:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2c/beta:0', 'scene_model/res_net/private__identity_block_6/res4c_branch2a/kernel:0', 'scene_model/res_net/private__identity_block_6/res4c_branch2a/bias:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2a/gamma:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2a/beta:0', 'scene_model/res_net/private__identity_block_6/res4c_branch2b/kernel:0', 'scene_model/res_net/private__identity_block_6/res4c_branch2b/bias:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2b/gamma:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2b/beta:0', 'scene_model/res_net/private__identity_block_6/res4c_branch2c/kernel:0', 'scene_model/res_net/private__identity_block_6/res4c_branch2c/bias:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2c/gamma:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2c/beta:0', 'scene_model/res_net/private__identity_block_7/res4d_branch2a/kernel:0', 'scene_model/res_net/private__identity_block_7/res4d_branch2a/bias:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2a/gamma:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2a/beta:0', 'scene_model/res_net/private__identity_block_7/res4d_branch2b/kernel:0', 'scene_model/res_net/private__identity_block_7/res4d_branch2b/bias:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2b/gamma:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2b/beta:0', 'scene_model/res_net/private__identity_block_7/res4d_branch2c/kernel:0', 'scene_model/res_net/private__identity_block_7/res4d_branch2c/bias:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2c/gamma:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2c/beta:0', 'scene_model/res_net/private__identity_block_8/res4e_branch2a/kernel:0', 'scene_model/res_net/private__identity_block_8/res4e_branch2a/bias:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2a/gamma:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2a/beta:0', 'scene_model/res_net/private__identity_block_8/res4e_branch2b/kernel:0', 'scene_model/res_net/private__identity_block_8/res4e_branch2b/bias:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2b/gamma:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2b/beta:0', 'scene_model/res_net/private__identity_block_8/res4e_branch2c/kernel:0', 'scene_model/res_net/private__identity_block_8/res4e_branch2c/bias:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2c/gamma:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2c/beta:0', 'scene_model/res_net/private__identity_block_9/res4f_branch2a/kernel:0', 'scene_model/res_net/private__identity_block_9/res4f_branch2a/bias:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2a/gamma:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2a/beta:0', 'scene_model/res_net/private__identity_block_9/res4f_branch2b/kernel:0', 'scene_model/res_net/private__identity_block_9/res4f_branch2b/bias:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2b/gamma:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2b/beta:0', 'scene_model/res_net/private__identity_block_9/res4f_branch2c/kernel:0', 'scene_model/res_net/private__identity_block_9/res4f_branch2c/bias:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2c/gamma:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2c/beta:0', 'scene_model/res_net/private__conv_block_3/res5a_branch2a/kernel:0', 'scene_model/res_net/private__conv_block_3/res5a_branch2a/bias:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2a/gamma:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2a/beta:0', 'scene_model/res_net/private__conv_block_3/res5a_branch2b/kernel:0', 'scene_model/res_net/private__conv_block_3/res5a_branch2b/bias:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2b/gamma:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2b/beta:0', 'scene_model/res_net/private__conv_block_3/res5a_branch2c/kernel:0', 'scene_model/res_net/private__conv_block_3/res5a_branch2c/bias:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2c/gamma:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2c/beta:0', 'scene_model/res_net/private__conv_block_3/res5a_branch1/kernel:0', 'scene_model/res_net/private__conv_block_3/res5a_branch1/bias:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch1/gamma:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch1/beta:0', 'scene_model/res_net/private__identity_block_10/res5b_branch2a/kernel:0', 'scene_model/res_net/private__identity_block_10/res5b_branch2a/bias:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2a/gamma:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2a/beta:0', 'scene_model/res_net/private__identity_block_10/res5b_branch2b/kernel:0', 'scene_model/res_net/private__identity_block_10/res5b_branch2b/bias:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2b/gamma:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2b/beta:0', 'scene_model/res_net/private__identity_block_10/res5b_branch2c/kernel:0', 'scene_model/res_net/private__identity_block_10/res5b_branch2c/bias:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2c/gamma:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2c/beta:0', 'scene_model/res_net/private__identity_block_11/res5c_branch2a/kernel:0', 'scene_model/res_net/private__identity_block_11/res5c_branch2a/bias:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2a/gamma:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2a/beta:0', 'scene_model/res_net/private__identity_block_11/res5c_branch2b/kernel:0', 'scene_model/res_net/private__identity_block_11/res5c_branch2b/bias:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2b/gamma:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2b/beta:0', 'scene_model/res_net/private__identity_block_11/res5c_branch2c/kernel:0', 'scene_model/res_net/private__identity_block_11/res5c_branch2c/bias:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2c/gamma:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2c/beta:0', 'scene_model/model/fcn_score_fr/kernel:0', 'scene_model/model/fcn_score_fr/bias:0', 'scene_model/model/fcn_upsampling_upscore2/fcn_upsampling_upscore2/kernel:0', 'scene_model/model/fcn_upsampling_upscore2/fcn_upsampling_upscore2/bias:0', 'scene_model/model/fcn_score_feed2/kernel:0', 'scene_model/model/fcn_score_feed2/bias:0', 'scene_model/model/fcn_upsampling_upscore4/fcn_upsampling_upscore4/kernel:0', 'scene_model/model/fcn_upsampling_upscore4/fcn_upsampling_upscore4/bias:0', 'scene_model/model/fcn_score_feed4/kernel:0', 'scene_model/model/fcn_score_feed4/bias:0', 'scene_model/model/fcn_upsampling_upscore32/fcn_upsampling_upscore32/kernel:0', 'scene_model/model/fcn_upsampling_upscore32/fcn_upsampling_upscore32/bias:0', 'scene_model/res_net/bn_conv1/moving_mean:0', 'scene_model/res_net/bn_conv1/moving_variance:0', 'scene_model/res_net/private__conv_block/bn2a_branch2a/moving_mean:0', 'scene_model/res_net/private__conv_block/bn2a_branch2a/moving_variance:0', 'scene_model/res_net/private__conv_block/bn2a_branch2b/moving_mean:0', 'scene_model/res_net/private__conv_block/bn2a_branch2b/moving_variance:0', 'scene_model/res_net/private__conv_block/bn2a_branch2c/moving_mean:0', 'scene_model/res_net/private__conv_block/bn2a_branch2c/moving_variance:0', 'scene_model/res_net/private__conv_block/bn2a_branch1/moving_mean:0', 'scene_model/res_net/private__conv_block/bn2a_branch1/moving_variance:0', 'scene_model/res_net/private__identity_block/bn2b_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block/bn2b_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block/bn2b_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block/bn2b_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block/bn2b_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block/bn2b_branch2c/moving_variance:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block_1/bn2c_branch2c/moving_variance:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2a/moving_mean:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2a/moving_variance:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2b/moving_mean:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2b/moving_variance:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2c/moving_mean:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch2c/moving_variance:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch1/moving_mean:0', 'scene_model/res_net/private__conv_block_1/bn3a_branch1/moving_variance:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block_2/bn3b_branch2c/moving_variance:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block_3/bn3c_branch2c/moving_variance:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block_4/bn3d_branch2c/moving_variance:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2a/moving_mean:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2a/moving_variance:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2b/moving_mean:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2b/moving_variance:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2c/moving_mean:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch2c/moving_variance:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch1/moving_mean:0', 'scene_model/res_net/private__conv_block_2/bn4a_branch1/moving_variance:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block_5/bn4b_branch2c/moving_variance:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block_6/bn4c_branch2c/moving_variance:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block_7/bn4d_branch2c/moving_variance:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block_8/bn4e_branch2c/moving_variance:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block_9/bn4f_branch2c/moving_variance:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2a/moving_mean:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2a/moving_variance:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2b/moving_mean:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2b/moving_variance:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2c/moving_mean:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch2c/moving_variance:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch1/moving_mean:0', 'scene_model/res_net/private__conv_block_3/bn5a_branch1/moving_variance:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block_10/bn5b_branch2c/moving_variance:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2a/moving_mean:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2a/moving_variance:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2b/moving_mean:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2b/moving_variance:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2c/moving_mean:0', 'scene_model/res_net/private__identity_block_11/bn5c_branch2c/moving_variance:0'], var_name_to_vocab_info={}, var_name_to_prev_var_name={})
[2019-05-15 15:41:26,157] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,157] {warm_starting_util.py:417} INFO - Warm-starting from: ('/home/benjamintan/workspace/darkrai/logs/shapes_768_1024_20190515T1541/init/',)
[2019-05-15 15:41:26,364] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,363] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/conv1/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,364] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,364] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/conv1/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,364] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,364] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/bn_conv1/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,364] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,364] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/bn_conv1/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,364] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,364] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/res2a_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,364] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,364] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/res2a_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,365] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,365] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,365] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,365] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,365] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,365] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/res2a_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,365] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,365] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/res2a_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,365] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,365] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,365] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,365] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,366] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,365] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/res2a_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,366] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,366] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/res2a_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,366] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,366] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,366] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,366] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,366] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,366] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/res2a_branch1/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,366] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,366] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/res2a_branch1/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,366] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,366] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch1/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,367] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,367] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch1/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,367] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,367] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/res2b_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,367] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,367] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/res2b_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,367] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,367] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,367] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,367] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,367] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,367] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/res2b_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,368] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,368] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/res2b_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,368] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,368] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,368] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,368] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,368] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,368] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/res2b_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,368] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,368] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/res2b_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,368] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,368] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,368] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,368] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,369] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,369] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/res2c_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,369] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,369] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/res2c_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,369] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,369] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,369] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,369] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,369] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,369] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/res2c_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,369] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,369] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/res2c_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,370] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,370] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,370] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,370] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,370] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,370] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/res2c_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,370] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,370] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/res2c_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,370] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,370] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,370] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,370] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,371] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,370] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/res3a_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,371] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,371] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/res3a_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,371] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,371] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,371] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,371] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,371] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,371] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/res3a_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,371] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,371] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/res3a_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,371] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,371] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,372] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,372] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,372] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,372] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/res3a_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,372] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,372] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/res3a_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,372] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,372] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,372] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,372] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,372] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,372] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/res3a_branch1/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,373] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,372] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/res3a_branch1/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,373] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,373] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch1/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,373] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,373] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch1/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,373] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,373] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/res3b_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,373] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,373] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/res3b_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,373] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,373] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,373] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,373] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,374] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,374] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/res3b_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,374] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,374] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/res3b_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,374] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,374] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,374] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,374] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,374] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,374] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/res3b_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,374] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,374] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/res3b_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,375] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,375] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,375] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,375] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,375] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,375] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/res3c_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,375] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,375] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/res3c_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,375] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,375] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,375] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,375] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,375] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,375] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/res3c_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,376] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,376] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/res3c_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,376] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,376] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,376] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,376] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,376] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,376] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/res3c_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,376] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,376] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/res3c_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,376] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,376] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,377] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,377] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,377] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,377] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/res3d_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,377] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,377] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/res3d_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,377] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,377] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,377] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,377] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,377] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,377] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/res3d_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,378] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,377] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/res3d_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,378] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,378] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,378] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,378] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,378] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,378] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/res3d_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,378] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,378] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/res3d_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,378] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,378] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,378] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,378] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,379] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,379] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/res4a_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,379] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,379] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/res4a_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,379] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,379] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,379] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,379] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,379] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,379] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/res4a_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,379] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,379] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/res4a_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,380] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,379] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,380] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,380] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,380] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,380] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/res4a_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,380] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,380] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/res4a_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,380] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,380] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,380] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,380] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,380] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,380] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/res4a_branch1/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,381] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,381] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/res4a_branch1/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,381] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,381] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch1/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,381] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,381] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch1/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,381] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,381] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/res4b_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,381] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,381] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/res4b_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,381] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,381] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,382] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,382] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,382] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,382] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/res4b_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,382] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,382] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/res4b_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,382] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,382] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,382] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,382] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,382] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,382] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/res4b_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,382] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,382] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/res4b_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,383] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,383] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,383] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,383] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,383] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,383] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/res4c_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,383] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,383] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/res4c_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,383] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,383] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,383] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,383] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,384] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,384] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/res4c_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,384] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,384] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/res4c_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,384] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,384] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,384] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,384] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,384] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,384] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/res4c_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,384] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,384] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/res4c_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,385] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,384] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,385] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,385] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,385] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,385] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/res4d_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,385] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,385] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/res4d_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,385] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,385] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,385] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,385] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,385] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,385] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/res4d_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,386] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,386] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/res4d_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,386] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,386] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,386] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,386] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,386] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,386] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/res4d_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,386] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,386] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/res4d_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,386] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,386] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,387] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,386] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,387] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,387] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/res4e_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,387] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,387] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/res4e_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,387] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,387] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,387] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,387] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,387] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,387] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/res4e_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,387] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,387] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/res4e_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,388] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,388] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,388] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,388] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,388] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,388] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/res4e_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,388] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,388] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/res4e_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,388] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,388] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,388] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,388] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,389] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,389] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/res4f_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,389] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,389] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/res4f_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,389] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,389] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,389] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,389] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,389] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,389] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/res4f_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,389] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,389] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/res4f_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,390] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,389] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,390] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,390] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,390] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,390] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/res4f_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,390] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,390] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/res4f_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,390] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,390] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,390] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,390] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,390] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,390] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/res5a_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,391] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,391] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/res5a_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,391] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,391] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,391] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,391] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,391] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,391] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/res5a_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,391] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,391] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/res5a_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,391] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,391] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,392] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,391] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,392] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,392] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/res5a_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,392] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,392] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/res5a_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,392] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,392] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,392] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,392] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,392] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,392] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/res5a_branch1/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,392] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,392] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/res5a_branch1/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,393] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,393] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch1/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,393] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,393] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch1/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,393] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,393] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/res5b_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,393] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,393] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/res5b_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,393] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,393] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,393] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,393] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,394] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,394] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/res5b_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,394] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,394] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/res5b_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,394] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,394] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,394] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,394] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,394] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,394] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/res5b_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,394] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,394] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/res5b_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,394] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,394] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,395] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,395] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,395] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,395] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/res5c_branch2a/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,395] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,395] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/res5c_branch2a/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,395] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,395] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2a/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,395] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,395] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2a/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,395] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,395] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/res5c_branch2b/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,396] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,396] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/res5c_branch2b/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,396] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,396] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2b/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,396] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,396] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2b/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,396] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,396] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/res5c_branch2c/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,396] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,396] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/res5c_branch2c/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,396] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,396] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2c/gamma; prev_var_name: Unchanged
[2019-05-15 15:41:26,397] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,396] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2c/beta; prev_var_name: Unchanged
[2019-05-15 15:41:26,397] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,397] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_score_fr/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,397] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,397] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_score_fr/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,397] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,397] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_upsampling_upscore2/fcn_upsampling_upscore2/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,397] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,397] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_upsampling_upscore2/fcn_upsampling_upscore2/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,397] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,397] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_score_feed2/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,397] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,397] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_score_feed2/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,398] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,398] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_upsampling_upscore4/fcn_upsampling_upscore4/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,398] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,398] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_upsampling_upscore4/fcn_upsampling_upscore4/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,398] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,398] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_score_feed4/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,398] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,398] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_score_feed4/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,398] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,398] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_upsampling_upscore32/fcn_upsampling_upscore32/kernel; prev_var_name: Unchanged
[2019-05-15 15:41:26,398] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,398] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/model/fcn_upsampling_upscore32/fcn_upsampling_upscore32/bias; prev_var_name: Unchanged
[2019-05-15 15:41:26,399] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,398] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/bn_conv1/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,399] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,399] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/bn_conv1/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,399] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,399] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,399] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,399] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,399] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,399] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,399] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,399] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,399] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,399] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,400] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,400] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,400] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,400] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch1/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,400] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,400] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block/bn2a_branch1/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,400] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,400] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,400] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,400] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,400] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,400] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,401] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,401] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,401] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,401] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,401] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,401] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block/bn2b_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,401] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,401] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,401] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,401] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,401] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,401] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,401] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,401] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,402] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,402] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,402] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,402] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_1/bn2c_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,402] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,402] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,402] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,402] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,402] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,402] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,402] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,402] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,403] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,403] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,403] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,403] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,403] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,403] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch1/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,403] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,403] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_1/bn3a_branch1/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,403] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,403] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,403] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,403] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,404] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,403] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,404] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,404] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,404] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,404] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,404] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,404] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_2/bn3b_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,404] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,404] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,404] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,404] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,404] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,404] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,405] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,405] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,405] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,405] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,405] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,405] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_3/bn3c_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,405] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,405] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,405] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,405] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,405] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,405] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,406] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,405] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,406] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,406] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,406] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,406] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_4/bn3d_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,406] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,406] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,406] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,406] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,406] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,406] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,406] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,406] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,407] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,407] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,407] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,407] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,407] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,407] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch1/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,407] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,407] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_2/bn4a_branch1/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,407] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,407] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,407] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,407] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,408] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,408] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,408] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,408] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,408] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,408] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,408] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,408] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_5/bn4b_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,408] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,408] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,408] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,408] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,408] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,408] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,409] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,409] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,409] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,409] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,409] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,409] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_6/bn4c_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,409] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,409] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,409] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,409] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,409] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,409] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,410] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,410] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,410] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,410] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,410] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,410] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_7/bn4d_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,410] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,410] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,410] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,410] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,410] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,410] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,411] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,410] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,411] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,411] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,411] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,411] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_8/bn4e_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,411] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,411] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,411] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,411] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,411] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,411] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,411] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,411] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,412] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,412] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,412] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,412] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_9/bn4f_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,412] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,412] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,412] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,412] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,412] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,412] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,412] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,412] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,413] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,412] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,413] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,413] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,413] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,413] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch1/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,413] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,413] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__conv_block_3/bn5a_branch1/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,413] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,413] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,413] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,413] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,413] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,413] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,414] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,414] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,414] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,414] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,414] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,414] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_10/bn5b_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,414] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,414] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2a/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,414] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,414] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2a/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,414] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,414] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2b/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,415] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,415] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2b/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:26,415] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,415] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2c/moving_mean; prev_var_name: Unchanged
[2019-05-15 15:41:26,415] {logging_mixin.py:95} INFO - [2019-05-15 15:41:26,415] {warm_starting_util.py:466} INFO - Warm-starting variable: scene_model/res_net/private__identity_block_11/bn5c_branch2c/moving_variance; prev_var_name: Unchanged
[2019-05-15 15:41:30,779] {logging_mixin.py:95} INFO - [2019-05-15 15:41:30,778] {monitored_session.py:222} INFO - Graph was finalized.
[2019-05-15 15:41:32,622] {logging_mixin.py:95} INFO - [2019-05-15 15:41:32,622] {session_manager.py:491} INFO - Running local_init_op.
[2019-05-15 15:41:32,803] {logging_mixin.py:95} INFO - [2019-05-15 15:41:32,803] {session_manager.py:493} INFO - Done running local_init_op.
[2019-05-15 15:41:36,666] {logging_mixin.py:95} INFO - [2019-05-15 15:41:36,666] {basic_session_run_hooks.py:594} INFO - Saving checkpoints for 0 into /home/benjamintan/workspace/darkrai/logs/shapes_768_1024_20190515T1541/model.ckpt.
[2019-05-15 15:41:39,034] {logging_mixin.py:95} INFO - [2019-05-15 15:41:39,034] {util.py:164} INFO - Initialize strategy
[2019-05-15 15:41:40,518] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:40,580] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:40,684] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:41,827] {logging_mixin.py:95} INFO - [2019-05-15 15:41:41,827] {basic_session_run_hooks.py:249} INFO - loss = 2.4945161, step = 0
[2019-05-15 15:41:43,135] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:44,310] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:45,406] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:46,485] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:47,531] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:48,552] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:49,582] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:50,627] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:51,666] {base_task_runner.py:101} INFO - Job 1566: Subtask darkrai.Trainer.exec [OHAI]
[2019-05-15 15:41:52,682] {logging_mixin.py:95} INFO - [2019-05-15 15:41:52,682] {basic_session_run_hooks.py:594} INFO - Saving checkpoints for 10 into /home/benjamintan/workspace/darkrai/logs/shapes_768_1024_20190515T1541/model.ckpt.
[2019-05-15 15:41:53,570] {logging_mixin.py:95} INFO - **** EVAL INPUT FN ****
[2019-05-15 15:41:53,570] {logging_mixin.py:95} INFO - /home/benjamintan/tfx/pipelines/darkrai/Transform/transformed_examples/11/eval/*
[2019-05-15 15:41:53,571] {logging_mixin.py:95} INFO - Before build_gt_data_dataset
[2019-05-15 15:41:54,068] {logging_mixin.py:95} INFO - Before filter
[2019-05-15 15:41:54,079] {logging_mixin.py:95} INFO - After filter
[2019-05-15 15:41:54,079] {logging_mixin.py:95} INFO - <DatasetV1Adapter shapes: {images: (?, ?, 3), image_meta: (12,), image_rel_path: (), gt_class_ids: (?,), gt_boxes: (?, 4), loss_activation: (2,), gt_instance_masks: (?, ?, ?), gt_seg_masks: (?, ?, 6)}, types: {images: tf.uint8, image_meta: tf.float32, image_rel_path: tf.string, gt_class_ids: tf.int32, gt_boxes: tf.int32, loss_activation: tf.bool, gt_instance_masks: tf.bool, gt_seg_masks: tf.bool}>
[2019-05-15 15:41:54,122] {logging_mixin.py:95} INFO - BATCH SIZE
[2019-05-15 15:41:54,122] {logging_mixin.py:95} INFO - 1
[2019-05-15 15:41:54,124] {logging_mixin.py:95} INFO - Before Map
[2019-05-15 15:41:54,139] {logging_mixin.py:95} INFO - After Map
[2019-05-15 15:41:54,139] {logging_mixin.py:95} INFO - <DatasetV1Adapter shapes: ({images: (?, ?, ?, 3), image_meta: (?, 12), gt_seg_masks: (?, ?, ?, 6)}, {images: (?, ?, ?, 3), image_meta: (?, 12), image_rel_path: (?,), loss_activation: (?, 2), gt_seg_masks: (?, ?, ?, 6)}), types: ({images: tf.uint8, image_meta: tf.float32, gt_seg_masks: tf.bool}, {images: tf.uint8, image_meta: tf.float32, image_rel_path: tf.string, loss_activation: tf.bool, gt_seg_masks: tf.bool})>
[2019-05-15 15:41:54,183] {logging_mixin.py:95} INFO - [2019-05-15 15:41:54,182] {estimator.py:1111} INFO - Calling model_fn.
[2019-05-15 15:41:57,884] {logging_mixin.py:95} INFO - [2019-05-15 15:41:57,883] {deprecation.py:323} WARNING - From /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/ops/metrics_impl.py:363: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
[2019-05-15 15:41:57,923] {logging_mixin.py:95} INFO - [2019-05-15 15:41:57,923] {estimator.py:1113} INFO - Done calling model_fn.
[2019-05-15 15:41:57,952] {logging_mixin.py:95} INFO - [2019-05-15 15:41:57,951] {evaluation.py:257} INFO - Starting evaluation at 2019-05-15T07:41:57Z
[2019-05-15 15:41:59,444] {logging_mixin.py:95} INFO - [2019-05-15 15:41:59,443] {monitored_session.py:222} INFO - Graph was finalized.
[2019-05-15 15:41:59,444] {logging_mixin.py:95} INFO - [2019-05-15 15:41:59,444] {deprecation.py:323} WARNING - From /home/benjamintan/miniconda3/envs/mask_rcnn_gpu/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1266: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
[2019-05-15 15:41:59,446] {logging_mixin.py:95} INFO - [2019-05-15 15:41:59,446] {saver.py:1270} INFO - Restoring parameters from /home/benjamintan/workspace/darkrai/logs/shapes_768_1024_20190515T1541/model.ckpt-10
[2019-05-15 15:42:00,452] {logging_mixin.py:95} INFO - [2019-05-15 15:42:00,451] {session_manager.py:491} INFO - Running local_init_op.
[2019-05-15 15:42:00,560] {logging_mixin.py:95} INFO - [2019-05-15 15:42:00,559] {session_manager.py:493} INFO - Done running local_init_op.
[2019-05-15 15:52:38,147] {logging_mixin.py:95} INFO - [2019-05-15 15:52:38,145] {jobs.py:2562} INFO - Task exited with return code -9

benjamintanweihao · 2019-05-16T05:45:29Z

I'm using ImportExampleGen and pointing it to a known TFRecord directory that has been used successfully for training (without TFX)

benjamintanweihao · 2019-05-16T07:52:23Z

I think I found the problem! I parsed the filenames parameter from eval_input_fn(filenames, transform_output) wrongly. It turns out that filenames is a string and not a list?

ruoyu90 · 2019-05-16T20:03:54Z

@benjamintanweihao Thanks for reporting this! Actually this helps surfaced a bug in our trainer executor. The train_files and eval_files should have consistent type. Will send a PR to fix this soon.

…r module to have same type (List[str]). PiperOrigin-RevId: 248586432

…r module to have same type (List[str]). Also removing unused output_dir in params passed to user module. PiperOrigin-RevId: 248586432

…r module to have same type (List[str]). Also removing unused output_dir in params passed to user module. PiperOrigin-RevId: 248637693

* RFC proposal on standartizing composite operations in tensorflow. * RFC: Standardizing Composite Operations In TensorFlow * typo fix * Update 20190610-standartizing-composite_ops.md typo fixes * Update rfcs/20190610-standartizing-composite_ops.md Co-Authored-By: Mehdi Amini <joker.eph@gmail.com> * Update 20190610-standartizing-composite_ops.md * Update 20190610-standartizing-composite_ops.md * Update 20190610-standartizing-composite_ops.md * Fixes type in file name

gowthamkpr self-assigned this May 13, 2019

gowthamkpr added type:support stat:awaiting response labels May 13, 2019

gowthamkpr added stat:awaiting response and removed stat:awaiting response labels May 14, 2019

gowthamkpr added stat:awaiting tensorflower and removed stat:awaiting response labels May 15, 2019

ruoyu90 removed the stat:awaiting tensorflower label May 15, 2019

gowthamkpr added the stat:awaiting response label May 16, 2019

tfx-copybara pushed a commit that referenced this issue May 16, 2019

Fix issue #113: Make train_files and eval_files passed to trainer use…

f7cc5b8

…r module to have same type (List[str]). PiperOrigin-RevId: 248586432

tfx-copybara mentioned this issue May 16, 2019

Fix issue #113: Make train_files and eval_files passed to trainer user module to have same type (List[str]). #142

Merged

tfx-copybara pushed a commit that referenced this issue May 16, 2019

Fix issue #113: Make train_files and eval_files passed to trainer use…

4677f16

…r module to have same type (List[str]). PiperOrigin-RevId: 248586432

tfx-copybara pushed a commit that referenced this issue May 17, 2019

Fix issue #113: Make train_files and eval_files passed to trainer use…

cb0404d

…r module to have same type (List[str]). Also removing unused output_dir in params passed to user module. PiperOrigin-RevId: 248586432

tfx-copybara pushed a commit that referenced this issue May 17, 2019

Fix issue #113: Make train_files and eval_files passed to trainer use…

83b8477

…r module to have same type (List[str]). Also removing unused output_dir in params passed to user module. PiperOrigin-RevId: 248637693

benjamintanweihao closed this as completed May 17, 2019

gowthamkpr added type:bug and removed type:support labels May 17, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eval gets stuck forever in the Trainer Component #113

Eval gets stuck forever in the Trainer Component #113

benjamintanweihao commented May 13, 2019

1025KB commented May 13, 2019

benjamintanweihao commented May 13, 2019 via email

1025KB commented May 14, 2019

benjamintanweihao commented May 15, 2019 •

edited

ruoyu90 commented May 15, 2019

benjamintanweihao commented May 16, 2019

benjamintanweihao commented May 16, 2019

benjamintanweihao commented May 16, 2019

ruoyu90 commented May 16, 2019

Eval gets stuck forever in the Trainer Component #113

Eval gets stuck forever in the Trainer Component #113

Comments

benjamintanweihao commented May 13, 2019

1025KB commented May 13, 2019

benjamintanweihao commented May 13, 2019 via email

1025KB commented May 14, 2019

benjamintanweihao commented May 15, 2019 • edited

ruoyu90 commented May 15, 2019

benjamintanweihao commented May 16, 2019

benjamintanweihao commented May 16, 2019

benjamintanweihao commented May 16, 2019

ruoyu90 commented May 16, 2019

benjamintanweihao commented May 15, 2019 •

edited