/usr/local/lib/python3.5/dist-packages/object_detection/utils/visualization_utils.py:25: UserWarning: 
This call to matplotlib.use() has no effect because the backend has already
been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
or matplotlib.backends is imported for the first time.

The backend was *originally* set to 'TkAgg' by the following code:
  File "/usr/lib/python3.5/runpy.py", line 184, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.5/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.5/dist-packages/object_detection/model_main.py", line 26, in <module>
    from object_detection import model_lib
  File "/usr/local/lib/python3.5/dist-packages/object_detection/model_lib.py", line 26, in <module>
    from object_detection import eval_util
  File "/usr/local/lib/python3.5/dist-packages/object_detection/eval_util.py", line 28, in <module>
    from object_detection.metrics import coco_evaluation
  File "/usr/local/lib/python3.5/dist-packages/object_detection/metrics/coco_evaluation.py", line 20, in <module>
    from object_detection.metrics import coco_tools
  File "/usr/local/lib/python3.5/dist-packages/object_detection/metrics/coco_tools.py", line 47, in <module>
    from pycocotools import coco
  File "/usr/local/lib/python3.5/dist-packages/pycocotools-2.0-py3.5-linux-x86_64.egg/pycocotools/coco.py", line 49, in <module>
    import matplotlib.pyplot as plt
  File "/usr/local/lib/python3.5/dist-packages/matplotlib/pyplot.py", line 71, in <module>
    from matplotlib.backends import pylab_setup
  File "/usr/local/lib/python3.5/dist-packages/matplotlib/backends/__init__.py", line 16, in <module>
    line for line in traceback.format_stack()


  import matplotlib; matplotlib.use('Agg')  # pylint: disable=multiple-statements
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7f74228e3378>) includes params argument, but params are not passed to Estimator.
2019-09-25 17:58:37.609852: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-09-25 17:58:37.715990: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-09-25 17:58:37.716504: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1392] Found device 0 with properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
pciBusID: 0000:00:04.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-09-25 17:58:37.716552: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1471] Adding visible gpu devices: 0
2019-09-25 17:58:38.051270: I tensorflow/core/common_runtime/gpu/gpu_device.cc:952] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-25 17:58:38.051346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:958]      0 
2019-09-25 17:58:38.051360: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] 0:   N 
2019-09-25 17:58:38.051690: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1084] Created TensorFlow device (/job:chief/replica:0/task:0/device:GPU:0 with 10763 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:04.0, compute capability: 3.7)
2019-09-25 17:58:38.282984: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job chief -> {0 -> localhost:2222}
2019-09-25 17:58:38.283049: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job ps -> {0 -> pets-train-dist-ps-0.kubeflow.svc:2222}
2019-09-25 17:58:38.283063: I tensorflow/core/distributed_runtime/rpc/grpc_channel.cc:215] Initialize GrpcChannelCache for job worker -> {0 -> pets-train-dist-worker-0.kubeflow.svc:2222, 1 -> pets-train-dist-worker-1.kubeflow.svc:2222}
2019-09-25 17:58:38.284459: I tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc:334] Started server with target: grpc://localhost:2222
WARNING:tensorflow:num_readers has been reduced to 10 to match input file shards.
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/object_detection/predictors/heads/box_head.py:92: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/object_detection/meta_architectures/faster_rcnn_meta_arch.py:2082: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
WARNING:root:Variable [SecondStageBoxPredictor/BoxEncodingPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[360]], model variable shape: [[148]]. This variable will not be initialized from the checkpoint.
WARNING:root:Variable [SecondStageBoxPredictor/BoxEncodingPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[2048, 360]], model variable shape: [[2048, 148]]. This variable will not be initialized from the checkpoint.
WARNING:root:Variable [SecondStageBoxPredictor/ClassPredictor/biases] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[91]], model variable shape: [[38]]. This variable will not be initialized from the checkpoint.
WARNING:root:Variable [SecondStageBoxPredictor/ClassPredictor/weights] is available in checkpoint, but has an incompatible shape with model variable. Checkpoint shape: [[2048, 91]], model variable shape: [[2048, 38]]. This variable will not be initialized from the checkpoint.
WARNING:root:Variable [global_step] is not available in checkpoint
WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/object_detection/core/losses.py:317: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

WARNING:tensorflow:From /usr/local/lib/python3.5/dist-packages/object_detection/core/losses.py:317: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

2019-09-25 17:59:14.253407: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 17:59:24.253639: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 17:59:34.253857: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 17:59:44.254051: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 17:59:54.254253: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:00:04.254522: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:00:14.254755: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:00:24.254959: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:00:34.255174: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:00:44.255493: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:00:54.255766: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:01:04.255972: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:01:14.256239: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:01:24.256508: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:01:34.256718: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:01:44.256964: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:01:54.257214: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:02:04.257465: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:02:14.257735: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:02:24.257982: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:02:34.258251: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:02:44.258510: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:02:54.258745: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:03:04.259022: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:03:14.259332: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:03:24.259575: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:03:34.259824: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:03:44.260055: I tensorflow/core/distributed_runtime/master.cc:224] CreateSession still waiting for response from worker: /job:worker/replica:0/task:1
2019-09-25 18:03:45.815722: I tensorflow/core/distributed_runtime/master_session.cc:1150] Start master session c3d906a3d28870ec with config: allow_soft_placement: true graph_options { rewrite_options { meta_optimizer_iterations: ONE } }
Corrupt JPEG data: 240 extraneous bytes before marker 0xd9