Adding new features to extend the functionality and capability of the API

research/object_detection/README.md

@@ -77,6 +77,10 @@ Extras:
       Run an instance segmentation model</a><br>
   * <a href='g3doc/challenge_evaluation.md'>
       Run the evaluation for the Open Images Challenge 2018</a><br>
+  * <a href='g3doc/tpu_compatibility.md'>
+      TPU compatible detection pipelines</a><br>
+  *  <a href='g3doc/running_on_mobile_tensorflowlite.md'>
+      Running object detection on mobile devices with TensorFlow Lite</a><br>
 
 ## Getting Help
 
@@ -95,6 +99,28 @@ reporting an issue.
 
 ## Release information
 
+### July 13, 2018
+
+There are many new updates in this release, extending the functionality and
+capability of the API:
+
+* Moving from slim-based training to [Estimator](https://www.tensorflow.org/api_docs/python/tf/estimator/Estimator)-based
+training.
+* Support for [RetinaNet](https://arxiv.org/abs/1708.02002), and a [MobileNet](https://ai.googleblog.com/2017/06/mobilenets-open-source-models-for.html)
+adaptation of RetinaNet.
+* A novel SSD-based architecture called the [Pooling Pyramid Network](https://arxiv.org/abs/1807.03284) (PPN).
+* Releasing several [TPU](https://cloud.google.com/tpu/)-compatible models.
+These can be found in the `samples/configs/` directory with a comment in the
+pipeline configuration files indicating TPU compatibility.
+* Support for quantized training.
+* Updated documentation for new binaries, Cloud training, and [Tensorflow Lite](https://www.tensorflow.org/mobile/tflite/).
+
+See also our [expanded announcement blogpost](https://ai.googleblog.com/2018/07/accelerated-training-and-inference-with.html) and accompanying tutorial at the [TensorFlow blog](https://medium.com/tensorflow/training-and-serving-a-realtime-mobile-object-detector-in-30-minutes-with-cloud-tpus-b78971cf1193).
+
+<b>Thanks to contributors</b>: Sara Robinson, Aakanksha Chowdhery, Derek Chow,
+Pengchong Jin, Jonathan Huang, Vivek Rathod, Zhichao Lu, Ronny Votel
+
+
 ### June 25, 2018
 
 Additional evaluation tools for the [Open Images Challenge 2018](https://storage.googleapis.com/openimages/web/challenge.html) are out.

research/object_detection/dockerfiles/android/Dockerfile

@@ -0,0 +1,136 @@
+# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# #==========================================================================
+
+FROM tensorflow/tensorflow:nightly-devel
+
+# Get the tensorflow models research directory, and move it into tensorflow
+# source folder to match recommendation of installation
+RUN git clone --depth 1 https://github.com/tensorflow/models.git && \
+    mv models /tensorflow/models
+
+
+# Install gcloud and gsutil commands
+# https://cloud.google.com/sdk/docs/quickstart-debian-ubuntu
+RUN export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" && \
+    echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
+    curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
+    apt-get update -y && apt-get install google-cloud-sdk -y
+
+
+# Install the Tensorflow Object Detection API from here
+# https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/installation.md
+
+# Install object detection api dependencies
+RUN apt-get install -y protobuf-compiler python-pil python-lxml python-tk && \
+    pip install Cython && \
+    pip install contextlib2 && \
+    pip install jupyter && \
+    pip install matplotlib
+
+# Install pycocoapi
+RUN git clone --depth 1 https://github.com/cocodataset/cocoapi.git && \
+    cd cocoapi/PythonAPI && \
+    make -j8 && \
+    cp -r pycocotools /tensorflow/models/research && \
+    cd ../../ && \
+    rm -rf cocoapi
+
+# Get protoc 3.0.0, rather than the old version already in the container
+RUN curl -OL "https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip" && \
+    unzip protoc-3.0.0-linux-x86_64.zip -d proto3 && \
+    mv proto3/bin/* /usr/local/bin && \
+    mv proto3/include/* /usr/local/include && \
+    rm -rf proto3 protoc-3.0.0-linux-x86_64.zip
+
+# Run protoc on the object detection repo
+RUN cd /tensorflow/models/research && \
+    protoc object_detection/protos/*.proto --python_out=.
+
+# Set the PYTHONPATH to finish installing the API
+ENV PYTHONPATH $PYTHONPATH:/tensorflow/models/research:/tensorflow/models/research/slim
+
+
+# Install wget (to make life easier below) and editors (to allow people to edit
+# the files inside the container)
+RUN apt-get install -y wget vim emacs nano
+
+
+# Grab various data files which are used throughout the demo: dataset,
+# pretrained model, and pretrained TensorFlow Lite model. Install these all in
+# the same directories as recommended by the blog post.
+
+# Pets example dataset
+RUN mkdir -p /tmp/pet_faces_tfrecord/ && \
+    cd /tmp/pet_faces_tfrecord && \
+    curl "http://download.tensorflow.org/models/object_detection/pet_faces_tfrecord.tar.gz" | tar xzf -
+
+# Pretrained model
+# This one doesn't need its own directory, since it comes in a folder.
+RUN cd /tmp && \
+    curl -O "http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_0.75_depth_300x300_coco14_sync_2018_07_03.tar.gz" && \
+    tar xzf ssd_mobilenet_v1_0.75_depth_300x300_coco14_sync_2018_07_03.tar.gz && \
+    rm ssd_mobilenet_v1_0.75_depth_300x300_coco14_sync_2018_07_03.tar.gz
+
+# Trained TensorFlow Lite model. This should get replaced by one generated from
+# export_tflite_ssd_graph.py when that command is called.
+RUN cd /tmp && \
+    curl -L -o tflite.zip \
+    https://storage.googleapis.com/download.tensorflow.org/models/tflite/frozengraphs_ssd_mobilenet_v1_0.75_quant_pets_2018_06_29.zip && \
+    unzip tflite.zip -d tflite && \
+    rm tflite.zip
+
+
+# Install Android development tools
+# Inspired by the following sources:
+# https://github.com/bitrise-docker/android/blob/master/Dockerfile
+# https://github.com/reddit/docker-android-build/blob/master/Dockerfile
+
+# Set environment variables
+ENV ANDROID_HOME /opt/android-sdk-linux
+ENV ANDROID_NDK_HOME /opt/android-ndk-r14b
+ENV PATH ${PATH}:${ANDROID_HOME}/tools:${ANDROID_HOME}/tools/bin:${ANDROID_HOME}/platform-tools
+
+# Install SDK tools
+RUN cd /opt && \
+    curl -OL https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip && \
+    unzip sdk-tools-linux-4333796.zip -d ${ANDROID_HOME} && \
+    rm sdk-tools-linux-4333796.zip
+
+# Accept licenses before installing components, no need to echo y for each component
+# License is valid for all the standard components in versions installed from this file
+# Non-standard components: MIPS system images, preview versions, GDK (Google Glass) and Android Google TV require separate licenses, not accepted there
+RUN yes | sdkmanager --licenses
+
+# Install platform tools, SDK platform, and other build tools
+RUN yes | sdkmanager \
+    "tools" \
+    "platform-tools" \
+    "platforms;android-27" \
+    "platforms;android-23" \
+    "build-tools;27.0.3" \
+    "build-tools;23.0.3"
+
+# Install Android NDK (r14b)
+RUN cd /opt && \
+    curl -L -o android-ndk.zip http://dl.google.com/android/repository/android-ndk-r14b-linux-x86_64.zip && \
+    unzip -q android-ndk.zip && \
+    rm -f android-ndk.zip
+
+# Configure the build to use the things we just downloaded
+RUN cd /tensorflow && \
+    printf '\n\nn\ny\nn\nn\nn\ny\nn\nn\nn\nn\nn\nn\n\ny\n%s\n\n\n' ${ANDROID_HOME}|./configure
+
+
+WORKDIR /tensorflow

research/object_detection/dockerfiles/android/README.md

@@ -0,0 +1,69 @@
+# Dockerfile for the TPU and TensorFlow Lite Object Detection tutorial
+
+This Docker image automates the setup involved with training
+object detection models on Google Cloud and building the Android TensorFlow Lite
+demo app. We recommend using this container if you decide to work through our
+tutorial on ["Training and serving a real-time mobile object detector in
+30 minutes with Cloud TPUs"](https://medium.com/tensorflow/training-and-serving-a-realtime-mobile-object-detector-in-30-minutes-with-cloud-tpus-b78971cf1193), though of course it may be useful even if you would
+like to use the Object Detection API outside the context of the tutorial.
+
+A couple words of warning:
+
+1. Docker containers do not have persistent storage. This means that any changes
+   you make to files inside the container will not persist if you restart
+   the container. When running through the tutorial,
+   **do not close the container**.
+2. To be able to deploy the [Android app](
+   https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/examples/android/app)
+   (which you will build at the end of the tutorial),
+   you will need to kill any instances of `adb` running on the host machine. You
+   can accomplish this by closing all instances of Android Studio, and then
+   running `adb kill-server`.
+
+You can install Docker by following the [instructions here](
+https://docs.docker.com/install/).
+
+## Running The Container
+
+From this directory, build the Dockerfile as follows (this takes a while):
+
+```
+docker build --tag detect-tf .
+```
+
+Run the container:
+
+```
+docker run --rm -it --privileged -p 6006:6006 detect-tf
+```
+
+When running the container, you will find yourself inside the `/tensorflow`
+directory, which is the path to the TensorFlow [source
+tree](https://github.com/tensorflow/tensorflow).
+
+## Text Editing
+
+The tutorial also
+requires you to occasionally edit files inside the source tree.
+This Docker images comes with `vim`, `nano`, and `emacs` preinstalled for your
+convenience.
+
+## What's In This Container
+
+This container is derived from the nightly build of TensorFlow, and contains the
+sources for TensorFlow at `/tensorflow`, as well as the
+[TensorFlow Models](https://github.com/tensorflow/models) which are available at
+`/tensorflow/models` (and contain the Object Detection API as a subdirectory
+at `/tensorflow/models/research/object_detection`).
+The Oxford-IIIT Pets dataset, the COCO pre-trained SSD + MobileNet (v1)
+checkpoint, and example
+trained model are all available in `/tmp` in their respective folders.
+
+This container also has the `gsutil` and `gcloud` utilities, the `bazel` build
+tool, and all dependencies necessary to use the Object Detection API, and
+compile and install the TensorFlow Lite Android demo app.
+
+At various points throughout the tutorial, you may see references to the
+*research directory*.  This refers to the `research` folder within the
+models repository, located at
+`/tensorflow/models/resesarch`.

research/object_detection/g3doc/detection_model_zoo.md

@@ -1,8 +1,9 @@
 # Tensorflow detection model zoo
 
 We provide a collection of detection models pre-trained on the [COCO
-dataset](http://mscoco.org), the [Kitti dataset](http://www.cvlibs.net/datasets/kitti/), and the
-[Open Images dataset](https://github.com/openimages/dataset). These models can
+dataset](http://mscoco.org), the [Kitti dataset](http://www.cvlibs.net/datasets/kitti/),
+the [Open Images dataset](https://github.com/openimages/dataset) and the
+[AVA v2.1 dataset](https://research.google.com/ava/). These models can
 be useful for
 out-of-the-box inference if you are interested in categories already in COCO
 (e.g., humans, cars, etc) or in Open Images (e.g.,
@@ -57,19 +58,26 @@ Some remarks on frozen inference graphs:
   a detector (and discarding the part past that point), which negatively impacts
   standard mAP metrics.
 * Our frozen inference graphs are generated using the
-  [v1.5.0](https://github.com/tensorflow/tensorflow/tree/v1.5.0)
+  [v1.8.0](https://github.com/tensorflow/tensorflow/tree/v1.8.0)
   release version of Tensorflow and we do not guarantee that these will work
   with other versions; this being said, each frozen inference graph can be
   regenerated using your current version of Tensorflow by re-running the
   [exporter](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/exporting_models.md),
-  pointing it at the model directory as well as the config file inside of it.
+  pointing it at the model directory as well as the corresponding config file in
+  [samples/configs](https://github.com/tensorflow/models/tree/master/research/object_detection/samples/configs).
 
 
-## COCO-trained models {#coco-models}
+## COCO-trained models
 
 | Model name  | Speed (ms) | COCO mAP[^1] | Outputs |
 | ------------ | :--------------: | :--------------: | :-------------: |
 | [ssd_mobilenet_v1_coco](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz) | 30 | 21 | Boxes |
+| [ssd_mobilenet_v1_0.75_depth_coco ☆](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_0.75_depth_300x300_coco14_sync_2018_07_03.tar.gz) | 26 | 18 | Boxes |
+| [ssd_mobilenet_v1_quantized_coco ☆](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_quantized_300x300_coco14_sync_2018_07_03.tar.gz) | 29 | 18 | Boxes |
+| [ssd_mobilenet_v1_0.75_depth_quantized_coco ☆](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_0.75_depth_quantized_300x300_coco14_sync_2018_07_03.tar.gz) | 29 | 16 | Boxes |
+| [ssd_mobilenet_v1_ppn_coco ☆](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_ppn_shared_box_predictor_300x300_coco14_sync_2018_07_03.tar.gz) | 26 | 20 | Boxes |
+| [ssd_mobilenet_v1_fpn_coco ☆](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03.tar.gz) | 56 | 32 | Boxes |
+| [ssd_resnet_50_fpn_coco ☆](http://download.tensorflow.org/models/object_detection/ssd_resnet50_v1_fpn_shared_box_predictor_640x640_coco14_sync_2018_07_03.tar.gz) | 76 | 35 | Boxes |
 | [ssd_mobilenet_v2_coco](http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v2_coco_2018_03_29.tar.gz) | 31 | 22 | Boxes |
 | [ssdlite_mobilenet_v2_coco](http://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz) | 27 | 22 | Boxes |
 | [ssd_inception_v2_coco](http://download.tensorflow.org/models/object_detection/ssd_inception_v2_coco_2018_01_28.tar.gz) | 42 | 24 | Boxes |
@@ -88,29 +96,30 @@ Some remarks on frozen inference graphs:
 | [mask_rcnn_resnet101_atrous_coco](http://download.tensorflow.org/models/object_detection/mask_rcnn_resnet101_atrous_coco_2018_01_28.tar.gz) | 470 | 33 | Masks |
 | [mask_rcnn_resnet50_atrous_coco](http://download.tensorflow.org/models/object_detection/mask_rcnn_resnet50_atrous_coco_2018_01_28.tar.gz) | 343 | 29 | Masks |
 
+Note: The asterisk (☆) at the end of model name indicates that this model supports TPU training.
 
-
-## Kitti-trained models {#kitti-models}
+## Kitti-trained models
 
 Model name                                                                                                                                                        | Speed (ms) | Pascal mAP@0.5 | Outputs
 ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---: | :-------------: | :-----:
 [faster_rcnn_resnet101_kitti](http://download.tensorflow.org/models/object_detection/faster_rcnn_resnet101_kitti_2018_01_28.tar.gz) | 79  | 87              | Boxes
 
-## Open Images-trained models {#open-images-models}
+## Open Images-trained models
 
 Model name                                                                                                                                                        | Speed (ms) | Open Images mAP@0.5[^2] | Outputs
 ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---: | :-------------: | :-----:
 [faster_rcnn_inception_resnet_v2_atrous_oid](http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_resnet_v2_atrous_oid_2018_01_28.tar.gz) | 727 | 37              | Boxes
 [faster_rcnn_inception_resnet_v2_atrous_lowproposals_oid](http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_resnet_v2_atrous_lowproposals_oid_2018_01_28.tar.gz) | 347  |               | Boxes
 
 
-## AVA v2.1 trained models {#ava-models}
+## AVA v2.1 trained models
 
 Model name                                                                                                                                                        | Speed (ms) | Pascal mAP@0.5 | Outputs
 ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | :---: | :-------------: | :-----:
 [faster_rcnn_resnet101_ava_v2.1](http://download.tensorflow.org/models/object_detection/faster_rcnn_resnet101_ava_v2.1_2018_04_30.tar.gz) | 93  | 11              | Boxes
 
 
 [^1]: See [MSCOCO evaluation protocol](http://cocodataset.org/#detections-eval).
+
 [^2]: This is PASCAL mAP with a slightly different way of true positives computation: see [Open Images evaluation protocol](evaluation_protocols.md#open-images).
 

research/object_detection/g3doc/running_locally.md

@@ -34,37 +34,22 @@ A local training job can be run with the following command:
 
 ```bash
 # From the tensorflow/models/research/ directory
-python object_detection/train.py \
-    --logtostderr \
-    --pipeline_config_path=${PATH_TO_YOUR_PIPELINE_CONFIG} \
-    --train_dir=${PATH_TO_TRAIN_DIR}
+PIPELINE_CONFIG_PATH={path to pipeline config file}
+MODEL_DIR={path to model directory}
+NUM_TRAIN_STEPS=50000
+NUM_EVAL_STEPS=2000
+python object_detection/model_main.py \
+    --pipeline_config_path=${PIPELINE_CONFIG_PATH} \
+    --model_dir=${MODEL_DIR} \
+    --num_train_steps=${NUM_TRAIN_STEPS} \
+    --num_eval_steps=${NUM_EVAL_STEPS} \
+    --alsologtostderr
 ```
 
-where `${PATH_TO_YOUR_PIPELINE_CONFIG}` points to the pipeline config and
-`${PATH_TO_TRAIN_DIR}` points to the directory in which training checkpoints
-and events will be written to. By default, the training job will
-run indefinitely until the user kills it.
-
-## Running the Evaluation Job
-
-Evaluation is run as a separate job. The eval job will periodically poll the
-train directory for new checkpoints and evaluate them on a test dataset. The
-job can be run using the following command:
-
-```bash
-# From the tensorflow/models/research/ directory
-python object_detection/eval.py \
-    --logtostderr \
-    --pipeline_config_path=${PATH_TO_YOUR_PIPELINE_CONFIG} \
-    --checkpoint_dir=${PATH_TO_TRAIN_DIR} \
-    --eval_dir=${PATH_TO_EVAL_DIR}
-```
-
-where `${PATH_TO_YOUR_PIPELINE_CONFIG}` points to the pipeline config,
-`${PATH_TO_TRAIN_DIR}` points to the directory in which training checkpoints
-were saved (same as the training job) and `${PATH_TO_EVAL_DIR}` points to the
-directory in which evaluation events will be saved. As with the training job,
-the eval job run until terminated by default.
+where `${PIPELINE_CONFIG_PATH}` points to the pipeline config and
+`${MODEL_DIR}` points to the directory in which training checkpoints
+and events will be written to. Note that this binary will interleave both
+training and evaluation.
 
 ## Running Tensorboard
 
@@ -73,9 +58,9 @@ using the recommended directory structure, Tensorboard can be run using the
 following command:
 
 ```bash
-tensorboard --logdir=${PATH_TO_MODEL_DIRECTORY}
+tensorboard --logdir=${MODEL_DIR}
 ```
 
-where `${PATH_TO_MODEL_DIRECTORY}` points to the directory that contains the
+where `${MODEL_DIR}` points to the directory that contains the
 train and eval directories. Please note it may take Tensorboard a couple minutes
 to populate with data.