Merge main -> google (#3889)

* c801791 Disable WAIT_TYPE_SYNC_FILE until include issues on CI are resolved. (#3884) * b60aaf5 Fix submodule broken by #3854 (#3879) * 539c0f1 Update tensorflow and numpy in docker (#3854) * 9f8cb9d Porting iree::WaitHandle to C and adding Windows support. (#3841) * 15d52b6 Merge pull request #3769 from google/benvanik-tasks-pre * 1d6117c Fix a check bug in doesValueDependOnOperation. (#3861) * 0d90f6f Fixed yaml for synchronize submodules. (#3869) * eb4565a Merge pull request #3866 * 5f40d88 Revert "Merge google -> main (#3863)" (#3864) * b63aaf4 Merge google -> main (#3863) * 116a975 Removing unused tsan macros. * b409ace Adding a thread affinity pinning API. * 7c1545f Adding IREE_TRACE macro. * 2c2898e Exposing iree_hal_buffer_copy_data API. * e26f49e Adding the iree_atomic_*_ptr type. * 5c35b95 Moving errno/win32 error code handing to iree/base/api.h. PiperOrigin-RevId: 342957135
iree-org · Nov 17, 2020 · 00c8230 · 00c8230
2 parents b6eabe4 + c801791
commit 00c8230
Show file tree

Hide file tree

Showing 47 changed files with 3,069 additions and 1,592 deletions.
diff --git a/.github/workflows/update_llvm_dependent_submodules.yml b/.github/workflows/update_llvm_dependent_submodules.yml
@@ -37,7 +37,7 @@ jobs:
       - name: Updating submodules
         run: ./scripts/git/update_to_llvm_syncpoint.py
       - name: Calculating SHAs
-        run:
+        run: |
           echo "LLVM_SHA=$(git submodule status third_party/llvm-project | awk '{print $1}' | cut -c -12)" >> $GITHUB_ENV
           echo "TF_SHA=$(git submodule status third_party/tensorflow | awk '{print $1}' | cut -c -12)" >> $GITHUB_ENV
           echo "LLVM_BAZEL_SHA=$(git submodule status third_party/llvm-bazel | awk '{print $1}' | cut -c -12)" >> $GITHUB_ENV

diff --git a/WORKSPACE b/WORKSPACE
@@ -63,7 +63,7 @@ load("@bazel_toolchains//rules:rbe_repo.bzl", "rbe_autoconfig")
 rbe_autoconfig(
     name = "rbe_default",
     base_container_digest = "sha256:1a8ed713f40267bb51fe17de012fa631a20c52df818ccb317aaed2ee068dfc61",
-    digest = "sha256:417ffe630a3a933b93e6b06aea7d4969fdbc1b120e26e674be108b8a849e0e8c",
+    digest = "sha256:d6d895294076b5289e81489f664656211c41656cffe7c448ecb5c6f54f045974",
     registry = "gcr.io",
     repository = "iree-oss/rbe-toolchain",
     use_checked_in_confs = "Force",

diff --git a/build_tools/docker/README.md b/build_tools/docker/README.md
@@ -69,38 +69,97 @@ python3 build_tools/docker/manage_images.py --images all --tag latest --update_r
 This requires that the tagged image have a repository digest, which means it was
 pushed to or pulled from GCR.
 
-## Deploying New Images
+## Adding or Updating an Image
 
-1.  Modify the Dockerfiles as desired.
-2.  Update `manage_images.py` to include the new image and its dependencies.
-3.  Build and push the new image to GCR and update references to it:
+If you have worked with the `docker` images before, it is prudent to follow the
+steps in the "Debugging" section below before continuing.
 
-    ```shell
-    python3 build_tools/docker/manage_images.py --image "${IMAGE?}" --build --push --update_references
+### Part 1. Local Changes
+
+1. Update the `Dockerfile` for the image that you want to modify or add. If
+   you're adding a new image, or updating the dependencies between images, be
+   sure to update `IMAGES_TO_DEPENDENCIES` in `manage_images.py` as well.
+2. Build the image, push the image to GCR and update all references to the image
+   with the new GCR digest:
+
+   ```shell
+    python3 build_tools/docker/manage_images.py \
+      --image "${IMAGE?}" --build \
+      --tag latest \
+      --push \
+      --update_references
     ```
 
-4.  Commit changes and send a PR for review.
+3. Test that the changes behave as expected locally and iterate on the steps
+   above.
 
-5.  Merge your PR after is approved and all builds pass.
+### Part 2. Submitting to GitHub
 
-6.  Kokoro builds preload images tagged with `prod` on VM creation, so after
-    changing the images used, you should also update the images tagged as `prod`
-    in GCR. Update your local reference to the `prod` tag to point at the new
-    image:
+4. Commit the changes and send a PR for review. The CI will use the updated
+   digest references to test the new images.
 
-    ```shell
-    python3 build_tools/docker/manage_images.py --image "${IMAGE?}" --tag prod --build --update_references
+5. Merge your PR after is approved and all CI tests pass. **Please remember to
+   complete the rest of the steps below**.
+
+### Part 3. Updating the `:prod` tag
+
+Kokoro builds preload images tagged with `prod` on VM creation, so after
+changing the images used, you should also update the images tagged as `prod`
+in GCR. This also makes development significantly easier for others who need to
+modify the `docker` images.
+
+6. On the `main` branch, build (but don't push) the images and locally tag them
+   with the `:prod` tag:
+
+   ```shell
+    python3 build_tools/docker/manage_images.py \
+      --image "${IMAGE?}" --build \
+      --tag prod \
+      --update_references
     ```
 
-    The build steps here should all be cache hits and no references should
-    actually be changed. If they are, that indicates the images you've just
-    built are different from the ones that are being referenced. Stop and fix
-    this before proceeding. This relies on you keeping your local copy of the
-    Docker images. If you didn't, you'll have to manually pull the missing
-    images by their digest.
+    This build should be entirely cache hits.
+7. We include `--update_references` in the command above so that we can check
+   that none of the images or references to them have been changed. Check that
+   the following command produces no output before continuing:
+
+   ```shell
+   git status --porcelain
+   ```
 
-7.  Push the new images with the `prod` tag to GCR.
+   If the output is not empty then you'll need to find the source of the
+   discrepancy (e.g. a locally modified `Dockerfile`) and remove it, and repeat
+   steps 5 and 6 before continuing. (This relies on you keeping your local copy
+   of the Docker images. If you didn't, you'll have to manually pull the missing
+   images by their digest).
+8. Now that we've confirmed that none of the images were changed, we can push
+   them to GCR with the `:prod` tag.
 
     ```shell
-    python3 build_tools/docker/manage_images.py --image "${IMAGE?}" --tag prod --push
+    python3 build_tools/docker/manage_images.py \
+      --image "${IMAGE?}" \
+      --tag prod \
+      --push
     ```
+
+## Debugging
+
+Sometimes old versions of the `:latest` images can be stored locally and produce
+unexpected behaviors. The following commands will download all of the prod
+images and then update the images tagged with `:latest` on your machine (and on
+GCR).
+
+```shell
+# Pull all :prod images
+python3 build_tools/docker/manage_images.py --images all --pull --tag prod
+# Update the :latest images to match the :prod images.
+# If you have a clean workspace this _shouldn't_ require building anything as
+# everything should be cache hits from the :prod images downloaded above, but if
+# the :prod images are behind then that will not be the case and this may take
+# several hours (depending on your machine).
+python3 build_tools/docker/manage_images.py \
+  --images all --build \
+  --tag latest \
+  --push \
+  --update_references
+```
diff --git a/build_tools/docker/bazel-python/Dockerfile b/build_tools/docker/bazel-python/Dockerfile
@@ -24,6 +24,6 @@ RUN apt-get update \
     python3-pip \
     python3-setuptools \
   && python3 -m pip install --upgrade pip \
-  && python3 -m pip install numpy
+  && python3 -m pip install numpy==1.19.4
 
 ENV PYTHON_BIN /usr/bin/python3
diff --git a/build_tools/docker/bazel-tensorflow/Dockerfile b/build_tools/docker/bazel-tensorflow/Dockerfile
@@ -17,4 +17,4 @@
 FROM gcr.io/iree-oss/bazel-python AS final
 
 # Install tensorflow.
-RUN python3 -m pip install tf-nightly==2.4.0.dev20200902
+RUN python3 -m pip install tf-nightly==2.5.0.dev20201116
diff --git a/build_tools/docker/cmake-python/Dockerfile b/build_tools/docker/cmake-python/Dockerfile
@@ -14,10 +14,6 @@
 
 # An image for building IREE and its Python bindings using CMake.
 
-ARG CMAKE_MAJOR_VERSION=3
-ARG CMAKE_MINOR_VERSION=13
-ARG CMAKE_PATCH_VERSION=5
-
 FROM gcr.io/iree-oss/cmake AS final
 # Dependencies for the python bindings tests.
 RUN apt-get update \
@@ -26,6 +22,6 @@ RUN apt-get update \
     python3-pip \
     python3-setuptools \
   && python3 -m pip install --upgrade pip \
-  && python3 -m pip install numpy absl-py
+  && python3 -m pip install numpy==1.19.4 absl-py
 
 ENV PYTHON_BIN /usr/bin/python3
diff --git a/build_tools/docker/manage_images.py b/build_tools/docker/manage_images.py
@@ -17,6 +17,8 @@
 
 Includes information on their dependency graph and GCR URL.
 
+See the README for information on how to add and update images.
+
 Example usage:
 
 Rebuild the cmake image and all images that transitively on depend on it,

diff --git a/build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-swiftshader/bindings/build_kokoro.sh b/build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-swiftshader/bindings/build_kokoro.sh
@@ -32,7 +32,7 @@ source "${KOKORO_ARTIFACTS_DIR?}/github/iree/build_tools/kokoro/gcp_ubuntu/docke
 docker_setup
 
 docker run "${DOCKER_RUN_ARGS[@]?}" \
-  gcr.io/iree-oss/bazel-python@sha256:27d6222f5b8f1bc5dc55e003bd32dac1f85ed3e0b81dcd22ba2039dbbf546d9b \
+  gcr.io/iree-oss/bazel-python@sha256:473b7e294136bc38abc1941042f0c0404199de5827f141520f0b6757305b7a95 \
   build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-swiftshader/bindings/build.sh
 
 # Kokoro will rsync this entire directory back to the executor orchestrating the

diff --git a/build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-swiftshader/core/build_kokoro.sh b/build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-swiftshader/core/build_kokoro.sh
@@ -32,7 +32,7 @@ source "${KOKORO_ARTIFACTS_DIR?}/github/iree/build_tools/kokoro/gcp_ubuntu/docke
 docker_setup
 
 docker run "${DOCKER_RUN_ARGS[@]?}" \
-  gcr.io/iree-oss/bazel@sha256:ed7a2a729db3d83549180b74f6cc0f5c21ec7f9569ee56f4b04bba5a8b19d1c7 \
+  gcr.io/iree-oss/bazel@sha256:59da17e5cc8176890a6e1bda369b1f3d398e27af3d47e02e1ffd5b76729c215b \
   build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-swiftshader/core/build.sh
 
 # Kokoro will rsync this entire directory back to the executor orchestrating the

diff --git a/build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-swiftshader/integrations/build_kokoro.sh b/build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-swiftshader/integrations/build_kokoro.sh
@@ -32,7 +32,7 @@ source "${KOKORO_ARTIFACTS_DIR?}/github/iree/build_tools/kokoro/gcp_ubuntu/docke
 docker_setup
 
 docker run "${DOCKER_RUN_ARGS[@]?}" \
-  gcr.io/iree-oss/bazel-tensorflow-swiftshader@sha256:df2092f3f84f416e7b2b1379f9f6c147919ad68972bb419077274502cf3477ca \
+  gcr.io/iree-oss/bazel-tensorflow-swiftshader@sha256:39c0e43c503bddfacd69758a50f02450ad2322d35324e2f56997aebb33a1b20a \
   build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-swiftshader/integrations/build.sh
 
 # Kokoro will rsync this entire directory back to the executor orchestrating the

diff --git a/build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-turing/integrations/build_kokoro.sh b/build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-turing/integrations/build_kokoro.sh
@@ -36,7 +36,7 @@ docker_setup
 # TODO(#3550): Allow this to follow the checked-in Docker hierarchy.
 docker run "${DOCKER_RUN_ARGS[@]?}" \
   --gpus all \
-  gcr.io/iree-oss/bazel-tensorflow-nvidia@sha256:24bb83d04ed8b590accf5c2fed0f1d8ed0c2c631b60cc533bb90c80b1f21e27f \
+  gcr.io/iree-oss/bazel-tensorflow-nvidia@sha256:e5e96ec1709e83355ee2264c97c26fa5c3d40f749a62734f4787b17a83f2c3b8 \
   build_tools/kokoro/gcp_ubuntu/bazel/linux/x86-turing/integrations/build.sh
 
 # Kokoro will rsync this entire directory back to the executor orchestrating the

diff --git a/build_tools/kokoro/gcp_ubuntu/cmake/android/arm64-v8a/build_kokoro.sh b/build_tools/kokoro/gcp_ubuntu/cmake/android/arm64-v8a/build_kokoro.sh
@@ -32,7 +32,7 @@ source "${KOKORO_ARTIFACTS_DIR?}/github/iree/build_tools/kokoro/gcp_ubuntu/docke
 docker_setup
 
 docker run "${DOCKER_RUN_ARGS[@]?}" \
-  gcr.io/iree-oss/cmake-android@sha256:3b9de9f956df8b9805d84b5aa6ca400a366c8f6c11375a25d676e3c11329c5a9 \
+  gcr.io/iree-oss/cmake-android@sha256:7accda0b84e2ae337740f2ee71801ee30f2155900abf1cf7b73ea47c15dc694f \
   build_tools/kokoro/gcp_ubuntu/cmake/android/build.sh arm64-v8a
 
 # Kokoro will rsync this entire directory back to the executor orchestrating the

diff --git a/build_tools/kokoro/gcp_ubuntu/cmake/linux/x86-swiftshader/build_kokoro.sh b/build_tools/kokoro/gcp_ubuntu/cmake/linux/x86-swiftshader/build_kokoro.sh
@@ -32,7 +32,7 @@ source "${KOKORO_ARTIFACTS_DIR?}/github/iree/build_tools/kokoro/gcp_ubuntu/docke
 docker_setup
 
 docker run "${DOCKER_RUN_ARGS[@]?}" \
-  gcr.io/iree-oss/cmake-python-swiftshader@sha256:b58b914fed19512c9f4c0013e933fe9e5097f3ecd3c5413f5de3d9ed1511dfaa \
+  gcr.io/iree-oss/cmake-python-swiftshader@sha256:3e3d3427f3a58b32fa3ed578b610e411e0b81fd0e1984ac9b0fceae8bf8343dc \
   build_tools/kokoro/gcp_ubuntu/cmake/linux/x86-swiftshader/build.sh
 
 # Kokoro will rsync this entire directory back to the executor orchestrating the

diff --git a/build_tools/kokoro/gcp_ubuntu/cmake/linux/x86-turing/build_kokoro.sh b/build_tools/kokoro/gcp_ubuntu/cmake/linux/x86-turing/build_kokoro.sh
@@ -33,7 +33,7 @@ docker_setup
 
 docker run "${DOCKER_RUN_ARGS[@]?}" \
   --gpus all \
-  gcr.io/iree-oss/cmake-python-nvidia@sha256:449f619f6ac478bbc80bfe78195c63f75789331a7bfbc857863ba930c485936b \
+  gcr.io/iree-oss/cmake-python-nvidia@sha256:310e3b399717905bb2b485f3ebed32222915c7dc4dc075aa4e1b8551101fe607 \
   build_tools/kokoro/gcp_ubuntu/cmake/linux/x86-turing/build.sh
 
 # Kokoro will rsync this entire directory back to the executor orchestrating the

diff --git a/integrations/tensorflow/e2e/keras/layers/BUILD b/integrations/tensorflow/e2e/keras/layers/BUILD
@@ -287,6 +287,7 @@ FAILING_FULL_API = [
             "LSTM",
             "MaxPool1D",
             "MaxPool3D",
+            "SeparableConv1D",
             "SimpleRNN",
         ],
         "target_backends": "tflite",

diff --git a/iree/base/BUILD b/iree/base/BUILD
@@ -534,34 +534,34 @@ cc_library(
     ],
 )
 
-# Dependent code has been removed and wait_handle is currently incompatible
-# with Windows, so excluding entirely.
-# See google/iree/65
-# cc_library(
-#     name = "wait_handle",
-#     srcs = ["wait_handle.cc"],
-#     hdrs = ["wait_handle.h"],
-#     deps = [
-#         ":logging",
-#         ":ref_ptr",
-#         ":status",
-#         ":time",
-#         "@com_google_absl//absl/base:core_headers",
-#         "@com_google_absl//absl/container:fixed_array",
-#         "@com_google_absl//absl/strings",
-#         "@com_google_absl//absl/time",
-#         "@com_google_absl//absl/types:span",
-#     ],
-# )
-
-# cc_test(
-#     name = "wait_handle_test",
-#     srcs = ["wait_handle_test.cc"],
-#     deps = [
-#         ":status",
-#         ":wait_handle",
-#         "@com_google_absl//absl/time",
-#         "//iree/testing:gtest",
-#         "//iree/testing:gtest_main",
-#     ],
-# )
+cc_library(
+    name = "wait_handle",
+    srcs = [
+        "wait_handle.c",
+        "wait_handle_epoll.c",
+        "wait_handle_kqueue.c",
+        "wait_handle_poll.c",
+        "wait_handle_posix.c",
+        "wait_handle_posix.h",
+        "wait_handle_win32.c",
+    ],
+    hdrs = ["wait_handle.h"],
+    copts = [
+        "-D_GNU_SOURCE=1",
+    ],
+    deps = [
+        ":api",
+        ":target_platform",
+        ":tracing",
+    ],
+)
+
+cc_test(
+    name = "wait_handle_test",
+    srcs = ["wait_handle_test.cc"],
+    deps = [
+        ":wait_handle",
+        "//iree/testing:gtest",
+        "//iree/testing:gtest_main",
+    ],
+)
diff --git a/iree/base/CMakeLists.txt b/iree/base/CMakeLists.txt
@@ -636,36 +636,35 @@ else()
   )
 endif()
 
-# TODO(benvanik): get wait_handle ported to win32.
-# iree_cc_library(
-#   NAME
-#     wait_handle
-#   HDRS
-#     "wait_handle.h"
-#   SRCS
-#     "wait_handle.cc"
-#   DEPS
-#     absl::base
-#     absl::fixed_array
-#     absl::span
-#     absl::strings
-#     absl::time
-#     iree::base::logging
-#     iree::base::ref_ptr
-#     iree::base::status
-#     iree::base::time
-#   PUBLIC
-# )
-#
-# iree_cc_test(
-#   NAME
-#     wait_handle_test
-#   SRCS
-#     "wait_handle_test.cc"
-#   DEPS
-#     absl::time
-#     iree::base::status
-#     iree::base::wait_handle
-#     iree::testing::gtest
-#     iree::testing::gtest_main
-# )
+iree_cc_library(
+  NAME
+    wait_handle
+  HDRS
+    "wait_handle.h"
+  SRCS
+    "wait_handle.c"
+    "wait_handle_epoll.c"
+    "wait_handle_kqueue.c"
+    "wait_handle_poll.c"
+    "wait_handle_posix.c"
+    "wait_handle_posix.h"
+    "wait_handle_win32.c"
+  COPTS
+    "-D_GNU_SOURCE=1"
+  DEPS
+    ::api
+    ::target_platform
+    ::tracing
+  PUBLIC
+)
+
+iree_cc_test(
+  NAME
+    wait_handle_test
+  SRCS
+    "wait_handle_test.cc"
+  DEPS
+    ::wait_handle
+    iree::testing::gtest
+    iree::testing::gtest_main
+)