Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Examples Timeout #2

Closed
joaogui1 opened this issue Jun 28, 2021 · 5 comments
Closed

Examples Timeout #2

joaogui1 opened this issue Jun 28, 2021 · 5 comments

Comments

@joaogui1
Copy link

joaogui1 commented Jun 28, 2021 •
edited

Hi! I'm trying to use xmanager and while the setup went well all of the examples are timing out before even running the network. Any ideas what the error could be?

cifar10 pytorch log 
starting build "518b658c-c8ae-4595-9058-95eea6cdcaa5"

FETCHSOURCE
Fetching storage object: gs://revirainbow_bucket2/cifar10_torch-latest.tar.gz#1624743987085175
Copying gs://revirainbow_bucket2/cifar10_torch-latest.tar.gz#1624743987085175...
/ [0 files][    0.0 B/  7.1 KiB]                                                
/ [1 files][  7.1 KiB/  7.1 KiB]                                                
Operation completed over 1 objects/7.1 KiB.                                      
tar: Removing leading `/' from member names
BUILD
Pulling image: gcr.io/kaniko-project/executor:latest
latest: Pulling from kaniko-project/executor
Digest: sha256:6ecc43ae139ad8cfa11604b592aaedddcabff8cef469eda303f1fb5afe5e3034
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
gcr.io/kaniko-project/executor:latest
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-gpu.1-6 
�[36mINFO�[0m[0000] Retrieving image gcr.io/deeplearning-platform-release/pytorch-gpu.1-6 from registry gcr.io 
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-gpu.1-6 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Built cross stage deps: map[]                
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-gpu.1-6 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-gpu.1-6 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Executing 0 build triggers                   
�[36mINFO�[0m[0000] Checking for cached layer gcr.io/researchprojects-msc/cifar10_torch/cache:f6b49a2c721c492debdfe49e26c8073947a7c2f39c82709a8463ca794242a13b... 
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0001] No cached layer found for cmd RUN apt-get update && apt-get install -y git 
�[36mINFO�[0m[0001] Unpacking rootfs as cmd RUN apt-get update && apt-get install -y git requires it. 
�[36mINFO�[0m[0502] ENV LANG=C.UTF-8                             
�[36mINFO�[0m[0502] No files changed in this command, skipping snapshotting. 
�[36mINFO�[0m[0502] RUN apt-get update && apt-get install -y git 
�[36mINFO�[0m[0502] Taking snapshot of full filesystem...        
�[36mINFO�[0m[0811] cmd: /bin/sh                                 
�[36mINFO�[0m[0811] args: [-c apt-get update && apt-get install -y git] 
�[36mINFO�[0m[0811] Running: [/bin/sh -c apt-get update && apt-get install -y git] 
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:2 http://packages.cloud.google.com/apt gcsfuse-bionic InRelease [5385 B]
Get:3 http://packages.cloud.google.com/apt cloud-sdk-bionic InRelease [6780 B]
Ign:4 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease
Ign:6 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
Get:7 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release [697 B]
Get:8 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release [564 B]
Get:9 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release.gpg [836 B]
Err:2 http://packages.cloud.google.com/apt gcsfuse-bionic InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FEEA9169307EA071 NO_PUBKEY 8B57C5C2836F4BEB
Get:10 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release.gpg [833 B]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [473 kB]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2220 kB]
Err:3 http://packages.cloud.google.com/apt cloud-sdk-bionic InRelease
  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FEEA9169307EA071 NO_PUBKEY 8B57C5C2836F4BEB
Get:14 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [24.7 kB]
Get:15 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1418 kB]
Get:16 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Ign:17 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages
Get:17 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages [599 kB]
Get:18 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Packages [73.8 kB]
Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [505 kB]
Get:20 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2188 kB]
Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [33.5 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2653 kB]
Fetched 10.5 MB in 2s (4910 kB/s)
Reading package lists...
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://packages.cloud.google.com/apt gcsfuse-bionic InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FEEA9169307EA071 NO_PUBKEY 8B57C5C2836F4BEB
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://packages.cloud.google.com/apt cloud-sdk-bionic InRelease: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FEEA9169307EA071 NO_PUBKEY 8B57C5C2836F4BEB
W: Failed to fetch http://packages.cloud.google.com/apt/dists/gcsfuse-bionic/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FEEA9169307EA071 NO_PUBKEY 8B57C5C2836F4BEB
W: Failed to fetch http://packages.cloud.google.com/apt/dists/cloud-sdk-bionic/InRelease  The following signatures couldn't be verified because the public key is not available: NO_PUBKEY FEEA9169307EA071 NO_PUBKEY 8B57C5C2836F4BEB
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
Suggested packages:
  gettext-base git-daemon-run | git-daemon-sysvinit git-doc git-el git-email
  git-gui gitk gitweb git-cvs git-mediawiki git-svn
The following packages will be upgraded:
  git
1 upgraded, 0 newly installed, 0 to remove and 104 not upgraded.
Need to get 3916 kB of archives.
After this operation, 8192 B of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 3916 kB in 1s (4849 kB/s)
(Reading database ... 
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 91467 files and directories currently installed.)
Preparing to unpack .../git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...
Unpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...
Setting up git (1:2.17.1-1ubuntu0.8) ...
�[36mINFO�[0m[0819] Taking snapshot of full filesystem...        
�[36mINFO�[0m[1109] RUN python -m pip install --upgrade pip      
�[36mINFO�[0m[1109] cmd: /bin/sh                                 
�[36mINFO�[0m[1109] args: [-c python -m pip install --upgrade pip] 
�[36mINFO�[0m[1109] Running: [/bin/sh -c python -m pip install --upgrade pip] 
�[36mINFO�[0m[1109] Pushing layer gcr.io/researchprojects-msc/cifar10_torch/cache:f6b49a2c721c492debdfe49e26c8073947a7c2f39c82709a8463ca794242a13b to cache now 
�[36mINFO�[0m[1109] GET KEYCHAIN                                 
�[36mINFO�[0m[1110] Pushing image to gcr.io/researchprojects-msc/cifar10_torch/cache:f6b49a2c721c492debdfe49e26c8073947a7c2f39c82709a8463ca794242a13b 
Collecting pip
  Downloading pip-21.1.3-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.4
    Uninstalling pip-20.2.4:
�[36mINFO�[0m[1113] Pushed image to 1 destinations               
      Successfully uninstalled pip-20.2.4
Successfully installed pip-21.1.3
�[36mINFO�[0m[1115] Taking snapshot of full filesystem...        
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor:latest" failed: step exited with non-zero status: 2
Tensorflow take 1 
starting build "d153cbb4-f4f5-48f2-a935-40d4e6f47584"

FETCHSOURCE
Fetching storage object: gs://revirainbow_bucket2/cifar10_tensorflow-latest.tar.gz#1624821333936417
Copying gs://revirainbow_bucket2/cifar10_tensorflow-latest.tar.gz#1624821333936417...
/ [0 files][    0.0 B/  5.6 KiB]                                                
/ [1 files][  5.6 KiB/  5.6 KiB]                                                
Operation completed over 1 objects/5.6 KiB.                                      
tar: Removing leading `/' from member names
BUILD
Pulling image: gcr.io/kaniko-project/executor:latest
latest: Pulling from kaniko-project/executor
Digest: sha256:6ecc43ae139ad8cfa11604b592aaedddcabff8cef469eda303f1fb5afe5e3034
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
gcr.io/kaniko-project/executor:latest
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-gpu.2-1 
�[36mINFO�[0m[0000] Retrieving image gcr.io/deeplearning-platform-release/tf2-gpu.2-1 from registry gcr.io 
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-gpu.2-1 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Built cross stage deps: map[]                
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-gpu.2-1 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-gpu.2-1 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Executing 0 build triggers                   
�[36mINFO�[0m[0000] Checking for cached layer gcr.io/researchprojects-msc/cifar10_tensorflow/cache:331fd6441d19d384b5d8f21997642529c44fad394563eff5b2843bd14dae0f7d... 
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] No cached layer found for cmd RUN apt-get update && apt-get install -y git 
�[36mINFO�[0m[0000] Unpacking rootfs as cmd RUN apt-get update && apt-get install -y git requires it. 
�[36mINFO�[0m[0403] ENV LANG=C.UTF-8                             
�[36mINFO�[0m[0403] No files changed in this command, skipping snapshotting. 
�[36mINFO�[0m[0403] RUN apt-get update && apt-get install -y git 
�[36mINFO�[0m[0403] Taking snapshot of full filesystem...        
�[36mINFO�[0m[0662] cmd: /bin/sh                                 
�[36mINFO�[0m[0662] args: [-c apt-get update && apt-get install -y git] 
�[36mINFO�[0m[0662] Running: [/bin/sh -c apt-get update && apt-get install -y git] 
Get:1 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  InRelease
Get:3 http://archive.ubuntu.com/ubuntu bionic InRelease [242 kB]
Ign:4 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  InRelease
Get:5 http://packages.cloud.google.com/apt gcsfuse-bionic InRelease [5385 B]
Get:6 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release [697 B]
Get:7 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release [564 B]
Get:8 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Release.gpg [836 B]
Get:9 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Release.gpg [833 B]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [24.7 kB]
Get:11 http://packages.cloud.google.com/apt cloud-sdk-bionic InRelease [6780 B]
Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [473 kB]
Get:13 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2220 kB]
Get:14 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1418 kB]
Get:15 http://packages.cloud.google.com/apt gcsfuse-bionic/main amd64 Packages [339 B]
Get:16 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:17 https://packages.cloud.google.com/apt google-fast-socket InRelease [5405 B]
Get:18 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Ign:19 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages
Get:19 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64  Packages [599 kB]
Get:20 https://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64  Packages [73.8 kB]
Get:21 http://packages.cloud.google.com/apt cloud-sdk-bionic/main amd64 Packages [191 kB]
Get:22 http://archive.ubuntu.com/ubuntu bionic/restricted amd64 Packages [13.5 kB]
Get:23 https://packages.cloud.google.com/apt google-fast-socket/main amd64 Packages [431 B]
Get:24 http://archive.ubuntu.com/ubuntu bionic/universe amd64 Packages [11.3 MB]
Get:25 http://archive.ubuntu.com/ubuntu bionic/multiverse amd64 Packages [186 kB]
Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 Packages [1344 kB]
Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [505 kB]
Get:28 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [33.5 kB]
Get:29 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2188 kB]
Get:30 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2653 kB]
Get:31 http://archive.ubuntu.com/ubuntu bionic-backports/main amd64 Packages [11.3 kB]
Get:32 http://archive.ubuntu.com/ubuntu bionic-backports/universe amd64 Packages [11.4 kB]
Fetched 23.8 MB in 3s (6929 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
git is already the newest version (1:2.17.1-1ubuntu0.8).
0 upgraded, 0 newly installed, 0 to remove and 17 not upgraded.
�[36mINFO�[0m[0669] Taking snapshot of full filesystem...        
�[36mINFO�[0m[0907] Pushing layer gcr.io/researchprojects-msc/cifar10_tensorflow/cache:331fd6441d19d384b5d8f21997642529c44fad394563eff5b2843bd14dae0f7d to cache now 
�[36mINFO�[0m[0907] GET KEYCHAIN                                 
�[36mINFO�[0m[0907] RUN python -m pip install --upgrade pip      
�[36mINFO�[0m[0907] cmd: /bin/sh                                 
�[36mINFO�[0m[0907] args: [-c python -m pip install --upgrade pip] 
�[36mINFO�[0m[0907] Running: [/bin/sh -c python -m pip install --upgrade pip] 
�[36mINFO�[0m[0907] Pushing image to gcr.io/researchprojects-msc/cifar10_tensorflow/cache:331fd6441d19d384b5d8f21997642529c44fad394563eff5b2843bd14dae0f7d 
Requirement already satisfied: pip in /opt/conda/lib/python3.7/site-packages (21.1.2)
Collecting pip
  Downloading pip-21.1.3-py3-none-any.whl (1.5 MB)
�[36mINFO�[0m[0910] Pushed image to 1 destinations               
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.2
    Uninstalling pip-21.1.2:
      Successfully uninstalled pip-21.1.2
Successfully installed pip-21.1.3
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
�[36mINFO�[0m[0912] Taking snapshot of full filesystem...        
�[36mINFO�[0m[1148] Pushing layer gcr.io/researchprojects-msc/cifar10_tensorflow/cache:0466755a7b51465b8a5ebf0a031a24056603ae4c48f3aa8abca6faf15373ca69 to cache now 
�[36mINFO�[0m[1148] COPY cifar10_tensorflow/requirements.txt cifar10_tensorflow/requirements.txt 
�[36mINFO�[0m[1148] Taking snapshot of files...                  
�[36mINFO�[0m[1148] GET KEYCHAIN                                 
�[36mINFO�[0m[1148] RUN python -m pip install -r cifar10_tensorflow/requirements.txt 
�[36mINFO�[0m[1148] cmd: /bin/sh                                 
�[36mINFO�[0m[1148] args: [-c python -m pip install -r cifar10_tensorflow/requirements.txt] 
�[36mINFO�[0m[1148] Running: [/bin/sh -c python -m pip install -r cifar10_tensorflow/requirements.txt] 
�[36mINFO�[0m[1148] Pushing image to gcr.io/researchprojects-msc/cifar10_tensorflow/cache:0466755a7b51465b8a5ebf0a031a24056603ae4c48f3aa8abca6faf15373ca69 
Requirement already satisfied: absl-py in /opt/conda/lib/python3.7/site-packages (from -r cifar10_tensorflow/requirements.txt (line 1)) (0.8.1)
Requirement already satisfied: tensorflow in /opt/conda/lib/python3.7/site-packages (from -r cifar10_tensorflow/requirements.txt (line 2)) (2.1.4)
Requirement already satisfied: tensorflow-datasets in /opt/conda/lib/python3.7/site-packages (from -r cifar10_tensorflow/requirements.txt (line 3)) (2.0.0)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py->-r cifar10_tensorflow/requirements.txt (line 1)) (1.16.0)
Requirement already satisfied: wheel>=0.26 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.36.2)
Requirement already satisfied: h5py<=2.10.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2.10.0)
Requirement already satisfied: tensorflow-estimator<2.2.0,>=2.1.0rc0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2.1.0)
Requirement already satisfied: grpcio>=1.8.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.38.0)
Requirement already satisfied: google-pasta>=0.1.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.2.0)
Requirement already satisfied: astor>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.8.1)
Requirement already satisfied: keras-applications>=1.0.8 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.0.8)
Requirement already satisfied: gast==0.2.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.2.2)
Requirement already satisfied: opt-einsum>=2.3.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.3.0)
Requirement already satisfied: termcolor>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.1.0)
Requirement already satisfied: wrapt>=1.11.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.12.1)
Requirement already satisfied: protobuf>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.16.0)
Collecting numpy<1.19.0,>=1.16.0
  Downloading numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB)
�[36mINFO�[0m[1150] Pushed image to 1 destinations               
Requirement already satisfied: keras-preprocessing==1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.1.0)
Collecting tensorboard<2.2.0,>=2.1.0
  Downloading tensorboard-2.1.1-py3-none-any.whl (3.8 MB)
Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2.0.1)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.4.4)
Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.3.4)
Requirement already satisfied: requests<3,>=2.21.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2.25.1)
Requirement already satisfied: google-auth<2,>=1.6.3 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.30.2)
Requirement already satisfied: setuptools>=41.0.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (49.6.0.post20210108)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.2.7)
Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (4.7.2)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (4.2.2)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.3.0)
Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (4.5.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.4.8)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2.10)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (4.0.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.26.5)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2021.5.30)
Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.1.1)
Requirement already satisfied: tqdm in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (4.61.1)
Requirement already satisfied: promise in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (2.3)
Requirement already satisfied: dill in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (0.3.0)
Requirement already satisfied: attrs>=18.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (21.2.0)
Requirement already satisfied: tensorflow-metadata in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (0.21.2)
Requirement already satisfied: future in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (0.18.2)
Requirement already satisfied: typing-extensions>=3.6.4 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.10.0.0)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.4.1)
Requirement already satisfied: googleapis-common-protos in /opt/conda/lib/python3.7/site-packages (from tensorflow-metadata->tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (1.53.0)
Installing collected packages: numpy, tensorboard
  Attempting uninstall: numpy
    Found existing installation: numpy 1.19.5
    Uninstalling numpy-1.19.5:
      Successfully uninstalled numpy-1.19.5
  Attempting uninstall: tensorboard
    Found existing installation: tensorboard 2.5.0
    Uninstalling tensorboard-2.5.0:
      Successfully uninstalled tensorboard-2.5.0
Successfully installed numpy-1.18.5 tensorboard-2.1.1
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tfx-bsl 0.21.4 requires google-api-python-client<2,>=1.7.11, but you have google-api-python-client 2.9.0 which is incompatible.
tfx-bsl 0.21.4 requires pyarrow<0.16.0,>=0.15.0, but you have pyarrow 4.0.1 which is incompatible.
tensorflow-model-analysis 0.21.6 requires pyarrow<1,>=0.15, but you have pyarrow 4.0.1 which is incompatible.
tensorflow-model-analysis 0.21.6 requires scipy==1.4.1; python_version >= "3", but you have scipy 1.6.3 which is incompatible.
tensorflow-io 0.11.0 requires tensorflow==2.1.0, but you have tensorflow 2.1.4 which is incompatible.
tensorflow-data-validation 0.21.5 requires joblib<0.15,>=0.12, but you have joblib 1.0.1 which is incompatible.
tensorflow-data-validation 0.21.5 requires pandas<1,>=0.24, but you have pandas 1.2.4 which is incompatible.
tensorflow-data-validation 0.21.5 requires scikit-learn<0.22,>=0.18, but you have scikit-learn 0.24.2 which is incompatible.
tensorflow-cloud 0.1.13 requires tensorboard>=2.3.0, but you have tensorboard 2.1.1 which is incompatible.
apache-beam 2.17.0 requires httplib2<=0.12.0,>=0.8, but you have httplib2 0.19.1 which is incompatible.
apache-beam 2.17.0 requires pyarrow<0.16.0,>=0.15.1; python_version >= "3.0" or platform_system != "Windows", but you have pyarrow 4.0.1 which is incompatible.
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
�[36mINFO�[0m[1155] Taking snapshot of full filesystem...        
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor:latest" failed: step exited with non-zero status: 2
Tensorflow take 2 
starting build "29331116-f1e7-4f16-b5e4-6d40b248d099"

FETCHSOURCE
Fetching storage object: gs://revirainbow_bucket2/cifar10_tensorflow-latest.tar.gz#1624822680512194
Copying gs://revirainbow_bucket2/cifar10_tensorflow-latest.tar.gz#1624822680512194...
/ [0 files][    0.0 B/  5.6 KiB]                                                
/ [1 files][  5.6 KiB/  5.6 KiB]                                                
Operation completed over 1 objects/5.6 KiB.                                      
tar: Removing leading `/' from member names
BUILD
Pulling image: gcr.io/kaniko-project/executor:latest
latest: Pulling from kaniko-project/executor
Digest: sha256:6ecc43ae139ad8cfa11604b592aaedddcabff8cef469eda303f1fb5afe5e3034
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
gcr.io/kaniko-project/executor:latest
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-gpu.2-1 
�[36mINFO�[0m[0000] Retrieving image gcr.io/deeplearning-platform-release/tf2-gpu.2-1 from registry gcr.io 
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-gpu.2-1 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Built cross stage deps: map[]                
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-gpu.2-1 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/tf2-gpu.2-1 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Executing 0 build triggers                   
�[36mINFO�[0m[0000] Checking for cached layer gcr.io/researchprojects-msc/cifar10_tensorflow/cache:331fd6441d19d384b5d8f21997642529c44fad394563eff5b2843bd14dae0f7d... 
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] Using caching version of cmd: RUN apt-get update && apt-get install -y git 
�[36mINFO�[0m[0000] Checking for cached layer gcr.io/researchprojects-msc/cifar10_tensorflow/cache:0466755a7b51465b8a5ebf0a031a24056603ae4c48f3aa8abca6faf15373ca69... 
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0001] Using caching version of cmd: RUN python -m pip install --upgrade pip 
�[36mINFO�[0m[0001] Checking for cached layer gcr.io/researchprojects-msc/cifar10_tensorflow/cache:dde4d6174b68d54544a7b5309497e841f0a94c6bcf0d33ea4958d7c528a0c80f... 
�[36mINFO�[0m[0001] GET KEYCHAIN                                 
�[36mINFO�[0m[0001] No cached layer found for cmd RUN python -m pip install -r cifar10_tensorflow/requirements.txt 
�[36mINFO�[0m[0001] Unpacking rootfs as cmd COPY cifar10_tensorflow/requirements.txt cifar10_tensorflow/requirements.txt requires it. 
�[36mINFO�[0m[0433] ENV LANG=C.UTF-8                             
�[36mINFO�[0m[0433] No files changed in this command, skipping snapshotting. 
�[36mINFO�[0m[0433] RUN apt-get update && apt-get install -y git 
�[36mINFO�[0m[0433] Found cached layer, extracting to filesystem 
�[36mINFO�[0m[0435] RUN python -m pip install --upgrade pip      
�[36mINFO�[0m[0435] Found cached layer, extracting to filesystem 
�[36mINFO�[0m[0435] COPY cifar10_tensorflow/requirements.txt cifar10_tensorflow/requirements.txt 
�[36mINFO�[0m[0435] Taking snapshot of files...                  
�[36mINFO�[0m[0435] RUN python -m pip install -r cifar10_tensorflow/requirements.txt 
�[36mINFO�[0m[0435] Taking snapshot of full filesystem...        
�[36mINFO�[0m[0698] cmd: /bin/sh                                 
�[36mINFO�[0m[0698] args: [-c python -m pip install -r cifar10_tensorflow/requirements.txt] 
�[36mINFO�[0m[0698] Running: [/bin/sh -c python -m pip install -r cifar10_tensorflow/requirements.txt] 
WARNING: Ignoring invalid distribution -wh-pip (/opt/conda/lib/python3.7/site-packages)
WARNING: Ignoring invalid distribution -wh-pip (/opt/conda/lib/python3.7/site-packages)
Requirement already satisfied: absl-py in /opt/conda/lib/python3.7/site-packages (from -r cifar10_tensorflow/requirements.txt (line 1)) (0.8.1)
Requirement already satisfied: tensorflow in /opt/conda/lib/python3.7/site-packages (from -r cifar10_tensorflow/requirements.txt (line 2)) (2.1.4)
Requirement already satisfied: tensorflow-datasets in /opt/conda/lib/python3.7/site-packages (from -r cifar10_tensorflow/requirements.txt (line 3)) (2.0.0)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py->-r cifar10_tensorflow/requirements.txt (line 1)) (1.16.0)
Collecting tensorboard<2.2.0,>=2.1.0
  Downloading tensorboard-2.1.1-py3-none-any.whl (3.8 MB)
Requirement already satisfied: h5py<=2.10.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2.10.0)
Requirement already satisfied: keras-applications>=1.0.8 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.0.8)
Requirement already satisfied: gast==0.2.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.2.2)
Requirement already satisfied: opt-einsum>=2.3.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.3.0)
Requirement already satisfied: tensorflow-estimator<2.2.0,>=2.1.0rc0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2.1.0)
Requirement already satisfied: google-pasta>=0.1.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.2.0)
Requirement already satisfied: keras-preprocessing==1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.1.0)
Requirement already satisfied: wrapt>=1.11.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.12.1)
Collecting numpy<1.19.0,>=1.16.0
  Downloading numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB)
Requirement already satisfied: grpcio>=1.8.6 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.38.0)
Requirement already satisfied: astor>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.8.1)
Requirement already satisfied: protobuf>=3.8.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.16.0)
Requirement already satisfied: termcolor>=1.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.1.0)
Requirement already satisfied: wheel>=0.26 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.36.2)
Requirement already satisfied: werkzeug>=0.11.15 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2.0.1)
Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.3.4)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.4.4)
Requirement already satisfied: requests<3,>=2.21.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2.25.1)
Requirement already satisfied: google-auth<2,>=1.6.3 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.30.2)
Requirement already satisfied: setuptools>=41.0.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (49.6.0.post20210108)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (4.2.2)
Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (4.7.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.2.7)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.3.0)
Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (4.5.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (0.4.8)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (4.0.0)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (1.26.5)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (2021.5.30)
Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.1.1)
Requirement already satisfied: tensorflow-metadata in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (0.21.2)
Requirement already satisfied: tqdm in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (4.61.1)
Requirement already satisfied: dill in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (0.3.0)
Requirement already satisfied: attrs>=18.1.0 in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (21.2.0)
Requirement already satisfied: promise in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (2.3)
Requirement already satisfied: future in /opt/conda/lib/python3.7/site-packages (from tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (0.18.2)
Requirement already satisfied: typing-extensions>=3.6.4 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.10.0.0)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard<2.2.0,>=2.1.0->tensorflow->-r cifar10_tensorflow/requirements.txt (line 2)) (3.4.1)
Requirement already satisfied: googleapis-common-protos in /opt/conda/lib/python3.7/site-packages (from tensorflow-metadata->tensorflow-datasets->-r cifar10_tensorflow/requirements.txt (line 3)) (1.53.0)
WARNING: Ignoring invalid distribution -wh-pip (/opt/conda/lib/python3.7/site-packages)
Installing collected packages: numpy, tensorboard
  Attempting uninstall: numpy
    WARNING: Ignoring invalid distribution -wh-pip (/opt/conda/lib/python3.7/site-packages)
    Found existing installation: numpy 1.19.5
    Uninstalling numpy-1.19.5:
      Successfully uninstalled numpy-1.19.5
  Attempting uninstall: tensorboard
    WARNING: Ignoring invalid distribution -wh-pip (/opt/conda/lib/python3.7/site-packages)
    Found existing installation: tensorboard 2.5.0
    Uninstalling tensorboard-2.5.0:
      Successfully uninstalled tensorboard-2.5.0
WARNING: Ignoring invalid distribution -wh-pip (/opt/conda/lib/python3.7/site-packages)
WARNING: Ignoring invalid distribution -wh-pip (/opt/conda/lib/python3.7/site-packages)
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
tfx-bsl 0.21.4 requires google-api-python-client<2,>=1.7.11, but you have google-api-python-client 2.9.0 which is incompatible.
tfx-bsl 0.21.4 requires pyarrow<0.16.0,>=0.15.0, but you have pyarrow 4.0.1 which is incompatible.
tensorflow-model-analysis 0.21.6 requires pyarrow<1,>=0.15, but you have pyarrow 4.0.1 which is incompatible.
tensorflow-model-analysis 0.21.6 requires scipy==1.4.1; python_version >= "3", but you have scipy 1.6.3 which is incompatible.
tensorflow-io 0.11.0 requires tensorflow==2.1.0, but you have tensorflow 2.1.4 which is incompatible.
tensorflow-data-validation 0.21.5 requires joblib<0.15,>=0.12, but you have joblib 1.0.1 which is incompatible.
tensorflow-data-validation 0.21.5 requires pandas<1,>=0.24, but you have pandas 1.2.4 which is incompatible.
tensorflow-data-validation 0.21.5 requires scikit-learn<0.22,>=0.18, but you have scikit-learn 0.24.2 which is incompatible.
tensorflow-cloud 0.1.13 requires tensorboard>=2.3.0, but you have tensorboard 2.1.1 which is incompatible.
apache-beam 2.17.0 requires httplib2<=0.12.0,>=0.8, but you have httplib2 0.19.1 which is incompatible.
apache-beam 2.17.0 requires pyarrow<0.16.0,>=0.15.1; python_version >= "3.0" or platform_system != "Windows", but you have pyarrow 4.0.1 which is incompatible.
Successfully installed numpy-1.18.5 tensorboard-2.1.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: Ignoring invalid distribution -wh-pip (/opt/conda/lib/python3.7/site-packages)
WARNING: Ignoring invalid distribution -wh-pip (/opt/conda/lib/python3.7/site-packages)
�[36mINFO�[0m[0708] Taking snapshot of full filesystem...        
�[36mINFO�[0m[0950] COPY cifar10_tensorflow/ cifar10_tensorflow  
�[36mINFO�[0m[0950] Taking snapshot of files...                  
�[36mINFO�[0m[0950] WORKDIR cifar10_tensorflow                   
�[36mINFO�[0m[0950] cmd: workdir                                 
�[36mINFO�[0m[0950] Changed working directory to /cifar10_tensorflow 
�[36mINFO�[0m[0950] No files changed in this command, skipping snapshotting. 
�[36mINFO�[0m[0950] COPY entrypoint.sh ./entrypoint.sh           
�[36mINFO�[0m[0950] Taking snapshot of files...                  
�[36mINFO�[0m[0950] RUN chmod +x ./entrypoint.sh                 
�[36mINFO�[0m[0950] cmd: /bin/sh                                 
�[36mINFO�[0m[0950] args: [-c chmod +x ./entrypoint.sh]          
�[36mINFO�[0m[0950] Running: [/bin/sh -c chmod +x ./entrypoint.sh] 
�[36mINFO�[0m[0950] Taking snapshot of full filesystem...        
�[36mINFO�[0m[0950] Pushing layer gcr.io/researchprojects-msc/cifar10_tensorflow/cache:dde4d6174b68d54544a7b5309497e841f0a94c6bcf0d33ea4958d7c528a0c80f to cache now 
�[36mINFO�[0m[0950] GET KEYCHAIN                                 
�[36mINFO�[0m[0950] Pushing image to gcr.io/researchprojects-msc/cifar10_tensorflow/cache:dde4d6174b68d54544a7b5309497e841f0a94c6bcf0d33ea4958d7c528a0c80f 
�[36mINFO�[0m[0952] Pushed image to 1 destinations               
�[36mINFO�[0m[1180] Pushing layer gcr.io/researchprojects-msc/cifar10_tensorflow/cache:87d5ac879ebb7364fda6eb69e993ae5cba71fe1ec9bdce47bea59cdc6e3e9021 to cache now 
�[36mINFO�[0m[1180] RUN chmod +x ./wrapped_entrypoint.sh         
�[36mINFO�[0m[1180] cmd: /bin/sh                                 
�[36mINFO�[0m[1180] args: [-c chmod +x ./wrapped_entrypoint.sh]  
�[36mINFO�[0m[1180] Running: [/bin/sh -c chmod +x ./wrapped_entrypoint.sh] 
�[36mINFO�[0m[1180] GET KEYCHAIN                                 
�[36mINFO�[0m[1180] Taking snapshot of full filesystem...        
�[36mINFO�[0m[1180] Pushing image to gcr.io/researchprojects-msc/cifar10_tensorflow/cache:87d5ac879ebb7364fda6eb69e993ae5cba71fe1ec9bdce47bea59cdc6e3e9021 
�[36mINFO�[0m[1182] Pushed image to 1 destinations               
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor:latest" failed: step exited with non-zero status: 2
Torch XLA 
starting build "46618c1c-1f1d-409e-8b31-74240bda94e2"

FETCHSOURCE
Fetching storage object: gs://revirainbow_bucket2/cifar10_torch_xla-latest.tar.gz#1624824967473593
Copying gs://revirainbow_bucket2/cifar10_torch_xla-latest.tar.gz#1624824967473593...
/ [0 files][    0.0 B/  6.7 KiB]                                                
/ [1 files][  6.7 KiB/  6.7 KiB]                                                
Operation completed over 1 objects/6.7 KiB.                                      
tar: Removing leading `/' from member names
BUILD
Pulling image: gcr.io/kaniko-project/executor:latest
latest: Pulling from kaniko-project/executor
Digest: sha256:6ecc43ae139ad8cfa11604b592aaedddcabff8cef469eda303f1fb5afe5e3034
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
gcr.io/kaniko-project/executor:latest
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-xla.1-8 
�[36mINFO�[0m[0000] Retrieving image gcr.io/deeplearning-platform-release/pytorch-xla.1-8 from registry gcr.io 
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-xla.1-8 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Built cross stage deps: map[]                
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-xla.1-8 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-xla.1-8 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Executing 0 build triggers                   
�[36mINFO�[0m[0000] Checking for cached layer gcr.io/researchprojects-msc/cifar10_torch_xla/cache:d44b9071ffdd974e978b3e6db70a4690d0eb85a012e775e287ca60878f2f9f14... 
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] No cached layer found for cmd RUN apt-get update && apt-get install -y git 
�[36mINFO�[0m[0000] Unpacking rootfs as cmd RUN apt-get update && apt-get install -y git requires it. 
�[36mINFO�[0m[0333] ENV LANG=C.UTF-8                             
�[36mINFO�[0m[0333] No files changed in this command, skipping snapshotting. 
�[36mINFO�[0m[0333] RUN apt-get update && apt-get install -y git 
�[36mINFO�[0m[0333] Taking snapshot of full filesystem...        
�[36mINFO�[0m[0634] cmd: /bin/sh                                 
�[36mINFO�[0m[0634] args: [-c apt-get update && apt-get install -y git] 
�[36mINFO�[0m[0634] Running: [/bin/sh -c apt-get update && apt-get install -y git] 
Get:1 http://packages.cloud.google.com/apt gcsfuse-bionic InRelease [5385 B]
Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Get:3 http://packages.cloud.google.com/apt cloud-sdk-bionic InRelease [6780 B]
Hit:4 http://archive.ubuntu.com/ubuntu bionic InRelease
Get:5 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:6 http://packages.cloud.google.com/apt cloud-sdk-bionic/main amd64 Packages [191 kB]
Get:7 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [473 kB]
Get:8 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [2220 kB]
Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:10 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1418 kB]
Get:11 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [505 kB]
Get:12 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [2188 kB]
Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [2653 kB]
Fetched 9911 kB in 2s (5160 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
git is already the newest version (1:2.17.1-1ubuntu0.8).
0 upgraded, 0 newly installed, 0 to remove and 6 not upgraded.
�[36mINFO�[0m[0639] Taking snapshot of full filesystem...        
�[36mINFO�[0m[0901] RUN python -m pip install --upgrade pip      
�[36mINFO�[0m[0901] cmd: /bin/sh                                 
�[36mINFO�[0m[0901] args: [-c python -m pip install --upgrade pip] 
�[36mINFO�[0m[0901] Running: [/bin/sh -c python -m pip install --upgrade pip] 
�[36mINFO�[0m[0901] Pushing layer gcr.io/researchprojects-msc/cifar10_torch_xla/cache:d44b9071ffdd974e978b3e6db70a4690d0eb85a012e775e287ca60878f2f9f14 to cache now 
�[36mINFO�[0m[0901] GET KEYCHAIN                                 
�[36mINFO�[0m[0901] Pushing image to gcr.io/researchprojects-msc/cifar10_torch_xla/cache:d44b9071ffdd974e978b3e6db70a4690d0eb85a012e775e287ca60878f2f9f14 
Requirement already satisfied: pip in /opt/conda/lib/python3.7/site-packages (21.1.2)
Collecting pip
  Downloading pip-21.1.3-py3-none-any.whl (1.5 MB)
�[36mINFO�[0m[0903] Pushed image to 1 destinations               
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.1.2
    Uninstalling pip-21.1.2:
      Successfully uninstalled pip-21.1.2
WARNING: Running pip as root will break packages and permissions. You should install packages reliably by using venv: https://pip.pypa.io/warnings/venv
Successfully installed pip-21.1.3
�[36mINFO�[0m[0906] Taking snapshot of full filesystem...        
�[36mINFO�[0m[1170] Pushing layer gcr.io/researchprojects-msc/cifar10_torch_xla/cache:39c4fcff89964385cbe78b4c1701f08eff30706e32ce3559eedebce39738669a to cache now 
�[36mINFO�[0m[1170] GET KEYCHAIN                                 
�[36mINFO�[0m[1170] COPY cifar10_torch_xla/requirements.txt cifar10_torch_xla/requirements.txt 
�[36mINFO�[0m[1170] Taking snapshot of files...                  
�[36mINFO�[0m[1170] RUN python -m pip install -r cifar10_torch_xla/requirements.txt 
�[36mINFO�[0m[1170] cmd: /bin/sh                                 
�[36mINFO�[0m[1170] args: [-c python -m pip install -r cifar10_torch_xla/requirements.txt] 
�[36mINFO�[0m[1170] Running: [/bin/sh -c python -m pip install -r cifar10_torch_xla/requirements.txt] 
�[36mINFO�[0m[1170] Pushing image to gcr.io/researchprojects-msc/cifar10_torch_xla/cache:39c4fcff89964385cbe78b4c1701f08eff30706e32ce3559eedebce39738669a 
Collecting absl-py
  Downloading absl_py-0.13.0-py3-none-any.whl (132 kB)
Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from -r cifar10_torch_xla/requirements.txt (line 16)) (1.19.5)
Collecting tensorflow
  Downloading tensorflow-2.5.0-cp37-cp37m-manylinux2010_x86_64.whl (454.3 MB)
�[36mINFO�[0m[1172] Pushed image to 1 destinations               
Requirement already satisfied: torch in /opt/conda/lib/python3.7/site-packages (from -r cifar10_torch_xla/requirements.txt (line 18)) (1.8.0)
Requirement already satisfied: torchvision in /opt/conda/lib/python3.7/site-packages (from -r cifar10_torch_xla/requirements.txt (line 19)) (0.9.0+cu111)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py->-r cifar10_torch_xla/requirements.txt (line 15)) (1.16.0)
Collecting h5py~=3.1.0
  Downloading h5py-3.1.0-cp37-cp37m-manylinux1_x86_64.whl (4.0 MB)
Collecting six
  Downloading six-1.15.0-py2.py3-none-any.whl (10 kB)
Requirement already satisfied: wheel~=0.35 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (0.36.2)
Collecting flatbuffers~=1.12.0
  Downloading flatbuffers-1.12-py2.py3-none-any.whl (15 kB)
Collecting opt-einsum~=3.3.0
  Downloading opt_einsum-3.3.0-py3-none-any.whl (65 kB)
Collecting gast==0.4.0
  Downloading gast-0.4.0-py3-none-any.whl (9.8 kB)
Collecting typing-extensions~=3.7.4
  Downloading typing_extensions-3.7.4.3-py3-none-any.whl (22 kB)
Collecting tensorboard~=2.5
  Downloading tensorboard-2.5.0-py3-none-any.whl (6.0 MB)
Collecting termcolor~=1.1.0
  Downloading termcolor-1.1.0.tar.gz (3.9 kB)
Collecting tensorflow-estimator<2.6.0,>=2.5.0rc0
  Downloading tensorflow_estimator-2.5.0-py2.py3-none-any.whl (462 kB)
Collecting astunparse~=1.6.3
  Downloading astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
Collecting keras-preprocessing~=1.1.2
  Downloading Keras_Preprocessing-1.1.2-py2.py3-none-any.whl (42 kB)
Requirement already satisfied: protobuf>=3.9.2 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (3.16.0)
Requirement already satisfied: wrapt~=1.12.1 in /opt/conda/lib/python3.7/site-packages (from tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (1.12.1)
Collecting keras-nightly~=2.5.0.dev
  Downloading keras_nightly-2.5.0.dev2021032900-py2.py3-none-any.whl (1.2 MB)
Collecting google-pasta~=0.2
  Downloading google_pasta-0.2.0-py3-none-any.whl (57 kB)
Collecting grpcio~=1.34.0
  Downloading grpcio-1.34.1-cp37-cp37m-manylinux2014_x86_64.whl (4.0 MB)
Collecting cached-property
  Downloading cached_property-1.5.2-py2.py3-none-any.whl (7.6 kB)
Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /opt/conda/lib/python3.7/site-packages (from tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (0.4.4)
Collecting tensorboard-plugin-wit>=1.6.0
  Downloading tensorboard_plugin_wit-1.8.0-py3-none-any.whl (781 kB)
Requirement already satisfied: markdown>=2.6.8 in /opt/conda/lib/python3.7/site-packages (from tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (3.3.4)
Collecting werkzeug>=0.11.15
  Downloading Werkzeug-2.0.1-py3-none-any.whl (288 kB)
Requirement already satisfied: setuptools>=41.0.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (49.6.0.post20210108)
Requirement already satisfied: requests<3,>=2.21.0 in /opt/conda/lib/python3.7/site-packages (from tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (2.25.1)
Requirement already satisfied: google-auth<2,>=1.6.3 in /opt/conda/lib/python3.7/site-packages (from tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (1.30.2)
Collecting tensorboard-data-server<0.7.0,>=0.6.0
  Downloading tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
Requirement already satisfied: cachetools<5.0,>=2.0.0 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (4.2.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (0.2.7)
Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (4.7.2)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /opt/conda/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (1.3.0)
Requirement already satisfied: importlib-metadata in /opt/conda/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (4.5.0)
Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (0.4.8)
Requirement already satisfied: chardet<5,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (4.0.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (1.26.5)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (2.10)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (2021.5.30)
Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (3.1.1)
Requirement already satisfied: pillow>=4.1.1 in /opt/conda/lib/python3.7/site-packages (from torchvision->-r cifar10_torch_xla/requirements.txt (line 19)) (8.2.0)
Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata->markdown>=2.6.8->tensorboard~=2.5->tensorflow->-r cifar10_torch_xla/requirements.txt (line 17)) (3.4.1)
Building wheels for collected packages: termcolor
  Building wheel for termcolor (setup.py): started
  Building wheel for termcolor (setup.py): finished with status 'done'
  Created wheel for termcolor: filename=termcolor-1.1.0-py3-none-any.whl size=4829 sha256=21af378bdd76722c309847123ef4aa570b137ae5527136336d134b545f312b1c
  Stored in directory: /root/.cache/pip/wheels/3f/e3/ec/8a8336ff196023622fbcb36de0c5a5c218cbb24111d1d4c7f2
Successfully built termcolor
Installing collected packages: typing-extensions, six, werkzeug, tensorboard-plugin-wit, tensorboard-data-server, grpcio, cached-property, absl-py, termcolor, tensorflow-estimator, tensorboard, opt-einsum, keras-preprocessing, keras-nightly, h5py, google-pasta, gast, flatbuffers, astunparse, tensorflow
  Attempting uninstall: typing-extensions
    Found existing installation: typing-extensions 3.10.0.0
    Uninstalling typing-extensions-3.10.0.0:
      Successfully uninstalled typing-extensions-3.10.0.0
  Attempting uninstall: six
    Found existing installation: six 1.16.0
    Uninstalling six-1.16.0:
      Successfully uninstalled six-1.16.0
  Attempting uninstall: grpcio
    Found existing installation: grpcio 1.38.0
    Uninstalling grpcio-1.38.0:
      Successfully uninstalled grpcio-1.38.0
TIMEOUT
ERROR: context deadline exceeded

Thank you in advance for your help
@joaogui1
Copy link
Author

Hi, I found out about the timeout flag and increased it, but now I got a different error 137 (maybe OOM), can you help me out please?

Cloud Build log for torch example 
starting build "5ef494ca-c3b7-4b9e-86cb-42c3247ecec3"

FETCHSOURCE
Fetching storage object: gs://revirainbow_bucket2/cifar10_torch-latest.tar.gz#1624976910383291
Copying gs://revirainbow_bucket2/cifar10_torch-latest.tar.gz#1624976910383291...
/ [0 files][    0.0 B/  7.1 KiB]                                                
/ [1 files][  7.1 KiB/  7.1 KiB]                                                
Operation completed over 1 objects/7.1 KiB.                                      
tar: Removing leading `/' from member names
BUILD
Pulling image: gcr.io/kaniko-project/executor:latest
latest: Pulling from kaniko-project/executor
Digest: sha256:6ecc43ae139ad8cfa11604b592aaedddcabff8cef469eda303f1fb5afe5e3034
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
gcr.io/kaniko-project/executor:latest
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-gpu.1-6 
�[36mINFO�[0m[0000] Retrieving image gcr.io/deeplearning-platform-release/pytorch-gpu.1-6 from registry gcr.io 
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-gpu.1-6 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Built cross stage deps: map[]                
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-gpu.1-6 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Retrieving image manifest gcr.io/deeplearning-platform-release/pytorch-gpu.1-6 
�[36mINFO�[0m[0000] Returning cached image manifest              
�[36mINFO�[0m[0000] Executing 0 build triggers                   
�[36mINFO�[0m[0000] Checking for cached layer gcr.io/researchprojects-msc/cifar10_torch/cache:f6b49a2c721c492debdfe49e26c8073947a7c2f39c82709a8463ca794242a13b... 
�[36mINFO�[0m[0000] GET KEYCHAIN                                 
�[36mINFO�[0m[0001] Using caching version of cmd: RUN apt-get update && apt-get install -y git 
�[36mINFO�[0m[0001] Checking for cached layer gcr.io/researchprojects-msc/cifar10_torch/cache:de36dfecd19ea8b00bccc191d92ae33f47234e53e30490acc8f858f22295fa38... 
�[36mINFO�[0m[0001] GET KEYCHAIN                                 
�[36mINFO�[0m[0001] No cached layer found for cmd RUN python -m pip install --upgrade pip 
�[36mINFO�[0m[0001] Unpacking rootfs as cmd RUN python -m pip install --upgrade pip requires it. 
�[36mINFO�[0m[0499] ENV LANG=C.UTF-8                             
�[36mINFO�[0m[0499] No files changed in this command, skipping snapshotting. 
�[36mINFO�[0m[0499] RUN apt-get update && apt-get install -y git 
�[36mINFO�[0m[0499] Found cached layer, extracting to filesystem 
�[36mINFO�[0m[0501] RUN python -m pip install --upgrade pip      
�[36mINFO�[0m[0501] Taking snapshot of full filesystem...        
�[36mINFO�[0m[0849] cmd: /bin/sh                                 
�[36mINFO�[0m[0849] args: [-c python -m pip install --upgrade pip] 
�[36mINFO�[0m[0849] Running: [/bin/sh -c python -m pip install --upgrade pip] 
Collecting pip
  Downloading pip-21.1.3-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.4
    Uninstalling pip-20.2.4:
      Successfully uninstalled pip-20.2.4
Successfully installed pip-21.1.3
�[36mINFO�[0m[0854] Taking snapshot of full filesystem...        
�[36mINFO�[0m[1172] COPY cifar10_torch/requirements.txt cifar10_torch/requirements.txt 
�[36mINFO�[0m[1172] Taking snapshot of files...                  
�[36mINFO�[0m[1172] RUN python -m pip install -r cifar10_torch/requirements.txt 
�[36mINFO�[0m[1172] cmd: /bin/sh                                 
�[36mINFO�[0m[1172] args: [-c python -m pip install -r cifar10_torch/requirements.txt] 
�[36mINFO�[0m[1172] Running: [/bin/sh -c python -m pip install -r cifar10_torch/requirements.txt] 
�[36mINFO�[0m[1172] Pushing layer gcr.io/researchprojects-msc/cifar10_torch/cache:de36dfecd19ea8b00bccc191d92ae33f47234e53e30490acc8f858f22295fa38 to cache now 
�[36mINFO�[0m[1172] GET KEYCHAIN                                 
�[36mINFO�[0m[1172] Pushing image to gcr.io/researchprojects-msc/cifar10_torch/cache:de36dfecd19ea8b00bccc191d92ae33f47234e53e30490acc8f858f22295fa38 
Collecting absl-py
  Downloading absl_py-0.13.0-py3-none-any.whl (132 kB)
Requirement already satisfied: numpy in /opt/conda/lib/python3.7/site-packages (from -r cifar10_torch/requirements.txt (line 16)) (1.19.3)
Collecting torch==1.6.0
  Downloading torch-1.6.0-cp37-cp37m-manylinux1_x86_64.whl (748.8 MB)
�[36mINFO�[0m[1175] Pushed image to 1 destinations               
Collecting torchvision==0.7.0
  Downloading torchvision-0.7.0-cp37-cp37m-manylinux1_x86_64.whl (5.9 MB)
Requirement already satisfied: future in /opt/conda/lib/python3.7/site-packages (from torch==1.6.0->-r cifar10_torch/requirements.txt (line 17)) (0.18.2)
Requirement already satisfied: pillow>=4.1.1 in /opt/conda/lib/python3.7/site-packages (from torchvision==0.7.0->-r cifar10_torch/requirements.txt (line 18)) (8.0.1)
Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from absl-py->-r cifar10_torch/requirements.txt (line 15)) (1.15.0)
Installing collected packages: torch, torchvision, absl-py
  Attempting uninstall: torch
    Found existing installation: torch 1.6.0a0+9907a3e
    Uninstalling torch-1.6.0a0+9907a3e:
      Successfully uninstalled torch-1.6.0a0+9907a3e
  Attempting uninstall: torchvision
    Found existing installation: torchvision 0.7.0a0
    Uninstalling torchvision-0.7.0a0:
      Successfully uninstalled torchvision-0.7.0a0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Successfully installed absl-py-0.13.0 torch-1.6.0 torchvision-0.7.0
�[36mINFO�[0m[1210] Taking snapshot of full filesystem...        
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor:latest" failed: step exited with non-zero status: 137

@andrewluchen
Copy link
Collaborator

Hi Joao,

I see that these errors all use Google Cloud Build and Kaniko. There are two separate fixes you can try.

  1. Build images locally with Docker. For that, you need to install Docker. https://docs.docker.com/get-docker/

This also has the added benefit of caching builds so iterative builds become faster.

  1. Turn off kaniko by adding --use_kaniko=False to your command like xmanager launch script.py -- --use_kaniko=False. This should use Cloud Build without Kaniko and avoid the error that you are seeing.

@joaogui1
Copy link
Author

joaogui1 commented Jun 29, 2021 •
edited

How do I build them locally? @andrewluchen

@andrewluchen
Copy link
Collaborator

Localy builds are automatically done if Docker is installed as seen in this line:
https://github.com/deepmind/xmanager/blob/main/xmanager/cloud/build_image.py#L84

Do you see anything in your output logs like:

Failed to initialize local docker.

@joaogui1
Copy link
Author

joaogui1 commented Jul 1, 2021

Yeah was having that problem, looks like my docker wasn't initializing on boot. It worked now, thanks!

@joaogui1 joaogui1 closed this as completed Jul 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@andrewluchen @joaogui1