Failing to install in a Tensor-flow Jupyter Notebook Docker image

[perllaghu]

Prerequisites

Please answer the following questions for yourself before submitting an issue.

• I am using the latest TensorFlow Model Garden release and TensorFlow 2.
• I am reporting the issue to the correct repository. (Model Garden official or research directory)
• I checked to make sure that this issue has not been filed already.

1. The entire URL of the file you are using

pip install tf-models-official - which should be 2.5.1

Also tried pip install "tf-models-official==2.4.0" given that tensor-flow is 2.4.1

The details, and the error-report, are from the 1st, general, incantation

2. Describe the bug

The pip install spends a while trying to find dependency versions (as described in issue #10149), and either stalls, or crashes out

I've had a build sat stalled for 12 hours, and I've had a crash that ends:

Failed to build pycocotools
Installing collected packages: numpy, text-unidecode, tensorboard-data-server, httplib2, grpcio, googleapis-common-protos, absl-py, uritemplate, typeguard, tensorflow-metadata, tensorflow-estimator, tensorboard, python-slugify, promise, portalocker, keras-nightly, h5py, google-auth-httplib2, google-api-core, gast, future, dm-tree, tf-slim, tensorflow-model-optimization, tensorflow-hub, tensorflow-datasets, tensorflow-addons, tensorflow, seqeval, sentencepiece, sacrebleu, pycocotools, py-cpuinfo, opencv-python-headless, oauth2client, kaggle, google-api-python-client, gin-config, tf-models-official
  Attempting uninstall: numpy
    Found existing installation: numpy 1.21.1
    Uninstalling numpy-1.21.1:
      Successfully uninstalled numpy-1.21.1
  Attempting uninstall: grpcio
    Found existing installation: grpcio 1.37.1
    Uninstalling grpcio-1.37.1:
      Successfully uninstalled grpcio-1.37.1
  Attempting uninstall: absl-py
    Found existing installation: absl-py 0.13.0
    Uninstalling absl-py-0.13.0:
      Successfully uninstalled absl-py-0.13.0
  Attempting uninstall: tensorflow-estimator
    Found existing installation: tensorflow-estimator 2.4.0
    Uninstalling tensorflow-estimator-2.4.0:
      Successfully uninstalled tensorflow-estimator-2.4.0
  Attempting uninstall: tensorboard
    Found existing installation: tensorboard 2.4.1
    Uninstalling tensorboard-2.4.1:
      Successfully uninstalled tensorboard-2.4.1
  Attempting uninstall: h5py
    Found existing installation: h5py 2.10.0
    Uninstalling h5py-2.10.0:
      Successfully uninstalled h5py-2.10.0
  Attempting uninstall: gast
    Found existing installation: gast 0.3.3
    Uninstalling gast-0.3.3:
      Successfully uninstalled gast-0.3.3
  Attempting uninstall: tensorflow
    Found existing installation: tensorflow 2.4.1
    Uninstalling tensorflow-2.4.1:
      Successfully uninstalled tensorflow-2.4.1
    Running setup.py install for pycocotools: started
    Running setup.py install for pycocotools: finished with status 'error'
    ERROR: Command errored out with exit status 1:
     command: /opt/conda/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-5em9vkru/pycocotools_cda6092a36d3424585179786c853c385/setup.py'"'"'; __file__='"'"'/tmp/pip-install-5em9vkru/pycocotools_cda6092a36d3424585179786c853c385/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-40b93spd/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/include/python3.9/pycocotools
         cwd: /tmp/pip-install-5em9vkru/pycocotools_cda6092a36d3424585179786c853c385/
    Complete output (63 lines):
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.9
    creating build/lib.linux-x86_64-3.9/pycocotools
    copying pycocotools/cocoeval.py -> build/lib.linux-x86_64-3.9/pycocotools
    copying pycocotools/mask.py -> build/lib.linux-x86_64-3.9/pycocotools
    copying pycocotools/coco.py -> build/lib.linux-x86_64-3.9/pycocotools
    copying pycocotools/__init__.py -> build/lib.linux-x86_64-3.9/pycocotools
    running build_ext
    skipping 'pycocotools/_mask.c' Cython extension (up-to-date)
    building 'pycocotools._mask' extension
    creating build/temp.linux-x86_64-3.9
    creating build/temp.linux-x86_64-3.9/common
    creating build/temp.linux-x86_64-3.9/pycocotools
    gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/conda/include -fPIC -O2 -isystem /opt/conda/include -fPIC -I/opt/conda/lib/python3.9/site-packages/numpy/core/include -I./common -I/opt/conda/include/python3.9 -c ./common/maskApi.c -o build/temp.linux-x86_64-3.9/./common/maskApi.o -Wno-cpp -Wno-unused-function -std=c99
    ./common/maskApi.c: In function 'rleDecode':
    ./common/maskApi.c:46:7: warning: this 'for' clause does not guard... [-Wmisleading-indentation]
       46 |       for( k=0; k<R[i].cnts[j]; k++ ) *(M++)=v; v=!v; }}
          |       ^~~
    ./common/maskApi.c:46:49: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'for'
       46 |       for( k=0; k<R[i].cnts[j]; k++ ) *(M++)=v; v=!v; }}
          |                                                 ^
    ./common/maskApi.c: In function 'rleFrPoly':
    ./common/maskApi.c:166:3: warning: this 'for' clause does not guard... [-Wmisleading-indentation]
      166 |   for(j=0; j<k; j++) x[j]=(int)(scale*xy[j*2+0]+.5); x[k]=x[0];
          |   ^~~
    ./common/maskApi.c:166:54: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'for'
      166 |   for(j=0; j<k; j++) x[j]=(int)(scale*xy[j*2+0]+.5); x[k]=x[0];
          |                                                      ^
    ./common/maskApi.c:167:3: warning: this 'for' clause does not guard... [-Wmisleading-indentation]
      167 |   for(j=0; j<k; j++) y[j]=(int)(scale*xy[j*2+1]+.5); y[k]=y[0];
          |   ^~~
    ./common/maskApi.c:167:54: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'for'
      167 |   for(j=0; j<k; j++) y[j]=(int)(scale*xy[j*2+1]+.5); y[k]=y[0];
          |                                                      ^
    ./common/maskApi.c: In function 'rleToString':
    ./common/maskApi.c:212:7: warning: this 'if' clause does not guard... [-Wmisleading-indentation]
      212 |       if(more) c |= 0x20; c+=48; s[p++]=c;
          |       ^~
    ./common/maskApi.c:212:27: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
      212 |       if(more) c |= 0x20; c+=48; s[p++]=c;
          |                           ^
    ./common/maskApi.c: In function 'rleFrString':
    ./common/maskApi.c:220:3: warning: this 'while' clause does not guard... [-Wmisleading-indentation]
      220 |   while( s[m] ) m++; cnts=malloc(sizeof(uint)*m); m=0;
          |   ^~~~~
    ./common/maskApi.c:220:22: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'while'
      220 |   while( s[m] ) m++; cnts=malloc(sizeof(uint)*m); m=0;
          |                      ^~~~
    ./common/maskApi.c:228:5: warning: this 'if' clause does not guard... [-Wmisleading-indentation]
      228 |     if(m>2) x+=(long) cnts[m-2]; cnts[m++]=(uint) x;
          |     ^~
    ./common/maskApi.c:228:34: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
      228 |     if(m>2) x+=(long) cnts[m-2]; cnts[m++]=(uint) x;
          |                                  ^~~~
    gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/conda/include -fPIC -O2 -isystem /opt/conda/include -fPIC -I/opt/conda/lib/python3.9/site-packages/numpy/core/include -I./common -I/opt/conda/include/python3.9 -c pycocotools/_mask.c -o build/temp.linux-x86_64-3.9/pycocotools/_mask.o -Wno-cpp -Wno-unused-function -std=c99
    gcc -pthread -B /opt/conda/compiler_compat -Wl,--sysroot=/ -shared -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib -Wl,-rpath,/opt/conda/lib -Wl,-rpath-link,/opt/conda/lib -L/opt/conda/lib build/temp.linux-x86_64-3.9/./common/maskApi.o build/temp.linux-x86_64-3.9/pycocotools/_mask.o -o build/lib.linux-x86_64-3.9/pycocotools/_mask.cpython-39-x86_64-linux-gnu.so
    /opt/conda/compiler_compat/ld: cannot find /lib64/libpthread.so.0
    /opt/conda/compiler_compat/ld: cannot find /usr/lib64/libpthread_nonshared.a
    collect2: error: ld returned 1 exit status
    error: command '/opt/conda/bin/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/conda/bin/python3.9 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-5em9vkru/pycocotools_cda6092a36d3424585179786c853c385/setup.py'"'"'; __file__='"'"'/tmp/pip-install-5em9vkru/pycocotools_cda6092a36d3424585179786c853c385/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-40b93spd/install-record.txt --single-version-externally-managed --compile --install-headers /opt/conda/include/python3.9/pycocotools Check the logs for full command output.

See the attached build.log for details of the 12-hour process that stalled
build.log

3. Steps to reproduce

The the following Dockerfile:

# Ubuntu 20:02, python 3.8.8, hub 1.3.0, lab 3.0.11, notebook 6.2.0 - released 21/3/21
FROM jupyter/tensorflow-notebook:3395de4db93a
USER root
RUN pip install tf-models-official

then build the image:

docker build -t test .

(this step will take many minutes. Have a cup of tea. Reach a chapter or two, Take the dog for a walk... no, seriously... you're looking at several hours!)

4. Expected behaviour

I would expect the library to install, probably with some dependencies.

5. Additional context

Include any logs that would be helpful to diagnose the problem.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 20:02, python 3.8.8
• TensorFlow installed from (source or binary): binary - via pip
• TensorFlow version (use command below):

    docker run -it --rm jupyter/tensorflow-notebook:3395de4db93a pip list | grep tensor
    tensorboard                   2.4.1
    tensorboard-plugin-wit        1.8.0
    tensorflow                    2.4.1
    tensorflow-estimator          2.4.0

• Python version: 3.8.8

[perllaghu]

Further information:

When I built my full docker image back in June (june 14th), I had the following libraries:

tensorflow=2.5.0
tf-models-official=2.5.0

... and dependencies:

certifi=2021.5.30
chardet=4.0.0
idna=2.10
rsa=4.7.2
scipy=1.6.1

(pip list provides others.... )

[saikumarchalla]

@perllaghu Please check the official documentation here and hope it helps you. Thanks!

[perllaghu]

@perllaghu Please check the official documentation here and hope it helps you. Thanks!

Thank you for the automated reply.

As you will see from the given Dockerfile, the installation is as per the README, lines 103-110.

If there is some other piece of documentation I've missed, please enlighten me.

[saberkun]

Sorry, I have no idea about pycocotools crash.

For the slow installation,
we released: https://github.com/tensorflow/models/releases/tag/v2.5.1
The cause is google-cloud-bigquery. The new pip resolver is very lower to resolve it.
We removed it as 703688a
Now both 2.5.1 and nightly package should fix the problem.

[perllaghu]

I can confirm that defining tf-models-official==2.5.1 fixed the problem just lovely!