pip builds grpcio-tools from source making installation slow

[jleni]

grpci-reflection package installation freezes depending on other packages in the same command line

This can be easily reproduced by these two docker containers:

Dockerfile.fast - Container creation time ~1m 23s

#Download base ubuntu image
FROM ubuntu:16.04
RUN apt-get update && \
    apt-get -y install ca-certificates curl

# Prepare pip 
RUN apt-get -y install python-pip
RUN pip install -U pip
RUN pip install grpcio grpcio-tools
RUN pip install grpcio-reflection                   # Two lines is FAST

Dockerfile.slow - Container creation time ~6m 28s

#Download base ubuntu image
FROM ubuntu:16.04
RUN apt-get update && \
    apt-get -y install ca-certificates curl

# Prepare pip
RUN apt-get -y install python-pip
RUN pip install -U pip
RUN pip install grpcio grpcio-tools grpcio-reflection      # Single line is SLOW

Timing containers build time:

time docker build --rm --no-cache -f Dockerfile.fast -t repro_reflbug_fast:latest .
......
real	1m22.295s
user	0m0.060s
sys	0m0.040s

time docker build --rm --no-cache -f Dockerfile.slow -t repro_reflbug_slow:latest .
.....
real	6m28.290s
user	0m0.052s
sys	0m0.052s
.....

[nathanielmanistaatgoogle]

Wow. It's fascinating that that makes a difference. I wonder if we're structuring something wrong in our package or if it's a pip problem.

[jleni]

Yes, I was very surprised. Sorry I didn't have time to go deeper. I am really curious about what is the underlying reason for this.

[mehrdada]

Looks like the slow version tries to compile grpcio-tools from source for some reason. That's why it takes so long. I don't know why pip is doing that.

[mehrdada]

Simply running pip install grpcio-health-checking grpcio-tools can reproduce the problem too. For some reason, when pip installs grpcio-tools alone, it works fine, but when it is accompanied by grpcio-reflection or grpcio-health-checking, it tries building from source and invoking C compilers in the process (this happens irrespective of whether grpcio package is preinstalled or not).

FYI, I tried pip install grpcio-tools tensorflow and it works fine though.

[jleni]

@mehrdada kpayson64 Are you already working on this or a PR would be useful?

[kpayson64]

I'm not investigating this at the moment, but if you had a PR for a fix that would be great!

[mehrdada]

@jleni I played with it a bit and it is not clear what, or if, we are doing something wrong, so I gave up, since it does not seem like a blocking issue for users (is it?). At the moment, the root cause might be an issue with pip. A PR would be appreciated for sure if you know the root cause (or a workaround in our packaging that makes the symptom go away). Meanwhile, you may want to file a bug with pip as well.

[mehrdada]

I think I have narrowed down it to:

• grpcio_reflection package uses grpcio-tools to compile the reflection.proto file to pb2 when the package is built from source (same for grpcio_health_checking). [I am not sure if this is how we build the package in our CI system, because we do package the generated _pb2 in the .tar.gz already, but had we wanted to build it from source individually, it would have worked that way.]
• grpcio_reflection package has a setup.py that declares grpcio-tools package in its setup_requires.
• For some reason (my speculation is uploading the reflection package in .tar.gz format as opposed to binary wheel), pip tries to "build" grpcio_reflection from "source" and hands it to setuptools. Apparently setuptools is entirely out of pip's control, so the moment that gets handed over, it tries to hand it over to easy_install (my understanding might be wrong, but it would be another python build system altogether) instead to fetch and install grpcio-tools (but unlike pip it is dumb and installs it from source), unless it finds grpcio-tools package in PYTHONPATH already.

[mehrdada]

@jleni This should be fixed now that 1.8.0 is on PyPI. Please verify and let us know if it is not.

[xvzf]

@mehrdada

It seems like it is not fixed completely - when using tox and installing grpcio with a requirements.txt looking like this:

grpcio>=1.11.0
grpcio-tools>=1.11.0
pytest>=3.5.0

it starts compiling it again!

[mehrdada]

@xvzf That might be a separate problem with some tox interaction, I'm not sure. Please file a separate issue with repro instructions. I'm pretty confident that unless this is a new regression, the specific issue in this bug is not reproducible. Also please let us know if you manage to repro this exact issue again.