pecl install is slow and produces massive (55Mb) binary

[acoulton]

What version of gRPC and what language are you using?

• grpc 1.30.0
• php 7.2.24
• pecl 1.10.12

What operating system (Linux, Windows,...) and version?

Ubuntu 18.08

What runtime / compiler are you using (e.g. python version or version of gcc)

pecl

What did you do?

time sudo pecl install grpc

What did you expect to see?

The extension install in a reasonable timeframe and at a reasonable size

What did you see instead?

The pecl install takes 6m 21s (on a machine that is not usually slow):

real	6m21.135s
user	5m27.342s
sys	0m30.712s

I have seen similar execution times running the pecl install as part of Docker build on Google Cloud Build. This is far and away the longest part of our entire build process.

I realise in a Docker environment it's possible to use layer caching to reduce the impact of this, we are currently working in a hybrid environment and have some projects where we need to install onto VMs semi-regularly where a 6.5 minute build for one extension is somewhat problematic.

The generated binary is 55Mb in size:

$ ls -lh /usr/lib/php/20170718/grpc.so
-rw-r--r-- 1 root root 55M Jul 27 10:00 /usr/lib/php/20170718/grpc.so

On our VMs that's not such an issue, but on our Docker environments again that means grpc is actually one of the single biggest contributors to our overall image size.

Anything else we should know

I found #22040 (comment) which suggested the binary size is largely due to debug symbols. This appears to be the case, running strip --strip-debug /usr/lib/php/20170718/grpc.so reduces the size of the binary to 5.0Mb which is much more acceptable.

However I can't see any documentation / information on whether this is recommended and/or whether it's possible to get pecl to build the package with the debug symbols already stripped.

[stanley-cheung]

@acoulton Thanks for the report. Just to clarify - this is not a recent regression right? As in, the PECL extension has always been shipped this way. There's no one particular release where the binary size or build time significantly increased right?

[acoulton]

@stanley-cheung thanks - I can't say for sure as we've only started using it quite recently (and only really now are we starting to use it properly). But certainly it's been very consistent across all the (infrequent) builds in that time, both into Docker containers on GCB and now natively onto VMs.

Just looked and the first experimental build we did was on 19th May, we got 1.28.0 and it compiled to about 62M (so I guess it's improved 😮 😀 ):

1819842  60432 -rwxr-xr-x    1 root     root      61880688 May 19 14:22 /tmp/pear/temp/pear-build-defaultuserGNOAGk/install-grpc-1.28.0/usr/local/lib/php/extensions/no-debug-non-zts-20170718/grpc.so

And that build (on a GCB standard builder) took about 9 minutes to run the following initial section of the Dockerfile on php:7.2-cli-alpine:

	# Build dependencies
	apk add --no-cache --virtual .build-deps \
	   g++ \
	   linux-headers \
	   zlib-dev \
	   $PHPIZE_DEPS \
	; \
	# Runtime deps
	apk add --no-cache \
	   libstdc++ \
	; \
	# Install pecl extensions
	pecl install grpc

Unfortunately there's nothing in the logs with a timestamp between the end of APK and the start of pecl but we have similar package installs on other containers and they're always pretty fast so I'm pretty sure the grpc was the bulk of that 9 minutes...

[MrMage]

This is still an issue with gRPC 1.35; the grpc.so PHP extension has grown to 105 MB by now.

FYI, I have also been suffering from slow compilation. I am now using the following command to tremendously speed up the compilation process, which instructs make to use more than one processor:

MAKEFLAGS="-j $(nproc)" sudo -E pecl install grpc

I am also looking into stripping the binary during compilation now.

[MrMage]

Update: I have not found a way to pass CFLAGS without -g or LDFLAGS with -s -w to the gRPC compilation steps, so I am resorting to stripping the binary after build now with sudo strip --strip-debug /usr/lib/php/*/grpc.so.

[alanondra]

Chiming in to say I'm getting an extremely slow run time for this process, eventually failing with this message:

failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c pecl install grpc]: exit code: 1

[AymericPlanche]

I've also this behavior on my side, compiling from source or using pecl results to a 121M file.

by using strip --strip-debug /usr/lib/php/20190902/grpc.so it reduces to 5.5M

[acoulton]

@stanley-cheung I'm not sure where this was fixed and therefore if I should be waiting for a release?

But FYI I am still seeing this when installing grpc-1.51.1 from pecl on a standard php:8.1-cli-alpine container. This test case on github actions took 15m 51s just for the pecl install and creates a 173.6M binary:

This is the Dockerfile I'm using to test that, I don't think it's doing anything unusual:

FROM php:8.1-cli-alpine

RUN apk add --no-cache --virtual .build-deps g++ linux-headers zlib-dev $PHPIZE_DEPS \
  && apk add --no-cache libstdc++ \
  && date \
  && export MAKEFLAGS="-j $(nproc)" \
  && time pecl install grpc \
  && date \
  && ls -lh /usr/local/lib/php/extensions/no-debug-non-zts-20210902/

[Wirone]

Hi everyone! We also see gRPC's overhead in our builds. In the build's output we can see:

 => => # + strip --strip-all modules/bcmath.so
 => => # + strip --strip-all modules/sockets.so
 => => # + strip --strip-all modules/exif.so
 => => # + strip --strip-all modules/gd.so
 => => # + strip --strip-all modules/intl.so

somehow gRPC's module is not stripped (it's installed in the same batch as extensions seen above). Why? 🤔

Anyway, thanks to comments above I was able to optimise our build both in terms of size and speed 🙂 Thank you very much!

[cmius]

@Wirone

Hi

Mind sharing the dockerfile for this? And how long did it took you to build it?

[Wirone]

@cmius I can't share the Dockerfile since it's proprietary. What I can tell we install gRPC along with several other extensions and when gRPC was added to the Dockerfile, it significantly increased build time. We're migrating from 7.4 to 8.2 and both builds were affected (starting from official PHP images based on Debian).

[jakespencer]

@cmius

You don't really need @Wirone's Dockerfile if you just use the below suggestions, I don't believe @Wirone is doing anything else to decrease build times or container sizes.

Decreasing build times

You can often speed up extension build times by increasing the number of jobs make is allowed to run at once, using the -j flag. So wherever you are using pecl install you can use MAKEFLAGS="-j $(nproc)" pecl install instead and if you are using docker-php-ext-install you can use docker-php-ext-install -j$(nproc) instead.

In both of the above cases we use nproc to work out how many processing units/CPU cores are available and then get make to use that many cores. This of course assumes you want to use all possible cores.

Reducing container size

To strip all your php extensions you can just add this layer

# strip php extensions to decrease container size
RUN \
    set -eux \
    \
    && find "$(php-config --extension-dir)" -name '*.so' -type f -exec strip --strip-all {} \;

The relevant command being:

find "$(php-config --extension-dir)" -name '*.so' -type f -exec strip --strip-all {} \;

This uses find to list all the .so files inside of your php extension directory, that is what "$(php-config --extension-dir)" is for, and finally it runs strip --strip-all on those files.

Other improvements

• Use layers in your docker, so you can utilise the docker build cache - https://docs.docker.com/build/cache/
• Use your own internal base images that contain your standard extensions etc, so every time you build your application container you don't also have to rebuild PHP.

N.B. If you are building multi architecture images using emulation for the different architectures this will significantly impact build times, negatively.

[Wirone]

FYI: we couldn't use nproc because we build images in CI with Kubernetes runners and nproc returns total cores available while runners have limit, so it would lead to many zombie processes that would do nothing. I've hardcoded -j4 (core limit) and build went down from 50 to 20 minutes (we have complex, multi-stage build).

[jontro]

@Wirone 20 minutes is still a long time. Why is the build so large? Most php extensions build in a minute or less

[Wirone]

@jontro we have many OS-level packages to install, 11 extensions to install (9 for prod target, additional 2 for dev), we also copy some files from other internal, pre-built images, we clear few dirs after installation. When it sums up, it takes time 🤷‍♂️. And of course I don't say everything is optimal right now, I just wanted to share the info that we had the same problem with gRPC and were able to reduce this overhead with advices above 🙂

[tomekit]

Our pipeline slowed down by a factor of 10-20x when grpc was added:

We'll probably have to build an image from an already pre-built gRPC installed image, since these build times are getting out of control. pecl install grpc takes around 2200s on a 2 vCPU machine.

[akeebismail]

pre-built gRPC installed image

@tomekit how do you build an image from an already pre-built grpc installed image?
I'm having similar issues of building time longer than 22000s.
please help.

[AndrewCustomSumIT]

@akeebismail After trying many alternate ways to try and build the grpc extension that wouldn't take half an hour we reverted back to creating a separate image that contained grpc, building that and storing to the Docker Hub and then using it as a base image for our main application's Dockerfile and image.

This way means that when we build and deploy our application, it doesn't take 30+ minutes; just the time it takes to pull the grpc image and install the packages/extensions needed for the app.

e.g.
The gRPC image built separately from our app image. This is then tagged as grpc:1.0.0.

FROM php:8.3.10-fpm

RUN apt-get update \
  && apt-get install -y --no-install-recommends \
    libz-dev \
  && apt-get clean \
  && rm -rf /var/lib/apt/lists/*

RUN pecl install grpc \
  && docker-php-ext-enable grpc

RUN strip --strip-debug /usr/local/lib/php/extensions/no-debug-non-zts-20230831/grpc.so

The app image then contains it as a base:

FROM grpc:1.0.0 AS grpc

FROM grpc AS base

WORKDIR /var/www/html

RUN apt-get update
RUN apt-get install -y --no-install-recommends \
    libicu-dev \
    libnss3-dev \
    libonig-dev \
    libvips-dev \
    libyaml-dev \
 && apt-get clean \
 && rm -rf /var/lib/apt/lists/*