Make it possible to build Ruby gem with system OpenSSL

[stanhu]

This pull request adds a number of changes to enable compiling gRPC for s390x:

1. Bring back ability to compile against system OpenSSL

#23957 removed the ability to build against the system OpenSSL libraries. However, BoringSSL is not supported on such platforms as the IBM s390x. This commits restores the behavior of that merge request just for OpenSSL.

2. Make it possible to build Ruby gem with system OpenSSL

By installing the gem with EMBED_OPENSSL=false, the Ruby gem can be compiled using the system OpenSSL library. This is needed on systems where BoringSSL doesn't compile (e.g. s390x).

3. Allow configuration of number of jobs to build Ruby gem

Some systems may be memory-constrained, so launching N jobs for N processors will cause the compilation to run out of memory. We add an environment variable, MAKE_NPROC, that allows users to use a custom number of jobs.

For example, I was able to build on s390x via:

MAKE_NPROC=1 EMBED_OPENSSL=false gem install pkg/grpc-1.42.0.dev.gem

Closes #15997

@veblush @jtattermusch

[jtattermusch]

Unless we have a very strong demand for this, I'm generally against any changes that make our build system harder to maintain and that increases the number of build configurations we support (and which we should test consequently).

In particular, we're trying to keep the complexity of our Makefile as low as possible, which is also one of the reasons why we introduced #23957 while ago. Since s390x is not a platform we officially support, I don't think we're willing to take the maintenance burden of increasing our Makefile's complexity (e.g. by reintroducing support of building against OpenSSL). If anything, our Makefile should be getting simpler over time.

[stanhu]

@jtattermusch That's understandable, but note that the CMakeLists.txt still supports gRPC_SSL_PROVIDER and other flags that make it possible to build with an external libraries. Are you planning on removing that as well? I looked at using that for the Ruby gem, but the gem doesn't ship the cmake related files. I'm wondering if you would be open to a pull request that used cmake.

[jtattermusch]

@jtattermusch That's understandable, but note that the CMakeLists.txt still supports gRPC_SSL_PROVIDER and other flags that make it possible to build with an external libraries. Are you planning on removing that as well? I looked at using that for the Ruby gem, but the gem doesn't ship the cmake related files. I'm wondering if you would be open to a pull request that used cmake.

we're planning to keep the option to build against openssl with cmake (out of necessity), but we don't plan to introduce that option for other build systems (the complexity argument above).

I think building ruby gem using cmake would be very difficult as the build system for ruby extensions uses make internally (which is one of the main reasons why we have to keep supporting make at least for project-internal use - otherwise we would have removed the make support long time ago). I think trying to revamp the grpc ruby extension build to use cmake is a lost cause and I recommend against trying that - you'd probably burn a LOT of your own time and it wouldn't lead to anywhere.

[stanhu]

@jtattermusch Ok, thanks. FYI, I did spend a few minutes trying to use cmake in extconf.rb. I got pretty far, except the one hiccup is that the current CMakeLists.txt tries to build almost everything (e.g. compiling some .proto files), whereas the current Makefile only builds libgrpc.a. There would need to be changes to CMakeLists.txt to whittle down the build list. Alternatively, the gem would have to include more of the source than it does.

I will mention that gems like libgit2/rugged already requre cmake to build the gem, so supporting cmake isn't unprecedented.

I'd argue that providing the ability to use the system OpenSSL library is not just a architecture-specific issue. Having the ability to use the system OpenSSL library is important for FIPS compliance. I'd propose that the Ruby extension support something like a --use-system-ssl option to enable this.