Allow fetching the latest grpc and protobuf source packages with core_gems

[apolcyn]

Consider this as just one idea for how this could be done (I don't know much about the PR the testing in this repo and I've tested so far by checking the results of bin/gem_downloader core_gems.rb).

This should fix protocolbuffers/protobuf#4210, by allowing google-fluentd to not need to pin grpc and protobuf packages (grpc does not need a pin yet but it's tentative 1.9 release would start to need a pin).

This is basically a workaround for the facts that:

1. The grpc/protobuf packaging process (done using rake-compiler), outputs an unnecessary and also unusable shared object when creating the pre-built shared objects packages (e.g., grpc-1.8.7-x86_64-linux)

2. omnibus checks the health of every shared object in the gem installation directory using ldd.

Longer term solution might be to patch rake-compiler to provide an option to not generate that fallback binary when we're creating a package with pre-built shared objects.

[igorpeshansky]

Haven't taken a look at the PR yet, but wouldn't deleting the shared object as part of the grpc packaging process also accomplish this goal?

[apolcyn]

Haven't taken a look at the PR yet, but wouldn't deleting the shared object as part of the grpc packaging process also accomplish this goal?

That could also work but I don't have a great way to do that. The grpc package process just invokes the packaging rake tasks provided by rake-compiler, and then we upload the gems that rake-compiler creates. It might be possible to modify these gems after-the-fact and remove the fallback binary, but I'd like to avoid that if we can. IMO the best fix is if rake-compiler had that capability built in.

[qingling128]

Commented: protocolbuffers/protobuf#4210 (comment)

[qingling128]

Per discussion in protocolbuffers/protobuf#4210, we will live with the manual pins for now and wait for rake-compiler/rake-compiler#146 as a long term fix.

Closing this PR for now. Will reopen if necessary.