logging: use gRPC

[stephenplusplus]

RE: #1049

This transitions our Logging API to use gRPC.

To Dos

• Tests
  • System
  • Unit
    • GrpcService
    • Logging
      • entry
      • index
      • log
      • sink

⚠️ Breaking Change!

- log.createWriteStream();

Writing to a log from a stream is gone. gRPC (the library) doesn't see WriteLogEntries as a stream.

[jgeewax]

/cc @jayantkolhe

[murgatroid99]

On the gRPC side, we are working on static Node.js code generation using protoc, and I expect that to be complete within a few weeks. Since this is a first party solution, we would prefer that Google APIs use it to publish client libraries.

This will have a different API than the current dynamically generated code does: instead of passing and receiving bare JavaScript objects, users will build and use objects defined in protobuf-specific classes. This will allow for more immediate and informative input type-checking, and it will be more consistent with other languages.

As a side note, gRPC doesn't see WriteLogEntries as a stream because it is not declared as a stream in the proto file. protoc would see the same thing.

[stephenplusplus]

Thanks! I'm sorry to have to admit, but I'm not quite sure where to begin to realize that vision. Is there any documentation, example code, guides, or reading material you can point me to?

[murgatroid99]

The pull request protocolbuffers/protobuf#1215 contains a significant update to JavaScript protobuf code generation and documentation, so that is probably a good place to start.

[stephenplusplus]

So does this sound right:?

1. Added support for CommonJS require() protocolbuffers/protobuf#1215 merges
2. I use the CLI tool protoc to feed in proto files from the cloud services. We get a generated JS file back?
3. We give that generated file to gRPC, which returns an API that this library will interact with
4. That API won't be similar to what it is now... "...users will build and use objects defined in protobuf-specific classes" - can you elaborate on this?

[murgatroid99]

More accurately:

1. That PR merges
2. On the gRPC side, we create a plugin for protoc that generates service files. This is the part that will take a few weeks.
3. You use the CLI tool protoc to feed in proto files and get generated JS service and method definition files.
4. You require('generated_service.js') (which has its own require('grpc') internally) and you get something very similar to the output of grpc.load('service.proto').

Regarding the API change: currently, the objects you pass as arguments and receive as responses in Node gRPC are regular JavaScript objects that are converted on the fly to or from ProtoBuf.js Message objects. With this new generated code, users will instead pass in and receive protobuf Message objects of the relevant types.

[stephenplusplus]

Thanks for clearing that up.

With this new generated code, users will instead pass in and receive protobuf Message objects of the relevant types.
This will allow for more immediate and informative input type-checking, and it will be more consistent with other languages.

Can't mismatched input types from a plain JavaScript object be caught by gRPC at runtime?

[murgatroid99]

Can't mismatched input types from a plain JavaScript object be caught by gRPC at runtime?

It can, but currently that check is done when the message is written. If we require users to pass protobuf Message objects, they will get that type checking when they are building the message objects.

[stephenplusplus]

In our case, those events will be in succession, since I think we want to continue to allow plain objects as input for our API. Our library would have to do a lot of plumbing just to convert those objects to Message objects, so having that handled for us has been quite a help.

[stephenplusplus]

@murgatroid99 is there a PR or issue I can subscribe to for the protoc plugin that generates service files?

[murgatroid99]

The issue is at grpc/grpc#5418

[stephenplusplus]

I'm going to move forward with the goal of getting this released as-is before the end of the month. After that, we'll iterate where necessary to align with the best practices around gRPC.

[murgatroid99]

I have a PR for this at grpc/grpc#5189. I believe I can have this complete and submitted within the next few days; definitely by the end of the week.

[stephenplusplus]

Sorry if I'm just missing the connection, but did you mean grpc/grpc#5445?

[murgatroid99]

Yes, that's right. That was just a copy/paste error

[stephenplusplus]

I'm subscribed to that one, so I'll be sure to follow along and make the updates here when the PR is merged and a new release is cut. For now, @callmehiphop PTAL!

[callmehiphop]

@stephenplusplus aside from the convertBuffers_ bug, everything LGTM!