Failed to parse server response

[akoenig]

Thanks for compiling this library 👋 👍 I just tried to build a little prototype in order to get a feeling of how everything is like in Node.js land.

Started liftbridge as described in the quick start:

docker run -d --name=liftbridge-main -p 4222:4222 -p 9292:9292 -p 8222:8222 -p 6222:6222 liftbridge/liftbridge-docker

Created a stream, published a message and read via a subscription:

import LiftbridgeClient, { Message } from "liftbridge";
import LiftbridgeMessage from "liftbridge/lib/message";
import LiftbridgeStream from "liftbridge/lib/stream";

const app = async () => {
  const client = new LiftbridgeClient("localhost:9292");

  await client.connect();

  const name = `my-stream-v${Date.now()}`;
  const subject = `my-subject-v${Date.now()}`;

  const streamDef = new LiftbridgeStream({
    name,
    subject,
    partitions: 1,
    maxReplication: true
  });

  await client.createStream(streamDef);

  await client.publish(
    new LiftbridgeMessage({
      subject,
      key: Date.now().toString(),
      value: JSON.stringify({
        foo: "bar"
      })
    })
  );

  const subscription = client.subscribe(
    new LiftbridgeStream({
      subject,
      name
    })
  );

  subscription.on("data", (data: Message) => {
    console.log("message");
    console.log(data);
  });
};

app().catch(err => {
  process.exitCode = 1;

  console.error(err);
});

Creating the stream and publishing the message seems to work. Consuming the stream fails with a:

Error: 13 INTERNAL: Failed to parse server response

I also tried to generate a new gRPC binding, but this leads to the same result above 🙁

[paambaati]

@akoenig It is a known issue, and hence the note about the project being under development.

I've spent considerable time tracing the issue, and you can see the issues I've reported —

1. clients in other languages liftbridge-io/go-liftbridge#11 (comment)
2. Failed to parse server response when I try to read a stream grpc/grpc-node#1022
3. Empty strings read as undefined in Map fields protobufjs/protobuf.js#1293
4. Cannot deserialize empty strings as Map values in JavaScript protocolbuffers/protobuf-javascript#43

tl;dr

protocolbuffers/protobuf-javascript#43 needs to be fixed before this library can work, but I've been unable to come up with a fix for it because - a) I'm unfamiliar with the protobuf codebase and b) The maintainers haven't responded to the issue yet.

[tylertreat]

Btw we are planning to replace Protocol Buffers with Flatbuffers, so maybe this issue will go away with that. liftbridge-io/liftbridge-api#8

[paambaati]

@tylertreat Oh wow, that’ll be great! Although Flatbuffers is once again a Google project and support has been abysmal so far on even widely reported issues. Hopefully the situation is better for Flatbuffers; I’ll start looking at it.

[llchan]

In my experience Flatbuffers folks have generally been pretty responsive, esp compared to the larger projects like protobuf/bazel/TF/etc. If there's a real bug I think they'd be more than willing to work on solving it. That said, I don't know what the nodejs support looks like (I'm most familiar with the C++/Go interfaces).

[paambaati]

@llchan Thanks for chiming in!

I don't know what the nodejs support looks like

This is exactly what I’m worried about.

[minicuper]

I get this one

Error: Unknown base64 encoding at char: {
at c (.../nats/node_modules/google-protobuf/google-protobuf.js:186:210)
at Object.goog.crypt.base64.decodeStringInternal_ (.../nats/node_modules/google-protobuf/google-protobuf.js:186:312)
at Object.goog.crypt.base64.decodeStringToUint8Array (.../nats/node_modules/google-protobuf/google-protobuf.js:185:385)
at Function.jspb.Message.bytesAsU8 (.../nats/node_modules/google-protobuf/google-protobuf.js:200:385)

[digitalsanity]

@paambaati
Can you confirm that it runs correctly (npm run debug) if you manually add the final "" arg to the _pb.js file generated as in my example below? I think it is working but the debug app isn't exiting -- just hanging at the end?

proto.proto.Message.deserializeBinaryFromReader = function(msg, reader) {
// ..
   case 7:
      var value = msg.getHeadersMap();
      reader.readMessage(value, function(message, reader) {
        // NOTICE THE NUMBER OF ARGUMENTS HERE -->>
        jspb.Map.deserializeBinary(message, reader, jspb.BinaryReader.prototype.readString, jspb.BinaryReader.prototype.readBytes, null, "", "");
      });
      break;
// ...
}

[paambaati]

@zag2art @digitalsanity Hey I’m terribly sorry folks, but I plan to resume work here only once liftbridge-io/liftbridge-api#8 lands. It anyways requires changing a lot of the core stuff, so I’d rather wait for it than spending more time with Protobuf errors.

[paambaati]

@zag2art @digitalsanity protobufjs/protobuf.js#1348 should hopefully fix this issue.

Also, I recently learnt that Liftbridge v1.0.0 (expected to ship in March) is sticking to gRPC for now.