parseDelimitedFrom behavior has changed between 2.5 and 3.1

[saintstack]

parseDelimitedFrom didn't check limit in 2.5. In 3.1 it does throwing InvalidProtocolBufferException if we exceed the hard-coded max of 64MB.

This is issue is a bit like "Remove proto size restriction when parsing protos. #836" but parseDelimitedFrom only takes InputStream so we can't set max size.

64MB is large but previous there was no limit. There is a sort of a workaround by doing parsePartialFrom and creating a CodedInputStream probably set to this method but means creating a new object every time (or reading to see if > 64MB and then making new CIS...)

Let us know how you think best to address and we can put up a patch.

Thanks.

org.apache.hadoop.hbase.shaded.com.google.protobuf.InvalidProtocolBufferException: Protocol message was too large.  May be malicious.  Use CodedInputStream.setSizeLimit() to increase the size limit.
at org.apache.hadoop.hbase.shaded.com.google.protobuf.InvalidProtocolBufferException.sizeLimitExceeded(InvalidProtocolBufferException.java:134)
at org.apache.hadoop.hbase.shaded.com.google.protobuf.CodedInputStream$StreamDecoder.readRawBytesSlowPath(CodedInputStream.java:2768)
at org.apache.hadoop.hbase.shaded.com.google.protobuf.CodedInputStream$StreamDecoder.readBytes(CodedInputStream.java:2335)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos$Procedure.<init>(ProcedureProtos.java:564)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos$Procedure.<init>(ProcedureProtos.java:423)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos$Procedure$1.parsePartialFrom(ProcedureProtos.java:2498)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos$Procedure$1.parsePartialFrom(ProcedureProtos.java:2493)
at org.apache.hadoop.hbase.shaded.com.google.protobuf.CodedInputStream$StreamDecoder.readMessage(CodedInputStream.java:2314)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos$ProcedureWALEntry.<init>(ProcedureProtos.java:6565)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos$ProcedureWALEntry.<init>(ProcedureProtos.java:6505)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos$ProcedureWALEntry$1.parsePartialFrom(ProcedureProtos.java:7660)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ProcedureProtos$ProcedureWALEntry$1.parsePartialFrom(ProcedureProtos.java:7655)
at org.apache.hadoop.hbase.shaded.com.google.protobuf.AbstractParser.parsePartialFrom(AbstractParser.java:197)
at org.apache.hadoop.hbase.shaded.com.google.protobuf.AbstractParser.parsePartialDelimitedFrom(AbstractParser.java:238)

[carl-mastrangelo]

Where does gRPC fit in? I don't see the gRPC code anywhere in the stack trace, even the proto decoder.

[saintstack]

You are right. Pardon me. Will file at pb.