Windows: RProtoBuf fails for large messages (64MB+)

[maciejlach]

RProtoBuf fails on Windows, while parsing large messages with following error:

[libprotobuf ERROR ../../protobuf-3.0.0/src/google/protobuf/io/coded_stream.cc:207] 
A protocol message was rejected because it was too big (more than 67108864 bytes).  
To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.

Since protobuf 3.2, default parsing byte limit has been increased to 2GB. It seems that RProtoBuf on Windows uses protobuf library 3.0.0 provided by rwinlib.

Could you please upgrade Windows build to use protobuf 3.2.x+ library?

RProtoBuf version: 0.4.11

This issue doesn't occur on Linux if libprotobuf is installed in version 3.2.x+.

[eddelbuettel]

We provide the source package so you may have to provide such a beefed-up binary for Windows yourself.

[eddelbuettel]

Or else you could liaise with @jeroen and see if the prebuild Windows library can be update.

It is used by this function invoked from src/Makevars.win.

[jeroen]

It looks like I already did a build of 3.2.0 a year ago: https://github.com/rwinlib/protobuf/releases. Not sure why we didn't upgrade here. You could try to replace 3.0.0. with 3.2.0 in src/Makevars.win and tools/winlib.R and submit to winbuilder to try your luck.

[maciejlach]

@eddelbuettel @jeroen I have used the prebuilt 3.2.0 lib to build RProtoBuf locally and than the large messages are parsed correctly. I've linked this library in initial issue, but I was too sparse with description (sorry about that).

I've committed it to a fork: maciejlach@40bc429.

Can I provide a PR based on this commit?
I believe, it would be much more convenient to have this available at CRAN and at master repository. Especially for packages using a RProtoBuf as a dependency.

[eddelbuettel]

Looks like a clean change, so yes, please send a PR.

[jeroen]

OK merged :) I may try to build a new protobuf one of these days, seems like they're at 3.5.1 already.

[maciejlach]

Thank you.

[eddelbuettel]

For completeness, did anybody test this on win-builder?

[jeroen]

Yes I did. I wasn't able to build the vignettes so I used --no-build-vignettes but other than that it passes on win builder.