New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support snapshot files above 4GB #52

Closed
ongardie opened this Issue Feb 14, 2014 · 1 comment

Comments

Projects
None yet
1 participant
@ongardie
Copy link
Member

ongardie commented Feb 14, 2014

Server/SnapshotFile.{cc,h} uses the protobuf IO library, which internally uses 32-bit integer file offsets. So:

  1. Write a test program that creates a very big log and snapshot. The easiest way to do this is probably to add a number of objects flag to Examples/Benchmark.cc (it creates small snapshots now because it overwrites a single object over and over again).
  2. Rewrite Server/SnapshotFile.{cc,h} and their users in terms of normal write() operations rather than using protobuf IO. It's still fine to use protobufs for each individual record in the StateMachine code, just not ok to use protobuf IO to write the whole file.

@ongardie ongardie modified the milestone: 1.0.0 Mar 4, 2015

@ongardie

This comment has been minimized.

Copy link
Member

ongardie commented Mar 4, 2015

I'm getting this PANIC on boot with an 888MB snapshot:

[libprotobuf WARNING google/protobuf/io/coded_stream.cc:505] Reading dangerously large protocol message.  If the message turns out to be larger than 67108864 bytes, parsing will be halted for security reasons.  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message was rejected because it was too big (more than 67108864 bytes).  To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
2015-03-04 21:37:01.570170 Tree/Tree.cc:93 in loadSnapshot() ERROR[1:StateMachine]: couldn't read snapshot Exiting...

@ongardie ongardie self-assigned this Mar 4, 2015

ongardie added a commit that referenced this issue Mar 4, 2015

Isolate use of ProtoBuf::io library to Storage/SnapshotFile.cc
Towards #52: support snapshot files above 4GB

ongardie added a commit that referenced this issue Mar 4, 2015

Isolate use of ProtoBuf::io library to Storage/SnapshotFile.cc
Towards #52: support snapshot files above 4GB

@ongardie ongardie added the bigdata label Mar 5, 2015

@ongardie ongardie closed this in 5e6bf93 Mar 5, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment