[NEW EXAMPLE] Protobuf - Tokenizer

[ainzzorl]

General

• Project name: Protocol Buggers aka Protobuf
• Example name: Tokenizer
• Project home page: https://github.com/protocolbuffers/protobuf
• Programming language(s): C++

Description

Protocol Buffers (a.k.a., protobuf) are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. You can find protobuf's documentation on the Google Developers site.

Parsing protobufs depends on this custom stream tokenizer.

Why it needs a custom tokenizer and why it is the way it is is described in detail in this comment.

Links

• https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/io/tokenizer.cc
• https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/io/tokenizer_unittest.cc

What makes it interesting

• Protobuf is a very famous software.
• Very well documented code.
• Very well explained design decisions.

Related work

No.

Other

The tokenizer is not the most interesting part of Protobuf, but the rest of the code seems either overwhelmingly complex, or impossible to reason about in isolation, usually both.