Protocol Buffers are a language-neutral, platform-neutral, extensible way of serializing structured data for use in communications protocols, data storage, and more, originally designed at Google (see).
protobuf.js is a pure JavaScript implementation for node and the browser. It efficiently encodes plain objects and custom classes and works out of the box with .proto files.
Recommended read: Changes in protobuf.js 6.0
- Optimized for performance
- Exhaustive browser support
- Managed TypeScript definitions
- Elaborate API documentation
- Convenient CLI utilities
- Seamless browserify integration
-
Usage
How to include protobuf.js in your project. -
Examples
A few examples to get you started. -
Module Structure
A brief introduction to the structure of the exported module. -
Documentation
A list of available documentation resources. -
Command line
How to use the command line utility. -
Building
How to build the library and its components yourself. -
Performance
A few internals and a benchmark on performance. -
Compatibility
Notes on compatibility regarding browsers and optional libraries.
$> npm install protobufjs
var protobuf = require("protobufjs");
Development:
<script src="//cdn.rawgit.com/dcodeIO/protobuf.js/6.1.0/dist/protobuf.js"></script>
Production:
<script src="//cdn.rawgit.com/dcodeIO/protobuf.js/6.1.0/dist/protobuf.min.js"></script>
The protobuf
namespace will be available globally.
NOTE: Remember to replace the version tag with the exact release your project depends upon.
// awesome.proto
package awesomepackage;
syntax = "proto3";
message AwesomeMessage {
string awesome_field = 1; // becomes awesomeField
}
protobuf.load("awesome.proto", function(err, root) {
if (err) throw err;
// Obtain a message type
var AwesomeMessage = root.lookup("awesomepackage.AwesomeMessage");
// Create a new message
var message = AwesomeMessage.create({ awesomeField: "AwesomeString" });
// Encode a message
var buffer = AwesomeMessage.encode(message).finish();
// ... do something with buffer
// Or, encode a plain object
var buffer = AwesomeMessage.encode({ awesomeField: "AwesomeString" }).finish();
// ... do something with buffer
// Decode a buffer
var message = AwesomeMessage.decode(buffer);
// ... do something with message
// If your application uses length-delimited buffers, there is also encodeDelimited and decodeDelimited.
});
You can also use promises by omitting the callback:
protobuf.load("awesome.proto")
.then(function(root) {
...
});
...
var Root = protobuf.Root,
Type = protobuf.Type,
Field = protobuf.Field;
var AwesomeMessage = new Type("AwesomeMessage").add(new Field("awesomeField", 1, "string"));
var root = new Root().define("awesomepackage").add(AwesomeMessage);
// Continue at "Create a new message" above
...
...
function AwesomeMessage(properties) {
protobuf.Message.call(this, properties);
}
protobuf.Class.create(root.lookup("awesomepackage.AwesomeMessage") /* or use reflection */, AwesomeMessage);
var message = new AwesomeMessage({ awesomeField: "AwesomeString" });
// Continue at "Encode a message" above
Custom classes are automatically populated with static encode
, encodeDelimited
, decode
, decodeDelimited
and verify
methods and reference their reflected type via the $type
property. Note that there are no methods (just $type
) on instances by default as method names might conflict with field names.
// greeter.proto
service Greeter {
rpc SayHello (HelloRequest) returns (HelloReply) {}
}
message HelloRequest {
string name = 1;
}
message HelloReply {
string message = 1;
}
...
var Greeter = root.lookup("Greeter");
var greeter = Greeter.create(rpcImpl, false, false); // rpcImpl (see below), requestDelimited?, responseDelimited?
greeter.sayHello({ name: 'you' }, function(err, response) {
console.log('Greeting:', response.message);
});
To make this work, all you have to do is provide an rpcImpl
, which is an asynchronous function that takes the reflected service method, the binary HelloRequest and a node-style callback as its parameters. For example:
function rpcImpl(method, requestData, callback) {
// perform the request using an HTTP request or a WebSocket for example
var responseData = ...;
// and call the callback with the binary response afterwards:
callback(null, responseData);
}
There is also an example for streaming RPC.
/// <reference path="node_modules/protobufjs/types/protobuf.js.d.ts" />
import * as protobuf from "protobufjs";
...
The library exports a flat protobuf
namespace with the following members, ordered by category:
-
load(filename:
string|string[]
, [root:Root
], [callback:function(err: Error, [root: Root])
]):Promise|undefined
[source]
Loads one or multiple .proto or preprocessed .json files into a common root namespace. -
loadSync(filename:
string|string[]
, [root:Root
]):Root
[source]
Synchronously loads one or multiple .proto or preprocessed .json files into a common root namespace (node only). -
tokenize(source:
string
):Object
[source]
Tokenizes the given .proto source and returns an object with useful utility functions. -
parse(source:
string
):Object
[source]
Parses the given .proto source and returns an object with the parsed contents.-
package:
string|undefined
The package name, if declared. -
imports:
string[]|undefined
File names of imported files, if any. -
weakImports:
string[]|undefined
File names of weakly imported files, if any. -
syntax:
string|undefined
Source syntax, if defined. -
root:
Root
The root namespace.
-
-
Writer [source]
Wire format writer usingUint8Array
if available, otherwiseArray
. -
BufferWriter extends Writer [source]
Wire format writer using node buffers. -
Reader [source]
Wire format reader usingUint8Array
if available, otherwiseArray
. -
BufferReader extends Reader [source]
Wire format reader using node buffers. -
codegen [source]
A closure for generating functions programmatically.
-
ReflectionObject [source]
Base class of all reflection objects. -
Namespace extends ReflectionObject [source]
Base class of all reflection objects containing nested objects. -
Root extends Namespace [source]
Root namespace. -
Type extends Namespace [source]
Reflected message type. -
Field extends ReflectionObject [source]
Reflected message field. -
MapField extends Field [source]
Reflected message map field. -
Enum extends ReflectionObject [source]
Reflected enum. -
Service extends Namespace [source]
Reflected service. -
Method extends ReflectionObject [source]
Reflected service method.
-
Class [source]
Runtime class providing the tools to create your own custom classes. -
Message [source]
Abstract runtime message.
-
types:
Object
[source]
Common type constants. -
common(name:
string
, json:Object
) [source]
Provides common type definitions. -
rpc:
Object
[source]
Streaming RPC helpers. -
util:
Object
[source]
Various utility functions.
The pbjs
command line utility can be used to bundle and translate between .proto and .json files.
Consolidates imports and converts between file formats.
-t, --target Specifies the target format. Also accepts a path to require a custom target.
json JSON representation
json-module JSON representation as a module (AMD, CommonJS, global)
proto2 Protocol Buffers, Version 2
proto3 Protocol Buffers, Version 3
static Static code without reflection
static-module Static code without reflection as a module (AMD, CommonJS, global)
-p, --path Adds a directory to the include path.
-o, --out Saves to a file instead of writing to stdout.
-w, --wrap Specifies an alternative wrapper for any *-module target.
usage: pbjs [options] file1.proto file2.json ...
For production environments it is recommended to bundle all your .proto files to a single .json file, which reduces the number of network requests and parser invocations required:
$> pbjs -t json file1.proto file2.proto > bundle.json
Now, either include this file in your final bundle:
var root = protobuf.Root.fromJSON(require("./bundle.json"));
or load it the usual way:
protobuf.load("bundle.json", function(err, root) {
...
});
To build the library or its components yourself, clone it from GitHub and install the development dependencies:
$> git clone https://github.com/dcodeIO/protobuf.js.git
$> cd protobuf.js
$> npm install --dev
Building the development and production versions with their respective source maps to dist/
:
$> npm run build
Building the documentation to docs/
:
$> npm run docs
Building the TypeScript definition to types/
:
$> npm run types
protobuf.js integrates into any browserify build-process. There are a few possible tweaks:
- If performance is a concern or IE8 support is required, you should make sure to exclude the browserified
buffer
module and let protobuf.js do its thing with Uint8Array/Array instead. - If you do not need int64 support, you can exclude the
long
module. - If your application does not rely on the following modules and package size is a concern, you can also exclude
process
,_process
andfs
. - If you have any special requirements, there is the bundler as a reference.
The package includes a benchmark that tries to compare performance to native JSON as far as this is possible. On an i7-2600K running node 6.9.1 it yields:
benchmarking encoding performance ...
Type.encode to buffer x 479,876 ops/sec ±0.64% (92 runs sampled)
JSON.stringify to string x 311,489 ops/sec ±0.84% (87 runs sampled)
JSON.stringify to buffer x 175,079 ops/sec ±1.48% (82 runs sampled)
Type.encode to buffer was fastest
JSON.stringify to string was 35.2% slower
JSON.stringify to buffer was 63.8% slower
benchmarking decoding performance ...
Type.decode from buffer x 1,267,612 ops/sec ±1.18% (90 runs sampled)
JSON.parse from string x 291,707 ops/sec ±1.12% (92 runs sampled)
JSON.parse from buffer x 262,640 ops/sec ±0.77% (89 runs sampled)
Type.decode from buffer was fastest
JSON.parse from string was 77.0% slower
JSON.parse from buffer was 79.2% slower
benchmarking combined performance ...
Type to/from buffer x 248,897 ops/sec ±0.89% (90 runs sampled)
JSON to/from string x 126,848 ops/sec ±0.75% (92 runs sampled)
JSON to/from buffer x 89,854 ops/sec ±0.79% (93 runs sampled)
Type to/from buffer was fastest
JSON to/from string was 49.0% slower
JSON to/from buffer was 63.9% slower
benchmarking verifying performance ...
Type.verify x 5,941,014 ops/sec ±0.96% (90 runs sampled)
Type.verify was fastest
Note that JSON is a native binding nowadays and as such is about as fast as it possibly can get. So, how can protobuf.js be faster?
- The benchmark is somewhat flawed.
- Reader and writer interfaces configure themselves according to the environment to eliminate redundant conditionals.
- Node-specific reader and writer subclasses benefit from node's buffer binding.
- Reflection has built-in code generation that builds type-specific encoders, decoders and verifiers at runtime.
- Encoders and decoders do not verify that required fields are present (with proto3 this is dead code anyway). There is a
verify
method to check this manually instead - where applicable. - For entirely bogus values encoders intentionally rely on runtime errors to be thrown somewhere down the road.
- Quite a bit of V8-specific profiling is accountable for everything else.
Note that code generation requires new Function(...)
(basically eval
) support and that an equivalent but slower fallback will be used where unsupported.
You can also run the benchmark ...
$> npm run bench
and the profiler yourself (the latter requires a recent version of node):
$> npm run prof <encode|decode|encode-browser|decode-browser> [iterations=10000000]
Note that as of this writing, the benchmark suite performs significantly slower on node 7.2.0 compared to 6.9.1 because moths.
- Because the internals of this package do not rely on
google/protobuf/descriptor.proto
, options are parsed and presented literally. - If typed arrays are not supported by the environment, plain arrays will be used instead.
- Support for pre-ES5 environments like IE8 can be achieved by using a polyfill and, instead of using property getters and setters on reflection objects, calling the respective functions prefixed with
get
,set
oris
directly (i.e. callingType#getFieldsById()
instead of accessingType#fieldsById
). - If you need a proper way to work with 64 bit values (uint64, int64 etc.), you can install long.js alongside this library. All 64 bit numbers will then be returned as a
Long
instance instead of a possibly unsafe JavaScript number (see).
License: Apache License, Version 2.0, bundled external libraries may have their own license