Warning: only protocol buffers 3 is supported. Use protobuf-elixir if you need support for version 2 (protobuf-elixir was a major inspiration for this project).
This is a protocol buffer encoder and decoder. Its goal is to be fast at the cost of larger generated files. This is achieved by generating a significant part of the encoding and decoding logic at generation time with the protoc plugin.
Encoding and decoding performance is ~3-4x times faster than protobuf-elixir. For example, if we take the %Everything
structure used in our tests, which has all field types, including all array types (with 2 values per array) and a few maps, pbuf
takes ~14µs to encode and ~24µs to decode, versus 66µs and 67µs. However, the .beam file is quite a bit larger: 19K vs 7K.
(Note that there is limited support for version 2 syntax, but only enough to allow the protoc plugin to bootstrap itself. This may may or may not provide all the version 2 support you need).
Assuming you already have protoc installed, you'll want to run:
$ mix escript.install hex pbuf
to install the pbuf elixir generator. This will place protoc-gen-fast-elixir
in your ~/.mix/escript/
folder. This must be on your $PATH.
You can then generate elixir files using the protoc
command with the -fast-elixir_out=PATH
flag:
protoc --fast-elixir_out=generated/ myschema.proto
Note the name fast-elixir_out
. This allows you to also have protobuf installed in order to support proto2 syntax.
The generated code is normal Elixir modules with a defstruct
. Use new/1
to create new instances:
user = Models.User.new(name: "leto", age: 2000)
And Pbuf.encode!/1
and Pbuf.encode_to_iodata!/1
to encode them:
data = Pbuf.encode!(user)
Only structures generated by protoc
can be passed to encode!/1
and encode_to_iodata!/1
; you cannot pass maps or other structures.
These functions will raise a Pbuf.Encoder.Error
on invalid data (such as assigning a float
to a bool
field). There are currently no non-raising functions.
Decoding is done via Pbuf.decode!/2
or Pbuf.decode/2
:
user = Pbuf.decode!(Models.User, data)
As an alternative, you can also use: Models.User.decode!(data)
or decode/1
to avoid raising on invalid data.
A field declared as an enum
should be set to the atom representation of the protocol buffer name, or the integer value. For example, a message defined as:
message User {
UserType type = 1;
}
enum UserType {
USER_TYPE_UNKNOWN = 0;
USER_TYPE_PENDING = 1;
USER_TYPE_NORMAL = 2;
USER_TYPE_DELETED = 3;
}
Should be used as:
user = User.new(type: :USER_TYPE_PENDING)
# OR
user = User.new(type: 1)
(casing is preserved from the proto file)
You'll likely want to map your protocol buffer enums to specific atoms. With a bit of work, the generator can do this for you.
First, you'll need to specify a custom option, say in options.proto
:
syntax = "proto2";
import 'google/protobuf/descriptor.proto';
extend google.protobuf.EnumValueOptions {
optional string elixir_atom = 78832;
}
You can them import this .proto file like any other and use the option:
import 'options.proto';
enum HTTPMethod {
HTTP_METHOD_GET = 0 [(elixir_atom) = 'get'];
HTTP_METHOD_POST = 1 [(elixir_atom) = 'post'];
}
The value will now be :get
and :post
rather than :HTTP_METHOD_GET
and :HTTP_METHOD_POST
.
For this to work, Google's proto definitions must be available when you run protoc
:
protoc -I=/usr/local/include/proto/ -I=. ...
They are available from the protocol buffer source: https://github.com/protocolbuffers/protobuf/releases/download/v3.6.1/protoc-3.6.1-osx-x86_64.zip.
By default, a oneof field must be set to a tuple where the first element is the name of the field and the second is the value. Given:
message Event {
oneof event_oneof {
Commit commit = 1;
Wiki wiki = 2;
}
}
Then valid values for event_oneof
are: nil, {:commit, Commit.t}
or {:wiki, Wiki.t}
.
Generated structures have a @derive Jason.Encoder
. For simple messages, this means you can use Jason.encode(struct)
to generate a json representation of your messages.
This fails for oneofs, since Jason can't encode tuples ({:type, value
}). You can configure a different oneof
format by using the special elixir_oneof_format
option:
extend google.protobuf.EnumValueOptions {
optional string elixir_atom = 78832;
}
And then, at the file level, specifying either:
option (elixir_oneof_format) = 1
and using:
%{__type: :commit, value: Commit.t}
OR specifying
option (elixir_oneof_format) = 2
and using:
%{commit: Commit.t}
It's possible to not generate @derive Jason.Encoder
on a per-message basis by using a custom option, say in options.proto
:
syntax = "proto2";
extend google.protobuf.MessageOptions {
int32 json_message = 78832;
}
And then using it in your message:
message Something {
option (json_message) = 0;
...
}
It's possible to automatically encode and decode a bytes
field to and from Json. First, define a FieldOptions
:
syntax = "proto2";
extend google.protobuf.MessageOptions {
int32 json_field = 78832;
}
And then using it in your message:
message Something {
bytes data = 1 [(json_field) = 1];
}
Which results in:
something = [data: %{over: 9000}]
|> Something.new()
|> something.encode!()
|> Somethihg.decode!()
something.data == %{"over" => 9000}
Note that if you assign the json_field
a value of 2
, keys will be atomified.
It's possible to NOT encoding individual fields using json_field = -1
message Something {
string id = 1;
bytes secret = 2 [(json_field) = -1];
}