Protox is an Elixir library to work with Google's Protocol Buffers (version 2 and 3).
Protox uses Google's protoc
(>= 3.0) to parse .proto
files. It must be available in $PATH
.
You can get it here.
From files:
defmodule Foo do
@external_resource "./defs/foo.proto"
@external_resource "./defs/bar.proto"
@external_resource "./defs/baz/fiz.proto"
use Protox, files: [
"./defs/foo.proto",
"./defs/bar.proto",
"./defs/baz/fiz.proto",
]
end
From a textual description:
defmodule Bar do
use Protox, schema: """
syntax = "proto3";
package fiz;
message Baz {
}
message Foo {
int32 a = 1;
map<int32, Baz> b = 2;
}
"""
end
The previous example will generate two modules: Fiz.Baz
and Fiz.Foo
.
It's possible to prepend a namespace to all generated modules:
defmodule Bar do
use Protox, schema: """
syntax = "proto3";
enum Enum {
FOO = 0;
BAR = 1;
}
""",
namespace: Namespace
end
In this case, the module Namespace.Enum
will be generated.
Here's how to create a new message:
iex> %Fiz.Foo{a: 3, b: %{1 => %Fiz.Baz{}}} |> Protox.Encode.encode()
[[[], "\b", <<3>>], <<18>>, <<4>>, "\b", <<1>>, <<18>>, <<0>>]
Note that Protox.Encode.encode/1
returns an iolist, not a binary. Such iolists can be used
directly with files or sockets write operations.
However, you can use :binary.list_to_bin/1
to get a binary:
iex> %Fiz.Foo{a: 3, b: %{1 => %Fiz.Baz{}}} |> Protox.Encode.encode() |> :binary.list_to_bin()
<<8, 3, 18, 4, 8, 1, 18, 0>>
Finally, here's how to decode:
iex> <<8, 3, 18, 4, 8, 1, 18, 0>> |> Fiz.Foo.decode()
{:ok, %Fiz.Foo{a: 3, b: %{1 => %Fiz.Baz{}}}}
If any unknown fields are encountered when decoding, they are kept in the decoded message.
It's possible to access them with the function get_unknown_fields/1
defined with the message.
iex> msg = <<8, 42, 42, 4, 121, 97, 121, 101, 136, 241, 4, 83>> |> Msg.decode!()
%Msg{a: 42, b: "", z: -42, __unknown_fields__: [{5, 2, <<121, 97, 121, 101>>}]}
iex> msg |> Msg.get_unknown_fields()
[{5, 2, <<121, 97, 121, 101>>}]
You should always use get_unknown_fields/1
as the name of the struct field
(e.g. __unknown_fields__
) is generated at compile-time to avoid collision with the actual
fields of the protobuf message.
It returns a list of tuples {tag, wire_type, bytes}
.
- protobuf 3 JSON mapping
- groups
- rpc
Furthermore, all options other than packed
and default
are ignored.
-
Required fields (protobuf 2): an error is raised when decoding a message with a missing required field.
-
When decoding enum aliases, the last encountered constant will be used. For instance, in the following example,
:BAR
will always be used if the value1
is read on the wire.enum E { option allow_alias = true; FOO = 0; BAZ = 1; BAR = 1; }
-
Unset optionals
- For protobuf 2, unset optional fields are mapped to
nil
- For protobuf 3, unset optional fields are mapped to their default values, as mandated by the protobuf spec
- For protobuf 2, unset optional fields are mapped to
Protobuf | Elixir |
---|---|
int32 | integer() |
int64 | integer() |
uint32 | integer() |
uint64 | integer() |
sint32 | integer() |
sint64 | integer() |
fixed32 | integer() |
fixed64 | integer() |
sfixed32 | integer() |
sfixed64 | integer() |
float | float() |
double | float() |
bool | boolean() |
string | String.t |
bytes | binary() |
map | %{} |
oneof | {:field, value} |
enum | atom() |
message | struct() |
TODO. Do some benchmarks.
This library has been tested using the conformance checker provided by Google. Note that only the protobuf part is tested: as protox doesn't support JSON output, the corresponding tests are skipped.
Here's how to launch the conformance test:
-
Get conformance-test-runner (https://github.com/google/protobuf/tree/master/conformance)
-
mix protox.conformance --runner=/path/to/conformance-test-runner
A report will be generated in a file namedconformance_report.txt
. If everything's fine, something like the following should be displayed:CONFORMANCE TEST BEGIN ==================================== CONFORMANCE SUITE PASSED: 149 successes, 384 skipped, 0 expected failures, 0 unexpected failures.
Both gpb and exprotobuf were very useful in understanding how to implement Protocol Buffers.