An Elixir library for Protocol Buffers.
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
benchmarks Add first benchmarks Jan 8, 2017
config Initial commit Dec 14, 2016
lib Truth can be encoded by other values than 1 Jan 29, 2017
test Test over-encoded varints Jan 29, 2017
.gitignore Add inch_ex dependency Jan 1, 2017
.travis.yml Travis: skip Erlang 18.0 & 18.1 Jan 29, 2017
CHANGELOG 0.13.0 Jan 29, 2017
LICENSE Initial commit Dec 14, 2016
README.md Development now takes place at https://github.com/EasyMile/protox Feb 16, 2017
coveralls.json Prepare for a benchmark escript Dec 31, 2016
mix.exs 0.13.0 Jan 29, 2017
mix.lock Bump deps Jan 24, 2017

README.md

Development now takes place at https://github.com/EasyMile/protox.

Protox

Build Status Coverage Status Hex.pm Version Deps Status Inline docs

Protox is an Elixir library to work with Google's Protocol Buffers (version 2 and 3).

Prerequisites

Protox uses Google's protoc (>= 3.0) to parse .proto files. It must be available in $PATH. You can get it here.

Usage

From files:

defmodule Foo do
  @external_resource "./defs/foo.proto"
  @external_resource "./defs/bar.proto"
  @external_resource "./defs/baz/fiz.proto"

  use Protox, files: [
    "./defs/foo.proto",
    "./defs/bar.proto",
    "./defs/baz/fiz.proto",
  ]
end

From a textual description:

defmodule Bar do
  use Protox, schema: """
  syntax = "proto3";

  package fiz;

  message Baz {
  }

  message Foo {
    int32 a = 1;
    map<int32, Baz> b = 2;
  }
  """
end

The previous example will generate two modules: Fiz.Baz and Fiz.Foo.

It's possible to prepend a namespace to all generated modules:

defmodule Bar do
  use Protox, schema: """
    syntax = "proto3";

    enum Enum {
        FOO = 0;
        BAR = 1;
      }
    """,
    namespace: Namespace
end

In this case, the module Namespace.Enum will be generated.

Here's how to create a new message:

iex> %Fiz.Foo{a: 3, b: %{1 => %Fiz.Baz{}}} |> Protox.Encode.encode()
[[[], "\b", <<3>>], <<18>>, <<4>>, "\b", <<1>>, <<18>>, <<0>>]

Note that Protox.Encode.encode/1 returns an iolist, not a binary. Such iolists can be used directly with files or sockets write operations. However, you can use :binary.list_to_bin/1 to get a binary:

iex> %Fiz.Foo{a: 3, b: %{1 => %Fiz.Baz{}}} |> Protox.Encode.encode() |> :binary.list_to_bin()
<<8, 3, 18, 4, 8, 1, 18, 0>>

Finally, here's how to decode:

iex> <<8, 3, 18, 4, 8, 1, 18, 0>> |> Fiz.Foo.decode()
{:ok, %Fiz.Foo{a: 3, b: %{1 => %Fiz.Baz{}}}}

Unknown fields

If any unknown fields are encountered when decoding, they are kept in the decoded message. It's possible to access them with the function get_unknown_fields/1 defined with the message.

iex> msg = <<8, 42, 42, 4, 121, 97, 121, 101, 136, 241, 4, 83>> |> Msg.decode!()
%Msg{a: 42, b: "", z: -42, __unknown_fields__: [{5, 2, <<121, 97, 121, 101>>}]}

iex> msg |> Msg.get_unknown_fields()
[{5, 2, <<121, 97, 121, 101>>}]

You should always use get_unknown_fields/1 as the name of the struct field (e.g. __unknown_fields__) is generated at compile-time to avoid collision with the actual fields of the protobuf message.

It returns a list of tuples {tag, wire_type, bytes}.

Unsupported features

  • protobuf 3 JSON mapping
  • groups
  • rpc

Furthermore, all options other than packed and default are ignored.

Implementation choices

  • Required fields (protobuf 2): an error is raised when decoding a message with a missing required field.

  • When decoding enum aliases, the last encountered constant will be used. For instance, in the following example, :BAR will always be used if the value 1 is read on the wire.

    enum E {
      option allow_alias = true;
      FOO = 0;
      BAZ = 1;
      BAR = 1;
    }
    
  • Unset optionals

    • For protobuf 2, unset optional fields are mapped to nil
    • For protobuf 3, unset optional fields are mapped to their default values, as mandated by the protobuf spec

Types mapping

Protobuf Elixir
int32 integer()
int64 integer()
uint32 integer()
uint64 integer()
sint32 integer()
sint64 integer()
fixed32 integer()
fixed64 integer()
sfixed32 integer()
sfixed64 integer()
float float()
double float()
bool boolean()
string String.t
bytes binary()
map %{}
oneof  {:field, value}
enum atom()
message struct()

Performance

TODO. Do some benchmarks.

Conformance

This library has been tested using the conformance checker provided by Google. Note that only the protobuf part is tested: as protox doesn't support JSON output, the corresponding tests are skipped.

Here's how to launch the conformance test:

  • Get conformance-test-runner (https://github.com/google/protobuf/tree/master/conformance)

  • mix protox.conformance --runner=/path/to/conformance-test-runner A report will be generated in a file named conformance_report.txt. If everything's fine, something like the following should be displayed:

    CONFORMANCE TEST BEGIN ====================================
    
    CONFORMANCE SUITE PASSED: 149 successes, 384 skipped, 0 expected failures, 0 unexpected failures.
    

Credits

Both gpb and exprotobuf were very useful in understanding how to implement Protocol Buffers.