protostar

(Slightly unhinged) Protocol buffers in pure Ruby. protostar compiles .proto files to Ruby, encodes and decodes messages, reads them without decoding, writes them as JSON and as text, and can stand in for Google's google-protobuf gem so that grpc and friends run without it.

No protoc, no C extension, no runtime dependencies. No native memory handles, GC problems, no uninspectable blobs, no issues with fork(), Fibers or Ractors or Threads. Full introspection and control. Ruby 3.2 or newer.

Why bother? Of course pure Ruby will be slower, you say... and that's correct.

But: a Ruby app that speaks protobuf usually drags in a native gem that has to compile (or come prebuilt for your exact platform), plus a protoc binary of a matching version that someone has to install on every machine that regenerates code. protoc versions change. The DSL it outputs changes. The formatting changes. The proto messages are handles to native pointers - they can't be marshaled, they can't be cached, they can't be recreated without access to the native gem and the kitchen sink. They are, by any stretch of the imagination, as anti-Ruby spirited as things can be.

This is why. We sacrifice some speed (primarily in encoding and allocations) for the luxury of having a readable, predictable, Ruby-spirited protobuf universum.

One that doesn't scream "We despise you and your ridiculous slow toy language, but we made it work because we were told to" every time you touch it in your irb.

protostar is one gem. It passes protobuf's own conformance suite, reproduces protoc's descriptors byte for byte on protoc's own test files, and runs Google's Ruby tests through its google-protobuf shim.

This is young code. It passes protobuf's conformance suite and Google's own Ruby tests, but it has not yet carried anyone's production traffic, the API will move before 1.0, and the google-protobuf shim has a documented list of places where it differs from the C gem. Try it, kick it, and tell me what breaks.

Ideas are as welcome as bug reports. The llm/ directory is where this project thinks out loud: llm/todos/ holds feature briefs that are not scheduled yet (lazy readers, Marshal round trips for Rails caching, a native codec, arenas, and more), llm/instructions/ the briefs that became workstreams, and llm/outputs/ what each one reported back. If you have a feature in mind, open a PR that adds a brief to llm/todos/ in the same shape as the ones there: what you want, why, what it would take. That is the fastest way to get it built.

protostar is in prerelease, so name the version (or gem install protostar --pre):

gem "protostar", "0.1.0.beta1"

Take a schema:

// proto/acme/person.proto
syntax = "proto3";

package acme;

import "google/protobuf/timestamp.proto";

message Person {
  string name = 1;
  int32 id = 2;
  repeated string emails = 3;
  map<string, string> labels = 4;
  Address address = 5;
  google.protobuf.Timestamp born_at = 6;
  oneof contact {
    string phone = 7;
    string pager = 8;
  }
}

message Address {
  string street = 1;
  string city = 2;
}

service Directory {
  rpc Find(Person) returns (Person);
}

and compile it:

$ bundle exec protostar proto/acme/person.proto > person_pb.rb

With one file and no output flags the Ruby goes to stdout, and errors go to stderr. The generated file is meant to be read, and committed:

# frozen_string_literal: true
# typed: ignore

# Generated by protostar from person.proto. Do not edit.

require "protostar"
require "protostar/google/protobuf/timestamp_pb"

Protostar::DSL.file(
  "person.proto",
  package: "acme",
  syntax: :proto3,
  dependencies: ["google/protobuf/timestamp.proto"]
) do |f|
  f.message "Person" do |m|
    m.field :name, 1, :string
    m.field :id, 2, :int32
    m.field :emails, 3, :string, label: :repeated
    m.map :labels, 4, :string, :string
    m.field :address, 5, :message, ".acme.Address"
    m.field :born_at, 6, :message, ".google.protobuf.Timestamp"
    m.oneof :contact do |o|
      o.field :phone, 7, :string
      o.field :pager, 8, :string
    end
  end
  f.message "Address" do |m|
    m.field :street, 1, :string
    m.field :city, 2, :string
  end
  f.service "Directory" do |s|
    s.rpc :Find, ".acme.Person", ".acme.Person"
  end
end

module Acme
  Person = Protostar::DSL.message("acme.Person")
  Address = Protostar::DSL.message("acme.Address")
  Directory = Protostar::DSL.service("acme.Directory")
end

Without -I, the directory the file is in is where imports are looked up, and the file is named person.proto. Once protos import each other by paths like acme/person.proto, compile them like protoc, with protoc's flags:

$ bundle exec protostar compile -I proto --ruby_out=lib proto/acme/person.proto

That writes lib/acme/person_pb.rb, for the file named acme/person.proto under the include path proto, and next to it person_pb.rbs (RBS, for Steep and editors) and person_pb.rbi (for Sorbet, see Sorbet). --no-rbs and --no-rbi leave those out. --ruby_out=DIR also takes several files at once.

The well-known types (google/protobuf/timestamp.proto and the rest) are bundled, so importing them needs no -I. Other outputs:

• --grpc_out=DIR writes gRPC service stubs (*_services_pb.rb)
• --rbs_out=DIR and --rbi_out=DIR write the RBS and the RBI there instead of next to the Ruby; next to --google_out, or alone, they describe google-protobuf's classes
• --google_out=DIR writes exactly what protoc --ruby_out writes, for the google-protobuf API
• --descriptor_set_out=FILE writes a FileDescriptorSet, with --include_imports and --include_source_info

protostar --help lists them all. Errors look like protoc's, file and line first.

Let Rake recompile a proto when it changes:

rule "_pb.rb" => ->(target) { target.sub(/_pb\.rb\z/, ".proto") } do |t|
  sh "protostar", "compile", "-I", ".", "--ruby_out=.", t.source
end

desc "Compile every .proto here into a _pb.rb next to it"
task protos: FileList["*.proto"].pathmap("%X_pb.rb")

If a build already runs protoc, protostar plugs into it as protoc-gen-protostar:

$ bundle exec protoc -I proto --protostar_out=lib proto/acme/person.proto
$ bundle exec protoc -I proto --protostar_out=ruby,grpc:lib proto/acme/person.proto

The parameter picks the outputs: ruby (the default, with its .rbs and .rbi), google, rbs, rbi, grpc; no-rbs and no-rbi drop what ruby brings along.

Compile at runtime if you would rather not commit generated code:

files = Protostar::Compiler.new(include_paths: ["proto"]).compile(["acme/person.proto"])
registry = Protostar::Types::Registry.new
files.each { |f| registry.define(f) } # imports first, as compile returns them
person_class = registry.find("acme.Person") # => #<Class acme.Person>
person_class.new(name: "Ada")               # => #<acme.Person name: "Ada">

Classes defined this way are anonymous; they live in the registry, not in constants.

protostar lint compiles files (or every *.proto under a directory) and explains what is wrong the way rustc does, then checks what compiled against best-practice rules, under buf lint's names:

$ bundle exec protostar lint proto/
warning[W0214]: `Person` is the response of `Find` and already the request of `Find`
  --> proto/acme/person.proto:27:29
   |
27 |   rpc Find(Person) returns (Person);
   |            ------           ^^^^^^ used again
   |            |
   |            first used here
   |
   = help: give `Find` a `FindResponse` of its own
   = note: rule RPC_REQUEST_RESPONSE_UNIQUE (default set)

Compiler errors get the same treatment, with a code to look up (here with id = 1):

error[L0301]: field number 1 is already used in `acme.Person`
  --> proto/acme/person.proto:10:14
   |
 9 |   string name = 1;
   |                 - first used by `name`
10 |   int32 id = 1;
   |              ^ reused here
   |
   = help: the next available field number is 2

protostar lint --explain L0301 (or a rule, --explain FIELD_NOT_REQUIRED) says why and how to fix it; --list-rules lists the rules. They come in three sets, each including the one before:

• minimal (11 rules): the longevity ones, led by "required is forever" (FIELD_NOT_REQUIRED), plus groups, enum zero values, unreserved gaps in field numbers, imports and a missing syntax line
• default (28, what runs unless told otherwise): naming and package layout
• strict (37): comments on everything, version suffixes, field numbers 1 to 15, editions

Pick with --rules default,documentation (sets, categories or rule ids) and --except FIELD_NOT_REQUIRED, or in a .protostar.yml next to the inputs or in the current directory:

lint:
  rules: [default]
  except: [PACKAGE_VERSION_SUFFIX]
  ignore: ["vendor/**"]
  allow_comment_directives: true

A comment switches a rule off for the next line, or for a stretch of the file:

// protostar-lint: disable-next-line FIELD_NOT_REQUIRED
required string id = 1;

// protostar-lint: disable COMMENT_FIELD ... // protostar-lint: enable COMMENT_FIELD does the same for everything between. --fix applies the fixes that are mechanical (renaming a field, oneof or enum value nothing else in the file mentions, adding the missing syntax line) and prints the diff; --diff prints it without writing. --format json prints one object per finding, --format github workflow annotations for pull requests. The exit status is 0 when clean, 1 with warnings (unless --warnings-ok) and 2 with errors.

protostar compile --error_format=rich prints compile errors this way too; without it, compile prints protoc's one-liners, as before.

require "acme/person_pb"

person = Acme::Person.new(name: "Ada", id: 7, emails: ["ada@example.com"])
person.address = Acme::Address.new(city: "London")
person.labels["team"] = "engines"
person.born_at = Time.utc(1815, 12, 10)
person.phone = "+44 20 7946 0000"

person.name          # => "Ada"
person.which_contact # => :phone
person
# => #<acme.Person name: "Ada", id: 7, emails: ["ada@example.com"], labels: {"team" => "engines"}, address: #<acme.Address city: "London">, born_at: #<google.protobuf.Timestamp seconds: -4861728000>, phone: "+44 20 7946 0000">

Fields are keyword arguments or a Hash, repeated fields are Arrays, maps are Hashes, enums are Symbols. Values are checked when you assign them:

person.id = "seven"
# raises Protostar::TypeError: acme.Person.id expects an Integer in int32 range, got "seven"

Protostar::TypeError is a Ruby TypeError. Fields with presence get has_name? and clear_name; oneofs get which_contact; every field is also person[:name].

bytes = person.encode         # also Acme::Person.encode(person)
bytes.bytesize                # => 82
Acme::Person.decode(bytes) == person # => true

Fields the schema does not know (sent by a newer version of it) are kept and written back on encode; unknown_fields lists them. A broken input raises Protostar::DecodeError with the offset:

Acme::Person.decode("\x0a\x05ab")
# raises Protostar::DecodeError: length 5 runs past the end of the input at byte 2

For a stream of messages, encode_delimited(io) and Klass.decode_delimited(io), which returns nil at the end. msg.encode_to(io) writes to an IO without building the String.

The proto3 JSON mapping, as google-protobuf writes it:

person.to_json
# => "{\"name\":\"Ada\",\"id\":7,\"emails\":[\"ada@example.com\"],\"labels\":{\"team\":\"engines\"},\"address\":{\"city\":\"London\"},\"bornAt\":\"1815-12-10T00:00:00Z\",\"phone\":\"+44 20 7946 0000\"}"
Acme::Person.decode_json(person.to_json) == person # => true

to_json and Klass.encode_json take preserve_proto_field_names:, emit_defaults: and format_enums_as_integers:; decode_json takes ignore_unknown_fields:. JSON.pretty_generate(message) works:

puts JSON.pretty_generate(Acme::Address.new(city: "Delft", street: "Oude Delft 1"))

{
  "street": "Oude Delft 1",
  "city": "Delft"
}

protoc's text format, for logs and fixtures:

puts person.to_text

name: "Ada"
id: 7
emails: "ada@example.com"
labels {
  key: "team"
  value: "engines"
}
address {
  city: "London"
}
born_at {
  seconds: -4861728000
}
phone: "+44 20 7946 0000"

Protostar::Text.parse(Acme::Person, 'name: "Ada" address { city: "London" }')
# => #<acme.Person name: "Ada", address: #<acme.Address city: "London">>

A view reads fields straight from the bytes. It looks at the top level once, and decodes a field only when you read it; nested messages are skipped until you open them. On a 1 MiB message, reading two fields through a view allocates 8 objects where decode allocates 2,522.

view = Acme::Person.view(bytes)   # a String, or an IO::Buffer (a mapped file, say)
view.address.city                 # => "London"
view.emails.to_a                  # => ["ada@example.com"]
view.to_message                   # the decoded message, when you want all of it

Views are read-only, equal to the message they hold (view == message), and raise the same errors as decode.

Classes and descriptors are frozen and shareable, so messages can be decoded, encoded and read in any Ractor. Decode with frozen: true to get a message you can pass between them:

Ractor.shareable?(Acme::Person) # => true
frozen = Acme::Person.decode(bytes, frozen: true)
Ractor.shareable?(frozen)       # => true
Ractor.new(frozen) { |m| m.name.upcase }.take # => "ADA" (Ractor#value on Ruby 4.0)

Load your _pb.rb files in the main Ractor before starting others. require "protostar" loads the binary codec and autoloads the rest on first use, and on Ruby 3.2 and 3.3 nothing can be loaded from another Ractor, not even by autoload, so any part of protostar a Ractor will use has to be touched in the main Ractor first. Encoding, decoding and views need nothing beyond the classes. A first to_json or decode_json loads Protostar::JSON, to_text loads Protostar::Text, build loads Protostar::Builder, a to_proto or to_h on a descriptor loads descriptor.proto's classes, and Protostar::Timestamp and the other well-known types load theirs. Ruby 3.4 lifts this.

Each class answers descriptor (a Protostar::Schema::Message, with every field, option and resolved feature of the schema) and reflection, for code that handles messages it does not know in advance:

Acme::Person.reflection.fields.map(&:name)
# => ["name", "id", "emails", "labels", "address", "born_at", "phone", "pager"]
Acme::Person.reflection.get(person, :name) # => "Ada"
Acme::Person.descriptor.field("labels").map? # => true

docs/schema-api.md describes the schema objects, options and reflection.

When a response is large, building message objects only to encode them is waste. The Builder writes the wire bytes as the block runs; nested messages and packed fields are the only things it holds back.

bytes = Protostar::Builder.build(Acme::Person) do |pb|
  pb.name "Team"
  pb.labels "kind" => "group"
  pb.address do
    pb.city "London"
  end
end
Acme::Person.decode(bytes)
# => #<acme.Person name: "Team", labels: {"kind" => "group"}, address: #<acme.Address city: "London">>

Inside a nested block the same pb writes the nested message. A repeated message field takes a collection and a block per element (pb.friends(users) { |u| pb.name u.name }), an Array of Hashes, or messages. Builder.write(io, Klass) streams to an IO, Klass.build { |pb| ... } returns a message, and pb.cache!(key) { } caches a fragment's bytes in a Rails-style cache store. A singular field written twice raises unless you pass overwrites: true; then the later value wins when read, which relies on the reader applying protobuf's merge rules (every conforming one does). Building 1,000 nested elements this way takes a third (YJIT) to a half of the time and a fifth of the objects of new and encode (see the benchmarks).

Builder.write takes any sink: an IO or socket, a Rack or ActionController::Live stream, an Array, a Proc. Top-level fields go out through a 64 KiB buffer (buffer: 0 writes each one as it completes), and only one nested message per level is held, until it closes. For a nested message too large to hold, chunk_nested_at: 64 * 1024 writes it in pieces that every reader merges back into one; that relies on the default overwrites: false, the promise that no field is written twice. Builder.each(io, Klass, rows) { |pb, row| ... } writes a length-prefixed sequence for decode_delimited, and Builder.each_chunk(Klass) { |pb| ... } is an Enumerator of chunks for a Rack body. docs/concepts/streaming.md measures what streams and what has to be held.

The generated files are plain Ruby. You can write one yourself:

Protostar::DSL.file("shop/item.proto", package: "shop", syntax: :proto3) do |f|
  f.message "Item" do |m|
    m.field :sku, 1, :string
    m.field :price_cents, 2, :int64, proto3_optional: true
    m.field :tags, 3, :string, label: :repeated
    m.map :stock, 4, :string, :int32
  end
end

module Shop
  Item = Protostar::DSL.message("shop.Item")
end

item = Shop::Item.new(sku: "lamp", stock: {"ams" => 3})
item.has_price_cents?          # => false
Shop::Item.decode(item.encode) # => #<shop.Item sku: "lamp", stock: {"ams" => 3}>

Options are keywords (deprecated: true, packed: false, features: {...}), custom options go under extensions: {"acme.my_option" => value}.

Timestamp, Duration, Any, Struct, Value, ListValue, Empty, FieldMask and the wrappers have short names under Protostar, loaded on first use, and the helpers google-protobuf has:

Protostar::Timestamp.from_time(Time.utc(2024, 3, 1))
# => #<google.protobuf.Timestamp seconds: 1709251200>
Protostar::Duration.new(seconds: 90).to_f # => 90.0
any = Protostar::Any.pack(item)
any.type_url                              # => "type.googleapis.com/shop.Item"
any.unpack(Shop::Item) == item            # => true
Protostar::Struct.from_hash({"a" => 1, "b" => [true, nil]}).to_h
# => {"a" => 1.0, "b" => [true, nil]}

A Timestamp field takes a Time (as above, person.born_at = Time.utc(...)), a Duration field takes a number of seconds.

--grpc_out writes the stubs grpc-tools would, and they work with protostar's classes:

$ bundle exec protostar compile -I proto --ruby_out=lib --grpc_out=lib proto/acme/person.proto

require "acme/person_services_pb"

class Directory < Acme::Directory::Service
  def find(person, _call)
    Acme::Person.new(name: person.name, id: 42)
  end
end

server = GRPC::RpcServer.new
port = server.add_http2_port("127.0.0.1:0", :this_port_is_insecure)
server.handle(Directory)
Thread.new { server.run }
server.wait_till_running

stub = Acme::Directory::Stub.new("127.0.0.1:#{port}", :this_channel_is_insecure)
stub.find(Acme::Person.new(name: "Ada")) # => #<acme.Person name: "Ada", id: 42>
defined?(Google::Protobuf)               # => nil

The grpc gem depends on google-protobuf, so Bundler installs it, but with protostar's classes it is never loaded. Install the shim as that gem (below) and the C extension is gone from the bundle too.

--ruby_out writes a _pb.rbi next to every _pb.rb: a class per message with a typed reader and writer per field, initialize keywords, has_/clear_, which_ for oneofs, and a module per enum with its constants. The gem ships rbi/protostar.rbi for Protostar::Message and the rest of the runtime (decode returns the class it is called on), and the well-known types' RBI, which tapioca gem protostar picks up.

sig { params(bytes: String).returns(String) }
def email_of(bytes)
  order = Shop::Order.decode(bytes) # Shop::Order
  order.custmer_email               # error: Method `custmer_email` does not exist
end

The generated _pb.rb says # typed: ignore: its constants are assigned from DSL calls, which Sorbet cannot follow, so Sorbet reads the classes from the .rbi instead. Nothing needs configuring. Keep the .rbi where Sorbet looks, next to the Ruby or, with --rbi_out=sorbet/rbi/protos, in your RBI tree.

For protoc's google-protobuf files (--google_out, or protoc itself), --rbi_out=DIR writes RBI in the shape tapioca's protobuf compiler gives them. protoc's _pb.rb cannot carry a sigil, so list those files under --ignore in sorbet/config, as tapioca's own docs ask; Google::Protobuf::RepeatedField and Map get their type members from tapioca dsl.

protostar ships a shim that provides Google's Google::Protobuf API on protostar, so grpc, googleapis-common-protos-types and the _pb.rb files protoc generated run unchanged. It is off until you ask for it. Require it before anything requires google/protobuf:

require "protostar/google_protobuf_shim"
require "grpc"
Google::Protobuf::IMPLEMENTATION # => :PROTOSTAR

From then on every require "google/protobuf" in the process loads the shim, even with Google's gem installed alongside (grpc's dependency makes Bundler install it). If Google's gem was loaded first, the require raises LoadError.

To keep Google's gem out of the bundle entirely, use the shim as a gem named google-protobuf, from git. Every gem that depends on google-protobuf then gets it instead of Google's:

gem "protostar", github: "julik/protostar"
gem "google-protobuf", github: "julik/protostar", glob: "shims/google-protobuf/*.gemspec"

Its README has the other ways to install it, what it covers, and how to check it is the one loaded. docs/migration.md walks through moving an app or a gem from google-protobuf and protoc, either way.

• Extensions as fields. Extension values are kept, byte for byte, as unknown fields, and the text format, JSON and the shim's FieldDescriptor#get/#set read and write them by name. There is no msg.extension(...) on protostar's own classes. This is also the one conformance test protostar fails: UTF-8 checking of a string extension.
• The legacy DSL. _pb.rb files from protoc before 3.21 (add_file, add_message) do not load, natively or through the shim. Regenerate them.
• JSPB, protobuf's JavaScript array format.
• The C++ and Java options (cc_enable_arenas, java_package, ...) are kept in the descriptors and otherwise ignored, as protoc's Ruby generator ignores them.

Pure Ruby is slower than C, and there is no hiding it. On the benchmarks (Ruby 3.4, Apple M1 Pro), decoding a populated 413-byte message takes 42 µs, 30 µs with YJIT, where google-protobuf's C extension takes 3 µs; encoding it, 30 µs and 12 µs against 0.9 µs. A 50-byte message decodes in 2.5 µs and encodes in 3.6 µs with YJIT. Where protostar does well: with YJIT, a view reads two fields of a 1 MiB message in 53 µs, five times faster than the C gem decodes it; the Builder writes large responses without building objects; and nothing has to compile at install time. Defining classes costs about 1.1 ms per message class when a _pb.rb is loaded.

If protobuf encoding is your hot path, measure with your own messages (rake bench shows how) before you switch.

$ bundle install
$ bundle exec rake              # the tests and standard
$ bundle exec rake shim:test    # the shim's suite, with its own bundle

The tests check protostar against protoc's own output, not against what we think protoc does:

• test/corpus/protobuf/ is a slice of the protobuf repository at v36.2 (see PIN), BSD licensed. test/corpus/goldens/ holds, for each of its 107 .proto files, the FileDescriptorSet protoc 36.2 wrote (.binpb, with source info) and the same as JSON. The compiler must reproduce them exactly. Never edit either by hand: rake corpus:fetch and rake corpus:goldens regenerate them (the latter needs protoc, which rake protoc:install downloads into ~/.cache/protostar). The tests themselves never need protoc.
• BUNDLE_WITH=oracle bundle install and PROTOSTAR_ORACLE=1 bundle exec rake test cross-check the codec and JSON against the C-extension gem. Without it those tests skip.
• rake conformance:build builds protobuf's conformance runner (CMake and a C++ compiler); rake conformance:run runs it.
• rake test:warnings fails on any Ruby warning from lib/. Types are written inline in lib/: rake types:generate writes sig/ and rbi/protostar.rbi from them, rake rbs:validate and rake steep check them, and rake rdoc builds the API docs.
• Generated files are committed and checked: rake bootstrap:check, rake examples:check, rake conformance:check. After changing an emitter, run the matching :regenerate.
• test/release/ builds both gems, installs them into an empty GEM_HOME and uses them from a subprocess with no bundle.
• rake test runs the test files in a forked worker per core (test/parallel.rb); PARALLEL=0, or TEST= naming one file, runs them in one process. Tests make temp dirs with tmpdir, ports with free_port, and keep generated classes out of the process-wide registry; test/parallel/isolation_test.rb checks that they do.

docs/design.md is the map of the code; docs/compiler-internals.md follows the compiler stage by stage. examples/ has four runnable playbooks.

MIT (LICENSE.txt). The Google .proto files bundled under lib/protostar/compiler/include/, the code generated from them, and the vendored protobuf sources under test/corpus/ are Google's, under the BSD 3-Clause licence; THIRD_PARTY_NOTICES.md lists them and carries that licence.