Permalink
Find file
Fetching contributors…
Cannot retrieve contributors at this time
501 lines (459 sloc) 15.2 KB
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
"http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd" >
<article>
<title>The C Code Generator</title>
<section>
<title>Design</title>
<para>The overall goal is to keep the code-generator as simple
as possible. Hopefully performance isn't sacrificed to that end!</para>
<para>Anyways, we generate very little code: we mostly generate
structure definitions (for example enums and structures
for messages) and some metadata which is basically
reflection-type data.</para>
<para>The serializing and deserializing is implemented in a library,
called libprotobuf-c rather than generated code.</para>
</section>
<section>
<title>The Generated Code</title>
<para>
For each enum, we generate a C enum.
For each message, we generate a C structure
which can be cast to a <type>ProtobufCMessage</type>.
</para>
<para>
For each enum and message, we generate a descriptor
object that allows us to implement a kind of reflection
on the structures.
</para>
<section><title>Naming Conventions</title>
<para>First, some naming conventions:
<itemizedlist>
<listitem><para>
The name of the type for enums and messages and services
is camel case (meaning WordsAreCrammedTogether)
except that double-underscores are used to delimit
scopes. For example:
<programlisting><![CDATA[
package foo.bar;
message BazBah {
int32 val;
}
]]></programlisting>
would generate a C type <type>Foo__Bar__BazBah</type>.</para>
</listitem><listitem>
<para>Functions and globals are all lowercase, with camel-case
words separated by single underscores; namespaces are separated with
double-underscores.
For example:
<programlisting><![CDATA[
Foo__Bar__BazBah *foo__bar__baz_bah__unpack
(ProtobufCAllocator *allocator,
size_t length,
const unsigned char *data);
]]></programlisting>
</para>
</listitem><listitem>
<para>Enums values are all uppercase.</para>
</listitem>
<listitem><para>
Stuff we dd to your symbol names will also be
separated by a double-underscore. For example,
the unpack method above.</para></listitem>
</itemizedlist>
</para>
</section>
<section><title>Generated Descriptors</title>
<para>
We also generate descriptor objects for messages
and enums. These are declared in the .h files:
<programlisting><![CDATA[
extern const ProtobufCMessageDescriptor
foo__bar__baz_bah__descriptor;
]]></programlisting>
</para>
</section>
<section><title>Message Methods</title>
<para>
The message structures all begin with <type>ProtobufCMessage</type>,
so they may be cast to that type.
</para>
<para>
We generate some functions for each message:
<itemizedlist>
<listitem>
<para><function>unpack()</function>. Unpack data for a particular
message-format:
<programlisting><![CDATA[
Foo__Bar__BazBah *
foo__bar__baz_bah__unpack (ProtobufCAllocator *allocator,
size_t length,
const unsigned char *data);
]]></programlisting>
Note that <parameter>allocator</parameter> may be NULL.
</para>
</listitem>
<listitem>
<para><function>free_unpacked()</function>. Free a message
that you obtained with the unpack method:
<programlisting><![CDATA[
void
foo__bar__baz_bah__free_unpacked (Foo__Bar__BazBah *baz_bah,
ProtobufCAllocator *allocator);
]]></programlisting>
</para>
</listitem>
<listitem>
<para><function>get_packed_size()</function>. Find how long
the serialized representation of the data will be:
message-format:
<programlisting><![CDATA[
size_t
foo__bar__baz_bah__get_packed_size
(const Foo__Bar__BazBah *message);
]]></programlisting>
</para>
</listitem>
<listitem>
<para><function>pack()</function>. Pack message
into buffer; assumes that buffer is long enough (use get_packed_size first!).
<programlisting><![CDATA[
size_t
foo__bar__baz_bah__pack
(const Foo__Bar__BazBah *message,
unsigned char *packed_data_out);
]]></programlisting>
</para>
</listitem>
<listitem>
<para><function>pack_to_buffer()</function>. Pack message
into virtualize buffer.
<programlisting><![CDATA[
size_t
foo__bar__baz_bah__pack_to_buffer
(const Foo__Bar__BazBah *message,
ProtobufCBuffer *buffer);
]]></programlisting>
</para>
</listitem>
</itemizedlist>
</para>
</section>
<section><title>Services</title>
<para>
Services are collections of methods each having an input and output type.
Unlike messages where we generate a structure that corresponds
to the actual message object, for services we generate
a function that creates a <type>ProtobufCService</type>
from a collection of user-defined methods.
</para>
<para>
We also define simple functions that invoke each method of a service.
These functions work if the service is created by
the <function>create_service</function> generated function
or if the service is instantiated by an RPC system.
</para>
<para>
Suppose we have a .proto file:
<programlisting><![CDATA[
message A {
required uint32 val;
}
message B {
required string foo;
}
service Convert {
rpc Itoa (A) returns (B);
rpc Atoi (B) returns (A);
}
]]></programlisting>
We will get generated code:
<programlisting><![CDATA[
struct _Convert_Service {
ProtobufCService base;
void (*itoa) (Convert_Service *service,
const A *input,
B__Closure closure,
void *closure_data);
void (*atoi) (Convert_Service *service,
const B *input,
A__Closure closure,
void *closure_data);
void (*destroy) (Convert_Service *service);
};
]]></programlisting>
<programlisting><![CDATA[
/* structure derived from Convert_Service. */
typedef struct {
Convert_Service base; /* must be first member */
unsigned radix;
} Convert_WithRadix;
/* convert int to string (not really implemented) */
static void radix_itoa (Convert_Service *service,
const A *input,
B__Closure closure,
void *closure_data)
{
char buf[256];
Convert_WithRadix *wr = (Convert_WithRadix *) service;
B rv;
print_int_with_radix (input->val, wr->radix, buf);
rv.descriptor = &b__descriptor;
rv.str = buf;
closure (&rv, closure_data);
}
/* convert string to int: use strtoul */
static void radix_atoi (Convert_Service *service,
const B *input,
A__Closure closure,
void *closure_data)
{
Convert_WithRadix *wr = (Convert_WithRadix *) service;
A rv;
rv.val = strtoul (input->val, NULL, wr->radix);
rv.descriptor = &a__descriptor;
closure (&rv, closure_data);
}
/* create a new convert service by radix */
ProtobufCService *
create_convert_service_from_radix (unsigned radix)
{
Convert_WithRadix *wr = malloc (sizeof (Convert_WithRadix));
convert__init (wr, (Convert__ServiceDestroy) free);
wr->base.itoa = radix_itoa;
wr->base.atoi = radix_atoi;
wr->radix = radix;
return (ProtobufCService *) wr;
}
]]></programlisting>
Just like with messages, you may cast
from <type>Convert_Service</type> to <type>ProtobufCService</type>,
at least as long as you have run the __init function.
</para>
<para>
Conversely, we generate functions to help you invoke service
methods on generic <type>ProtobufCService</type> objects.
These go through the <function>invoke()</function> of service
and they work on both services created with create_service
as well as factory-provided services like those provided by RPC systems.
For example:
<programlisting><![CDATA[
void convert__itoa (ProtobufCService *service,
const B *input,
A__Closure closure,
void *closure_data);
]]></programlisting>
</para>
</section>
</section>
<section>
<title>The protobuf-c Library</title>
<para>This library is used by the generated code;
it includes common structures and enums,
as well as functions that most users of the generated code
will want.</para>
<para>
There are three main components:
<orderedlist>
<listitem><para>the Descriptor structures</para></listitem>
<listitem><para>helper structures and objects</para></listitem>
<listitem><para>packing and unpacking code</para></listitem>
</orderedlist>
</para>
</section>
<section>
<title>protobuf-c: the Descriptor structures</title>
<para>For example, enums are described in terms of structures:
<programlisting><![CDATA[
struct _ProtobufCEnumValue
{
const char *name;
const char *c_name;
int value;
};
struct _ProtobufCEnumDescriptor
{
const char *name;
const char *short_name;
const char *package_name;
/* sorted by value */
unsigned n_values;
const ProtobufCEnumValue *values;
/* sorted by name */
unsigned n_value_names;
const ProtobufCEnumValue *values_by_name;
};
]]></programlisting></para>
<para>Likewise, messages are described by:
<programlisting><![CDATA[
struct _ProtobufCFieldDescriptor
{
const char *name;
int id;
ProtobufCFieldLabel label;
ProtobufCFieldType type;
unsigned quantifier_offset;
unsigned offset;
void *descriptor; /* for MESSAGE and ENUM types */
};
struct _ProtobufCMessageDescriptor
{
const char *name;
const char *short_name;
const char *package_name;
/* sorted by field-id */
unsigned n_fields;
const ProtobufCFieldDescriptor *fields;
};
]]></programlisting></para>
<para>
And finally services are described by:
<programlisting><![CDATA[
struct _ProtobufCMethodDescriptor
{
const char *name;
const ProtobufCMessageDescriptor *input;
const ProtobufCMessageDescriptor *output;
};
struct _ProtobufCServiceDescriptor
{
const char *name;
unsigned n_methods;
ProtobufCMethodDescriptor *methods; // sorted by name
};
]]></programlisting></para>
</section>
<section>
<title>protobuf-c: helper structures and typedefs</title>
<para>We defined typedefs for a few types
which are used in .proto files but do not
have obvious standard C equivalents:
<itemizedlist>
<listitem><para>a boolean type (<type>protobuf_c_boolean</type>)</para></listitem>
<listitem><para>a binary-data (bytes) type (<type>ProtobufCBinaryData</type>)</para></listitem>
<listitem><para>the various int types (<type>int32_t</type>, <type>uint32_t</type>, <type>int64_t</type>, <type>uint64_t</type>)
are obtained by including <filename>inttypes.h</filename></para></listitem>
</itemizedlist>
</para>
<para>We also define a simple allocator object, ProtobufCAllocator
that let's you control how allocations are done.
This is predominately used for parsing.</para>
<para>There is a virtual buffer facility that
only has to implement a method to append binary-data
to the buffer. This can be used to serialize messages
to different targets (instead of a flat slab of data).</para>
<para>We define a base-type for all messages,
for code that handles messages generically.
All it has is the descriptor object.</para>
<section id="buffers">
<title>Buffers</title>
<para>One important helper type is the <type>ProtobufCBuffer</type>
which allows you to abstract the target of serialization. The only
thing that a buffer has is an <function>append</function> method:
<programlisting><![CDATA[
struct _ProtobufCBuffer
{
void (*append)(ProtobufCBuffer *buffer,
size_t len,
const unsigned char *data);
}
]]></programlisting>
ProtobufCBuffer subclasses are often defined on the stack.
</para>
<para>
For example, to write to a <type>FILE</type> you could make:
<programlisting><![CDATA[
typedef struct
{
ProtobufCBuffer base;
FILE *fp;
} BufferAppendToFile
static void my_buffer_file_append (ProtobufCBuffer *buffer,
unsigned len,
const unsigned char *data)
{
BufferAppendToFile *file_buf = (BufferAppendToFile *) buffer;
fwrite (data, len, 1, file_buf->fp); // XXX: no error handling!
}
]]></programlisting>
</para>
<para>
To use this new type of Buffer, you would do something like:
<programlisting><![CDATA[
...
BufferAppendToFile tmp;
tmp.base.append = my_buffer_file_append;
tmp.fp = fp;
protobuf_c_message_pack_to_buffer (&message, &tmp);
...
]]></programlisting>
</para>
<para>
A commonly builtin subtype is the BufferSimple
which is declared on the stack and uses a scratch buffer provided by the user
for its initial allocation. It does exponential resizing.
To create a BufferSimple, use code like:
<programlisting><![CDATA[
unsigned char pad[128];
ProtobufCBufferSimple buf = PROTOBUF_C_BUFFER_SIMPLE_INIT (pad);
ProtobufCBuffer *buffer = (ProtobufCBuffer *) &simple;
protobuf_c_buffer_append (buffer, 6, (unsigned char *) "hi mom");
]]></programlisting>
You can access the data as buf.len and buf.data. For example,
<programlisting><![CDATA[
assert (buf.len == 6);
assert (memcmp (buf.data, "hi mom", 6) == 0);
]]></programlisting>
To finish up, use:
<programlisting><![CDATA[
PROTOBUF_C_BUFFER_SIMPLE_CLEAR (&buf);
]]></programlisting>
</para>
</section>
</section>
<section>
<title>protobuf-c: packing and unpacking messages</title>
<para>
To pack messages one first computes their packed size,
then provide a buffer to pack into.
<programlisting><![CDATA[
size_t protobuf_c_message_get_packed_size
(ProtobufCMessage *message);
void protobuf_c_message_pack (ProtobufCMessage *message,
unsigned char *out);
]]></programlisting>
</para>
<para>
Or you can use the "streaming" approach:
<programlisting><![CDATA[
void protobuf_c_message_pack_to_buffer
(ProtobufCMessage *message,
ProtobufCBuffer *buffer);
]]></programlisting>
where <type>ProtobufCBuffer</type> is a base object with an append metod.
See <xref linkend="buffers" />.
</para>
<para>
To unpack messages, you should simple call
<programlisting><![CDATA[
ProtobufCMessage *
protobuf_c_message_unpack (const ProtobufCMessageDescriptor *,
ProtobufCAllocator *allocator,
size_t len,
const unsigned char *data);
]]></programlisting>
If you pass NULL for <parameter>allocator</parameter>, then
the default allocator will be used.
</para>
<para>
You can cast the result to the type that matches
the descriptor.
</para>
<para>
The result of unpacking should be freed with protobuf_c_message_free().
</para>
</section>
<section>
<title>Author</title>
<para>Dave Benson.</para>
</section>
</article>