Skip to content

Commit

Permalink
Introduce protobuf encoder with main types
Browse files Browse the repository at this point in the history
Introducing protobuf encoder that can convert four main protobuf types.
Varint, Len, I32, I64 protobuf types and enum type are supported now.
Encoder can create a new schema and encode data according to it.
As a result a binary string is returned that can be transported
by wire or decoded back by another protobuf decoder.
The future versions will add support for repeated fields, .proto files
parser and decode method for encoded data.

@TarantoolBot document
Title: Protobuf module
Product: Tarantool
Since: 3.1
Root document: -

Protobuf decoder API

Introducing protobuf encoder that can convert four main protobuf types
into wire format. Varint, Len, I32, I64 protobuf types and enum data
type are supported now. To encode data you need to create a schema
according to which data will be encoded.

The two main components of the schema are messages and enums.
To create them .message and .enum functions are used

Each message consists of name of the message and fields.
Each field has a name, a type and an id which are set according to
example:

```lua
protobuf.message('KeyValue', {
    key = {'bytes', 1},
    create_revision = {'int64', 2},
    mod_revision = {'int64', 3},
    version = {'int64', 4},
    value = {'bytes', 5},
    lease = {'int64', 6),
})
```

Each enum type consists of name of type and values.
Values must have a zero value to be set as default as in example:

```lua
protobuf.enum('EventType', {
    ['PUT'] = 0,
    ['DELETE'] = 1,
})
```

To create a schema .protocol function is used. This function
supports forward declared types and nested messages so the tuple
can be set according to example:

```lua
schema = protobuf.protocol({
    protobuf.message(<...>),
    protobuf.message(<...>),
    protobuf.enum(<...>),
    protobuf.enum(<...>),
})
```

Output schema can then be used for encoding entered data by the
method named encode. This method converts input data
according to chosen message definition from
protobuf schema into protobuf wireformat. All fields in message
definition are optional so if some input data is missing
it simply will not be encoded. Input data can be
submitted using luatypes or using cdata (for example
entering int64) according to the example.

```lua
result = schema:encode(‘KeyValue’,
    {
        key = 'protocol',
        version = 2,
        lease = 5,
    }
)
```

Output result will be a binary string encoded according to the
protobuf standard and can be transmitted to another user.
  • Loading branch information
Col-Waltz committed Apr 21, 2024
1 parent 7fd530f commit 1d2a75d
Show file tree
Hide file tree
Showing 11 changed files with 2,037 additions and 0 deletions.
4 changes: 4 additions & 0 deletions changelogs/unreleased/protobuf_encoder_module.md
@@ -0,0 +1,4 @@
## feature/lua

* Introduced Lua implementation of protobuf encoder.
Four main protobuf types are supported.
2 changes: 2 additions & 0 deletions src/CMakeLists.txt
Expand Up @@ -71,6 +71,8 @@ lua_source(lua_sources lua/print.lua print_lua)
lua_source(lua_sources lua/pairs.lua pairs_lua)
lua_source(lua_sources lua/compat.lua compat_lua)
lua_source(lua_sources lua/varbinary.lua varbinary_lua)
lua_source(lua_sources lua/protobuf_wireformat.lua protobuf_wireformat_lua)
lua_source(lua_sources lua/protobuf.lua protobuf_lua)
if (ENABLE_COMPRESS_MODULE)
lua_source(lua_sources ${COMPRESS_MODULE_LUA_SOURCE} compress_lua)
endif()
Expand Down
4 changes: 4 additions & 0 deletions src/lua/init.c
Expand Up @@ -136,6 +136,8 @@ extern char minifio_lua[],
utils_lua[],
argparse_lua[],
iconv_lua[],
protobuf_wireformat_lua[],
protobuf_lua[],
/* jit.* library */
jit_vmdef_lua[],
jit_bc_lua[],
Expand Down Expand Up @@ -211,6 +213,8 @@ static const char *lua_modules[] = {
"http.client", httpc_lua,
"iconv", iconv_lua,
"swim", swim_lua,
"internal.protobuf.wireformat", protobuf_wireformat_lua,
"protobuf", protobuf_lua,
COMPRESS_LUA_MODULES
/* jit.* library */
"jit.vmdef", jit_vmdef_lua,
Expand Down

0 comments on commit 1d2a75d

Please sign in to comment.