Skip to content

Commit

Permalink
Fixes gh-352 Deprecate msgpack.ibuf_decode
Browse files Browse the repository at this point in the history 
(cherry picked from commit 2cea32a)
  • Loading branch information
@pgulutzan
pgulutzan committed Jun 2, 2020
1 parent de0b4b4 commit ac3ccfe
Show file tree
Hide file tree
Showing 3 changed files with 151 additions and 58 deletions.
2 changes: 1 addition & 1 deletion doc/dev_guide/internals/box_protocol.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -626,7 +626,7 @@ response will look like this:
IPROTO_DATA is what we get with net_box and :ref:`Module buffer <buffer-module>`
so if we were using net_box we could decode with
:ref:`msgpack.decode_unchecked() <msgpack-decode_unchecked>`,
:ref:`msgpack.decode_unchecked() <msgpack-decode_unchecked_string>`,
or we could convert to a string with :samp:`ffi.string({pointer},{length})`.
The :ref:`pickle.unpack() <pickle-unpack>` function might also be helpful.

Expand Down
11 changes: 7 additions & 4 deletions doc/reference/reference_lua/buffer.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,19 @@
-------------------------------------------------------------------------------

The ``buffer`` module returns a dynamically resizable buffer which is solely
for use as an option for methods of the :ref:`net.box module <net_box-module>`.
for optional use by methods of the :ref:`net.box module <net_box-module>`
or the :ref:`msgpack module <msgpack-module>`.

Ordinarily the ``net.box`` methods return a Lua table.
If a ``buffer`` option is used, then the ``net.box`` methods return a
raw MsgPack_ string.
:ref:`raw MsgPack string <msgpack-definitions>`.
This saves time on the server, if the client application has
its own routine for decoding MsgPack strings.
its own routine for decoding raw MsgPack strings.

.. module:: buffer

.. _buffer-ibuf:

.. function:: ibuf()

:return: a descriptor of a buffer.
Expand All @@ -30,7 +33,7 @@ its own routine for decoding MsgPack strings.
for a ``conn.space...select()`` call.
The result will be in MsgPack_ format.
To show this, we will use
:ref:`msgpack.decode_unchecked() <msgpack-decode_unchecked>`
:ref:`msgpack.decode_unchecked() <msgpack-decode_unchecked_c_style_string_pointer>`
on ``ibuf.rpos`` (the "read position" of the buffer).
Thus we do not decode on the remote server, but we do
decode on the local server.
Expand Down
196 changes: 143 additions & 53 deletions doc/reference/reference_lua/msgpack.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,21 @@
Overview
===============================================================================

The ``msgpack`` module takes strings in MsgPack_ format and decodes them, or
takes a series of non-MsgPack values and encodes them.
The ``msgpack`` module decodes
:ref:`raw MsgPack strings <msgpack-definitions>` by converting them to Lua objects,
and encodes Lua objects by converting them to raw MsgPack strings.
Tarantool makes heavy internal use of MsgPack because tuples in Tarantool
are :ref:`stored <index-box_lua-vs-msgpack>` as MsgPack arrays.

.. _msgpack-definitions:

Definitions: MsgPack is short for `MessagePack <https://msgpack.org/index.html>`_.
A "raw MsgPack string" is a byte array formatted according to the
`MsgPack specification <https://github.com/msgpack/msgpack/blob/master/spec.md>`_
including type bytes and sizes.
The type bytes and sizes can be made displayable with :ref:`string.hex() <string-hex>`,
or the raw MsgPack strings can be converted to Lua objects with ``msgpack`` methods.

===============================================================================
Index
===============================================================================
Expand All @@ -24,60 +34,101 @@ Below is a list of all ``msgpack`` functions and members.
.. rst-class:: left-align-column-1
.. rst-class:: left-align-column-2

+--------------------------------------+---------------------------------+
| Name | Use |
+======================================+=================================+
| :ref:`msgpack.encode() | Convert a Lua object to an |
| <msgpack-encode>` | MsgPack string |
+--------------------------------------+---------------------------------+
| :ref:`msgpack.decode() | Convert a MsgPack string to a |
| <msgpack-decode>` | Lua object |
+--------------------------------------+---------------------------------+
| :ref:`msgpack.decode_unchecked() | Convert a MsgPack string to a |
| <msgpack-decode_unchecked>` | Lua object |
+--------------------------------------+---------------------------------+
| :ref:`msgpack.decode_array_header | Skip array header in a MsgPack |
| <msgpack-decode_array_header>` | string |
+--------------------------------------+---------------------------------+
| :ref:`msgpack.decode_map_header | Skip map header in a MsgPack |
| <msgpack-decode_map_header>` | string |
+--------------------------------------+---------------------------------+
| :ref:`__serialize parameter | Output structure specification |
| <msgpack-serialize>` | |
+--------------------------------------+---------------------------------+
| :ref:`msgpack.cfg | Change configuration |
| <msgpack-cfg>` | |
+--------------------------------------+---------------------------------+
| :ref:`msgpack.NULL | Analog of Lua's "nil" |
| <msgpack-null>` | |
+--------------------------------------+---------------------------------+
+--------------------------------------------------------+---------------------------------+
| Name | Use |
+========================================================+=================================+
| :ref:`msgpack.encode(lua_value) | Convert a Lua object to a |
| <msgpack-encode_lua_value>` | raw MsgPack string |
+--------------------------------------------------------+---------------------------------+
| :ref:`msgpack.encode(lua_value,ibuf) | Convert a Lua object to a |
| <msgpack-encode_lua_value_ibuf>` | raw MsgPack string in an ibuf |
+--------------------------------------------------------+---------------------------------+
| :ref:`msgpack.decode(msgpack_string) | Convert a raw MsgPack string to |
| <msgpack-decode_string>` | a Lua object |
+--------------------------------------------------------+---------------------------------+
| :ref:`msgpack.decode(C_style_string_pointer) | Convert a raw MsgPack string in |
| <msgpack-decode_c_style_string_pointer>` | an ibuf to a Lua object |
+--------------------------------------------------------+---------------------------------+
| :ref:`msgpack.decode_unchecked(mspack_string) | Convert a raw MsgPack string to |
| <msgpack-decode_unchecked_string>` | a Lua object |
+--------------------------------------------------------+---------------------------------+
| :ref:`msgpack.decode_unchecked(C_style_string_pointer) | Convert a raw MsgPack string to |
| <msgpack-decode_unchecked_c_style_string_pointer>` | a Lua object |
+--------------------------------------------------------+---------------------------------+
| :ref:`msgpack.decode_array_header | Skip array header in a raw |
| <msgpack-decode_array_header>` | MsgPack string |
+--------------------------------------------------------+---------------------------------+
| :ref:`msgpack.decode_map_header | Skip map header in a raw |
| <msgpack-decode_map_header>` | MsgPack string |
+--------------------------------------------------------+---------------------------------+
| :ref:`__serialize parameter | Output structure specification |
| <msgpack-serialize>` | |
+--------------------------------------------------------+---------------------------------+
| :ref:`msgpack.cfg | Change configuration |
| <msgpack-cfg>` | |
+--------------------------------------------------------+---------------------------------+
| :ref:`msgpack.NULL | Analog of Lua's "nil" |
| <msgpack-null>` | |
+--------------------------------------------------------+---------------------------------+

.. module:: msgpack

.. _msgpack-encode:
.. _msgpack-encode_lua_value:

.. function:: encode(lua_value)

Convert a Lua object to a MsgPack string.
Convert a Lua object to a raw MsgPack string.

:param lua_value: either a scalar value or a Lua table value.
:return: the original value reformatted as a MsgPack string.
:rtype: string

.. _msgpack-decode:
:return: the original contents formatted as a raw MsgPack string;

:rtype: raw MsgPack string

.. _msgpack-encode_lua_value_ibuf:

.. function:: encode(lua_value, ibuf)

Convert a Lua object to a raw MsgPack string in an ibuf,
which is a buffer such as :ref:`buffer.ibuf() <buffer-ibuf>` creates.
As with :ref:`encode(lua_value) <msgpack-encode_lua_value>`,
the result is a raw MsgPack string,
but it goes to the ``ibuf`` output instead of being returned.

:param lua-object lua_value: either a scalar value or a Lua table value.
:param buffer ibuf: (output parameter) where the result raw MsgPack string goes
:return: number of bytes in the output

:rtype: raw MsgPack string

Example using :ref:`buffer.ibuf() <buffer-ibuf>`
and `ffi.string() <https://luajit.org/ext_ffi_api.html>`_
and :ref:`string.hex() <string-hex>`:
The result will be '91a161' because 91 is the MessagePack encoding of "fixarray size 1",
a1 is the MessagePack encoding of "fixstr size 1",
and 61 is the UTF-8 encoding of 'a':

.. code-block:: none
ibuf = require('buffer').ibuf()
msgpack_string_size = require('msgpack').encode({'a'}, ibuf)
msgpack_string = require('ffi').string(ibuf.rpos, msgpack_string_size)
string.hex(msgpack_string)
.. _msgpack-decode_string:

.. function:: decode(msgpack_string [, start_position])

Convert a MsgPack string to a Lua object.
Convert a raw MsgPack string to a Lua object.

:param string msgpack_string: a string formatted as MsgPack.
:param string msgpack_string: a raw MsgPack string.
:param integer start_position: where to start, minimum = 1,
maximum = string length, default = 1.

:return:

* (if ``msgpack_string`` is in valid MsgPack format) the original contents
of ``msgpack_string``, formatted as a Lua table,
* (if ``msgpack_string`` is a valid raw MsgPack string) the original contents
of ``msgpack_string``, formatted as a Lua object, usually a Lua table,
(otherwise) a scalar value, such as a string or a number;
* "next_start_position". If ``decode`` stops after parsing as far as
byte N in ``msgpack_string``, then "next_start_position" will equal N + 1,
Expand All @@ -86,26 +137,65 @@ Below is a list of all ``msgpack`` functions and members.
Normally ``decode`` parses all of ``msgpack_string``, so
"next_start_position" will equal ``string.len(msgpack_string)`` + 1.

:rtype: table and number
:rtype: Lua object and number

.. _msgpack-decode_unchecked:
Example: The result will be ['a'] and 4:

.. function:: decode_unchecked(string)
.. code-block:: none
Convert a MsgPack string to a Lua object.
Because checking is skipped, ``decode_unchecked()``
can operate with string pointers to
buffers which ``decode()`` cannot handle.
For an example see the :ref:`buffer <buffer-module>` module.
msgpack_string = require('msgpack').encode({'a'})
require('msgpack').decode(msgpack_string, 1)
:param string: a string formatted as MsgPack.
.. _msgpack-decode_c_style_string_pointer:

.. function:: decode(C_style_string_pointer, size)

Convert a raw MsgPack string, whose address is supplied as a C-style string pointer
such as the ``rpos`` pointer which is inside an ibuf such as
:ref:`buffer.ibuf() <buffer-ibuf>` creates, to a Lua object.
A C-style string pointer may be described as ``cdata<char *>`` or ``cdata<const char *>``.

:param buffer C_style_string_pointer: a pointer to a raw MsgPack string.
:param integer size: number of bytes in the raw MsgPack string

:return:

* the original contents formatted as a Lua table;
* the number of bytes that were decoded.
* (if C_style_string_pointer points to a valid raw MsgPack string) the original contents
of ``msgpack_string``, formatted as a Lua object, usually a Lua table,
(otherwise) a scalar value, such as a string or a number;
* returned_pointer = a C-style pointer to the byte after
what was passed, so that C_style_string_pointer + size = returned_pointer

:rtype: table and C-style pointer to after what was passed

Example using :ref:`buffer.ibuf <buffer-ibuf>`
and pointer arithmetic:
The result will be ['a'] and 3 and true:

.. code-block:: none
:rtype: lua object
ibuf = require('buffer').ibuf()
msgpack_string_size = require('msgpack').encode({'a'}, ibuf)
a, b = require('msgpack').decode(ibuf.rpos, msgpack_string_size)
a, b - ibuf.rpos, msgpack_string_size == b - ibuf.rpos
.. _msgpack-decode_unchecked_string:

.. function:: decode_unchecked(msgpack_string [, start_position])

Input and output are the same as for
:ref:`decode(string) <msgpack-decode_string>`.

.. _msgpack-decode_unchecked_c_style_string_pointer:

.. function:: decode_unchecked(C_style_string_pointer[, size])

Input and output are the same as for
:ref:`decode(C_style_string_pointer) <msgpack-decode_c_style_string_pointer>`,
except that ``size`` is optional.
Some checking is skipped, and ``decode_unchecked(C_style_string_pointer)`` can operate with
string pointers to buffers which ``decode(C_style_string_pointer)`` cannot handle.
For an example see the :ref:`buffer <buffer-module>` module.

.. _msgpack-decode_array_header:

Expand All @@ -115,7 +205,7 @@ Below is a list of all ``msgpack`` functions and members.
and return the array size and a pointer to the first array component.
A subsequent call to ``msgpack_decode`` can decode the component instead of the whole array.

:param byte-array: a pointer to a byte array formatted as MsgPack.
:param byte-array: a pointer to a raw MsgPack string.
:param size: a number greater than or equal to the string's length

:return:
Expand Down Expand Up @@ -143,8 +233,8 @@ Below is a list of all ``msgpack`` functions and members.
and return the map size and a pointer to the first map component.
A subsequent call to ``msgpack_decode`` can decode the component instead of the whole map.

:param byte-array: a pointer to a byte array formatted as MsgPack.
:param size: a number greater than or equal to the byte array's length
:param byte-array: a pointer to a raw MsgPack string.
:param size: a number greater than or equal to the raw MsgPack string's length

:return:

Expand Down

0 comments on commit ac3ccfe

Please sign in to comment.