Fixes gh-352 Deprecate msgpack.ibuf_decode

doc/reference/reference_lua/buffer.rst

@@ -5,16 +5,19 @@
 -------------------------------------------------------------------------------
 
 The ``buffer`` module returns a dynamically resizable buffer which is solely
-for use as an option for methods of the :ref:`net.box module <net_box-module>`.
+for optional use by methods of the :ref:`net.box module <net_box-module>`
+or the :ref:`msgpack module <msgpack-module>`.
 
 Ordinarily the ``net.box`` methods return a Lua table.
 If a ``buffer`` option is used, then the ``net.box`` methods return a
-raw MsgPack_ string.
+:ref:`raw MsgPack string <msgpack-definitions>`.
 This saves time on the server, if the client application has
-its own routine for decoding MsgPack strings.
+its own routine for decoding raw MsgPack strings.
 
 .. module:: buffer
 
+.. _buffer-ibuf:
+
 .. function:: ibuf()
 
     :return: a descriptor of a buffer.
@@ -30,7 +33,7 @@ its own routine for decoding MsgPack strings.
     for a ``conn.space...select()`` call.
     The result will be in MsgPack_ format.
     To show this, we will use
-    :ref:`msgpack.decode_unchecked() <msgpack-decode_unchecked>`
+    :ref:`msgpack.decode_unchecked() <msgpack-decode_unchecked_c_style_string_pointer>`
     on ``ibuf.rpos`` (the "read position" of the buffer).
     Thus we do not decode on the remote server, but we do
     decode on the local server.
@@ -63,4 +66,65 @@ its own routine for decoding MsgPack strings.
         this case is ``msgpack.ibuf_decode(ibuf.rpos)``. Starting
         with Tarantool version 1.7.7, ``ibuf_decode`` is deprecated.
 
-.. _MsgPack: http://msgpack.org/
+
+.. _buffer-module_and_skip_header:
+
+**Module buffer and skip-header**
+
+The example in the previous section
+
+    .. code-block:: tarantoolsession
+
+        tarantool> msgpack.decode_unchecked(ibuf.rpos)
+        ---
+        - {48: [['ABCDE', 12345]]}
+        - 'cdata<char *>: 0x7f97ba10c041'
+        ...
+
+showed that, ordinarily, the response from net.box includes a header --
+48 (hexadecimal 30) is the :ref:`key <internals-unified_packet_structure>`
+for IPROTO_DATA. But in some situations,
+for example when passing the buffer to a C function
+that expects a MsgPack byte array without a header,
+the header can be skipped. This is done by specifying
+``skip-header=true`` as an option to
+:ref:`conn.space.space-name:select{...} <conn-select>` or
+:ref:`conn.space.space-name:insert{...} <conn-insert>` or
+:ref:`conn.space.space-name:replace{...} <conn-replace>` or
+:ref:`conn.space.space-name:update{...} <conn-update>` or
+:ref:`conn.space.space-name:upsert{...} <conn-upsert>` or
+:ref:`conn.space.space-name:delete{...} <conn-delete>`.
+The default is ``skip-header=false``.
+
+Now here is the same example, except that ``skip_header=true`` is used.
+
+    .. code-block:: lua
+
+        box.cfg{listen=3302}
+        buffer = require('buffer')
+        ibuf = buffer.ibuf()
+        net_box = require('net.box')
+        conn = net_box.connect('farhost:3301')
+        buffer = require('buffer')
+        conn.space.T:select({},{buffer=ibuf, skip_header=true})
+        msgpack = require('msgpack')
+        msgpack.decode_unchecked(ibuf.rpos)
+
+    The result of the final request looks like this:
+
+    .. code-block:: tarantoolsession
+
+        tarantool>         msgpack.decode_unchecked(ibuf.rpos)
+        ---
+        - [['ABCDE', 12345]]
+        - 'cdata<char *>: 0x7f8fd102803f'
+        ...
+
+    Notice that the IPROTO_DATA header (48) is gone.
+
+    The result is still inside an array, as is clear from the fact
+    that it is shown inside square brackets. It is possible to skip
+    the array header too, with
+    :ref:`msgpack.decode_array_header() <msgpack-decode_array_header>`.
+
+.. _MsgPack: http://msgpack.org/

doc/reference/reference_lua/msgpack.rst

@@ -8,11 +8,21 @@
                                    Overview
 ===============================================================================
 
-The ``msgpack`` module takes strings in MsgPack_ format and decodes them, or
-takes a series of non-MsgPack values and encodes them.
+The ``msgpack`` module decodes
+:ref:`raw MsgPack strings <msgpack-definitions>` by converting them to Lua objects,
+and encodes Lua objects by converting them to raw MsgPack strings.
 Tarantool makes heavy internal use of MsgPack because tuples in Tarantool
 are :ref:`stored <index-box_lua-vs-msgpack>` as MsgPack arrays.
 
+.. _msgpack-definitions:
+
+Definitions: MsgPack is short for `MessagePack <https://msgpack.org/index.html>`_.
+A "raw MsgPack string" is a byte array formatted according to the
+`MsgPack specification <https://github.com/msgpack/msgpack/blob/master/spec.md>`_ 
+including type bytes and sizes.
+The type bytes and sizes can be made displayable with :ref:`string.hex() <string-hex>`,
+or the raw MsgPack strings can be converted to Lua objects with ``msgpack`` methods.
+
 ===============================================================================
                                     Index
 ===============================================================================
@@ -24,55 +34,101 @@ Below is a list of all ``msgpack`` functions and members.
     .. rst-class:: left-align-column-1
     .. rst-class:: left-align-column-2
 
-    +--------------------------------------+---------------------------------+
-    | Name                                 | Use                             |
-    +======================================+=================================+
-    | :ref:`msgpack.encode()               | Convert a Lua object to an      |
-    | <msgpack-encode>`                    | MsgPack string                  |
-    +--------------------------------------+---------------------------------+
-    | :ref:`msgpack.decode()               | Convert a MsgPack string to a   |
-    | <msgpack-decode>`                    | Lua object                      |
-    +--------------------------------------+---------------------------------+
-    | :ref:`msgpack.decode_unchecked()     | Convert a MsgPack string to a   |
-    | <msgpack-decode_unchecked>`          | Lua object                      |
-    +--------------------------------------+---------------------------------+
-    | :ref:`__serialize parameter          | Output structure specification  |
-    | <msgpack-serialize>`                 |                                 |
-    +--------------------------------------+---------------------------------+
-    | :ref:`msgpack.cfg                    | Change configuration            |
-    | <msgpack-cfg>`                       |                                 |
-    +--------------------------------------+---------------------------------+
-    | :ref:`msgpack.NULL                   | Analog of Lua's "nil"           |
-    | <msgpack-null>`                      |                                 |
-    +--------------------------------------+---------------------------------+
-
+    +--------------------------------------------------------+---------------------------------+
+    | Name                                                   | Use                             |
+    +========================================================+=================================+
+    | :ref:`msgpack.encode(lua_value)                        | Convert a Lua object to a       |
+    | <msgpack-encode_lua_value>`                            | raw MsgPack string              |
+    +--------------------------------------------------------+---------------------------------+
+    | :ref:`msgpack.encode(lua_value,ibuf)                   | Convert a Lua object to a       |
+    | <msgpack-encode_lua_value_ibuf>`                       | raw MsgPack string in an ibuf   |
+    +--------------------------------------------------------+---------------------------------+
+    | :ref:`msgpack.decode(msgpack_string)                   | Convert a raw MsgPack string to |
+    | <msgpack-decode_string>`                               | a Lua object                    |
+    +--------------------------------------------------------+---------------------------------+
+    | :ref:`msgpack.decode(C_style_string_pointer)           | Convert a raw MsgPack string in |
+    | <msgpack-decode_c_style_string_pointer>`               | an ibuf to a Lua object         |
+    +--------------------------------------------------------+---------------------------------+
+    | :ref:`msgpack.decode_unchecked(mspack_string)          | Convert a raw MsgPack string to |
+    | <msgpack-decode_unchecked_string>`                     | a Lua object                    |
+    +--------------------------------------------------------+---------------------------------+
+    | :ref:`msgpack.decode_unchecked(C_style_string_pointer) | Convert a raw MsgPack string to |
+    | <msgpack-decode_unchecked_c_style_string_pointer>`     | a Lua object                    |
+    +--------------------------------------------------------+---------------------------------+
+    | :ref:`msgpack.decode_array_header                      | Skip array header in a raw      |
+    | <msgpack-decode_array_header>`                         | MsgPack string                  |
+    +--------------------------------------------------------+---------------------------------+
+    | :ref:`msgpack.decode_map_header                        | Skip map header in a raw        |
+    | <msgpack-decode_map_header>`                           | MsgPack string                  |
+    +--------------------------------------------------------+---------------------------------+
+    | :ref:`__serialize parameter                            | Output structure specification  |
+    | <msgpack-serialize>`                                   |                                 |
+    +--------------------------------------------------------+---------------------------------+
+    | :ref:`msgpack.cfg                                      | Change configuration            |
+    | <msgpack-cfg>`                                         |                                 |
+    +--------------------------------------------------------+---------------------------------+
+    | :ref:`msgpack.NULL                                     | Analog of Lua's "nil"           |
+    | <msgpack-null>`                                        |                                 |
+    +--------------------------------------------------------+---------------------------------+
 
 .. module:: msgpack
 
-.. _msgpack-encode:
+.. _msgpack-encode_lua_value:
 
 .. function:: encode(lua_value)
 
-    Convert a Lua object to a MsgPack string.
+    Convert a Lua object to a raw MsgPack string.
 
     :param lua_value: either a scalar value or a Lua table value.
-    :return: the original value reformatted as a MsgPack string.
-    :rtype: string
 
-.. _msgpack-decode:
+    :return: the original contents formatted as a raw MsgPack string;
+
+    :rtype: raw MsgPack string
+
+.. _msgpack-encode_lua_value_ibuf:
+
+.. function:: encode(lua_value, ibuf)
+
+    Convert a Lua object to a raw MsgPack string in an ibuf,
+    which is a buffer such as :ref:`buffer.ibuf() <buffer-ibuf>` creates.
+    As with :ref:`encode(lua_value) <msgpack-encode_lua_value>`,
+    the result is a raw MsgPack string,
+    but it goes to the ``ibuf`` output instead of being returned.
+
+    :param lua-object lua_value: either a scalar value or a Lua table value.
+    :param buffer ibuf: (output parameter) where the result raw MsgPack string goes
+    :return: number of bytes in the output
+
+    :rtype: raw MsgPack string
+
+    Example using :ref:`buffer.ibuf() <buffer-ibuf>`
+    and `ffi.string() <https://luajit.org/ext_ffi_api.html>`_
+    and :ref:`string.hex() <string-hex>`:
+    The result will be '91a161' because 91 is the MessagePack encoding of "fixarray size 1",
+    a1 is the MessagePack encoding of "fixstr size 1",
+    and 61 is the UTF-8 encoding of 'a':
+
+    .. code-block:: none
+
+        ibuf = require('buffer').ibuf()
+        msgpack_string_size = require('msgpack').encode({'a'}, ibuf)
+        msgpack_string = require('ffi').string(ibuf.rpos, msgpack_string_size)
+        string.hex(msgpack_string)
+
+.. _msgpack-decode_string:
 
 .. function:: decode(msgpack_string [, start_position])
 
-    Convert a MsgPack string to a Lua object.
+    Convert a raw MsgPack string to a Lua object.
 
-    :param string msgpack_string: a string formatted as MsgPack.
+    :param string msgpack_string: a raw MsgPack string.
     :param integer start_position: where to start, minimum = 1,
                                    maximum = string length, default = 1.
 
     :return:
 
-      * (if ``msgpack_string`` is in valid MsgPack format) the original contents
-        of ``msgpack_string``, formatted as a Lua table,
+      * (if ``msgpack_string`` is a valid raw MsgPack string) the original contents
+        of ``msgpack_string``, formatted as a Lua object, usually a Lua table,
         (otherwise) a scalar value, such as a string or a number;
       * "next_start_position". If ``decode`` stops after parsing as far as
         byte N in ``msgpack_string``, then "next_start_position" will equal N + 1,
@@ -81,26 +137,121 @@ Below is a list of all ``msgpack`` functions and members.
         Normally ``decode`` parses all of ``msgpack_string``, so
         "next_start_position" will equal ``string.len(msgpack_string)`` + 1.
 
-    :rtype: table and number
+    :rtype: Lua object and number
+
+    Example: The result will be ['a'] and 4:
+
+    .. code-block:: none
+
+        msgpack_string = require('msgpack').encode({'a'})
+        require('msgpack').decode(msgpack_string, 1)
+
+.. _msgpack-decode_c_style_string_pointer:
+
+.. function:: decode(C_style_string_pointer, size)
+
+    Convert a raw MsgPack string, whose address is supplied as a C-style string pointer
+    such as the ``rpos`` pointer which is inside an ibuf such as
+    :ref:`buffer.ibuf() <buffer-ibuf>` creates, to a Lua object.
+    A C-style string pointer may be described as ``cdata<char *>`` or ``cdata<const char *>``.
+
+    :param buffer C_style_string_pointer: a pointer to a raw MsgPack string.
+    :param integer size: number of bytes in the raw MsgPack string
+
+    :return:
+
+      * (if C_style_string_pointer points to a valid raw MsgPack string) the original contents
+        of ``msgpack_string``, formatted as a Lua object, usually a Lua table,
+        (otherwise) a scalar value, such as a string or a number;
+      * returned_pointer = a C-style pointer to the byte after
+        what was passed, so that C_style_string_pointer + size = returned_pointer
+
+    :rtype: table and C-style pointer to after what was passed
+
+    Example using :ref:`buffer.ibuf <buffer-ibuf>`
+    and pointer arithmetic:
+    The result will be ['a'] and 3 and true:
+
+    .. code-block:: none
+
+        ibuf = require('buffer').ibuf()
+        msgpack_string_size = require('msgpack').encode({'a'}, ibuf)
+        a, b = require('msgpack').decode(ibuf.rpos, msgpack_string_size)
+        a, b - ibuf.rpos, msgpack_string_size == b - ibuf.rpos
 
-.. _msgpack-decode_unchecked:
+.. _msgpack-decode_unchecked_string:
 
-.. function:: decode_unchecked(string)
+.. function:: decode_unchecked(msgpack_string [, start_position])
 
-    Convert a MsgPack string to a Lua object.
-    Because checking is skipped, ``decode_unchecked()``
-    can operate with string pointers to
-    buffers which ``decode()`` cannot handle.
+    Input and output are the same as for
+    :ref:`decode(string) <msgpack-decode_string>`.
+
+.. _msgpack-decode_unchecked_c_style_string_pointer:
+
+.. function:: decode_unchecked(C_style_string_pointer)
+
+    Input and output are the same as for
+    :ref:`decode(C_style_string_pointer) <msgpack-decode_c_style_string_pointer>`,
+    except that ``size`` is not needed.
+    Some checking is skipped, and ``decode_unchecked(C_style_string_pointer)`` can operate with
+    string pointers to buffers which ``decode(C_style_string_pointer)`` cannot handle. 
     For an example see the :ref:`buffer <buffer-module>` module.
 
-    :param string: a string formatted as MsgPack.
+.. _msgpack-decode_array_header:
+
+.. function:: decode_array_header(byte-array, size)
+
+    Call the mp_decode_array function in the `MsgPuck <http://rtsisyk.github.io/msgpuck/>`_ library
+    and return the array size and a pointer to the first array component.
+    A subsequent call to ``msgpack_decode`` can decode the component instead of the whole array.
+
+    :param byte-array: a pointer to a raw MsgPack string.
+    :param size: a number greater than or equal to the string's length
 
     :return:
 
-      * the original contents formatted as a Lua table;
-      * the number of bytes that were decoded.
+      * the size of the array;
+      * a pointer to after the array header.
+
+    .. code-block:: none
+
+        -- Example of decode_array_header
+        -- Suppose we have the raw data '\x93\x01\x02\x03'.
+        -- \x93 is MsgPack encoding for a header of a three-item array.
+        -- We want to skip it and decode the next three items.
+        msgpack=require('msgpack'); ffi=require('ffi')
+        x,y=msgpack.decode_array_header(ffi.cast('char*','\x93\x01\x02\x03'),4)
+        a=msgpack.decode(y,1);b=msgpack.decode(y+1,1);c=msgpack.decode(y+2,1);
+        a,b,c
+        -- The result will be: 1,2,3.
+
+.. _msgpack-decode_map_header:
+
+.. function:: decode_map_header(byte-array, size)
+
+    Call the mp_decode_map function in the `MsgPuck <http://rtsisyk.github.io/msgpuck/>`_ library
+    and return the map size and a pointer to the first map component.
+    A subsequent call to ``msgpack_decode`` can decode the component instead of the whole map.
+
+    :param byte-array: a pointer to a raw MsgPack string.
+    :param size: a number greater than or equal to the raw MsgPack string's length
+
+    :return:
+
+      * the size of the map;
+      * a pointer to after the map header.
+
+    .. code-block:: none
 
-    :rtype: lua object
+        -- Example of decode_map_header
+        -- Suppose we have the raw data '\x81\xa2\x41\x41\xc3'.
+        -- \x81 is MsgPack encoding for a header of a one-item map.
+        -- We want to skip it and decode the next map item.
+        msgpack=require('msgpack'); ffi=require('ffi')
+        x,y=msgpack.decode_map_header(ffi.cast('char*','\x81\xa2\x41\x41\xc3'),5)
+        a=msgpack.decode(y,3);b=msgpack.decode(y+3,1)
+        x,a,b
+        -- The result will be: 1,"AA", true.
 
 .. _msgpack-serialize: