lua: return raw msgpack from a function #4861
Labels
Milestone
Comments
This was referenced Sep 21, 2020
Merged
locker
added a commit
to locker/tarantool
that referenced
this issue
Dec 9, 2021
Closes tarantool#1629 Closes tarantool#3909 Closes tarantool#5316 Needed for tarantool#3349 Needed for tarantool#4861 @TarantoolBot document Title: Document msgpack.object The following new methods were added to the Lua `msgpack` module: - `msgpack.object()`: Encodes an arbitrary Lua object given in its only argument in MsgPack and returns the encoded MsgPack data encapsulated in a MsgPack object. - `msgpack.object_from_raw()`: Creates a MsgPack object from raw data given either as a string or as a pointer and size. - `msgpack.is_object()`: Checks is the given argument is a MsgPack object. Example: ```lua local buffer = require('buffer') local msgpack = require('msgpack') -- Create an object from a Lua object. local mp = msgpack.object(123) local mp = msgpack.object("foobar") local mp = msgpack.object({1, 2, 3}) local mp = msgpack.object({foo = 1, bar = 2}) local mp = msgpack.object(box.tuple.new(1, 2, 3)) -- Create an object from raw data given as a string. local data = msgpack.encode({1, 2, 3}) local mp = msgpack.object_from_raw(data) -- Create an object from raw data given as ptr and size. local buf = buffer.ibuf() msgpack.encode({1, 2, 3}, buf) local mp = msgpack.object_from_raw(buf.buf, buf:size()) -- Check object. msgpack.is_object(mp) -- true msgpack.is_object({}) -- false ``` A MsgPack object has the following methods: - `mp:decode()`: Decodes MsgPack and returns a Lua object. - `mp:iterator()`: Returns an iterator over MsgPack data. A MsgPack iterator has the following methods: - `it:decode_array_header()`: Decodes a MsgPack array header under the cursor and returns the number of elements in the array. After calling this function the iterator points to the first element of the array or to the value following the array if the array is empty. Raises an error if the type of the value under the iterator cursor is not `MP_ARRAY`. - `it:decode_map_header()`: Decodes a MsgPack map header under the cursor and returns the number of key value pairs in the map. After calling this function the iterator points to the first key stored in the map or to the value following the map if the map is empty. Raises an error if the type of the value under the iterator cursor is not `MP_MAP`. - `it:decode()`: Decodes a MsgPack value under the iterator cursor and advances the cursor. Returns a Lua object corresponding to the MsgPack value. Raises a Lua error if there's no data to decode. - `it:take()`: Returns a MsgPack value under the iterator cursor as a MsgPack object (without decoding). Raises a Lua error if there's no data to decode. This function doesn't copy MsgPack data - instead it takes a reference to the original object. - `it:skip()`: Advances the iterator cursor by skipping one MsgPack value under the cursor. Returns nothing. Raises a Lua error if there's not data to skip. Usage example: ```lua local msgpack = require('msgpack') local mp = msgpack.object({foo = 123, bar = {1, 2, 3}}) local it = mp:iterator() it:decode_map_header() -- returns 2 it:decode() -- returns 'foo' it:decode() -- returns 123 it:skip() -- returns none, skips 'bar' local mp2 = it:take() mp2:decode() -- returns {1, 2, 3} ``` For more examples, see `test/app-luatest/msgpack_test.lua`. A MsgPack object can be passed to MsgPack encoder with the effect being the same as passing the original Lua object: ```lua local msgpack = require('msgpack') local mp = msgpack.object(123) msgpack.object({mp, mp}):decode() -- returns {123, 123} msgpack.decode(msgpack.encode({mp, mp})) -- returns {123, 123} ``` In particular, this means that if a MsgPack object stores an array, it can be inserted into a database space: ```lua box.space.my_space:insert(msgpack.object({1, 2, 3})) ```
locker
added a commit
to locker/tarantool
that referenced
this issue
Dec 9, 2021
Closes tarantool#4861 @TarantoolBot document Title: Document return_raw net.box option If the `return_raw` flag is set, net.box will return response data wrapped in a msgpack object (see `msgpack.object()`) instead of decoding it to Lua. This can be useful when a response is supposed to be passed through without decoding or with partial decoding - creating a msgpack object can reduce pressure on the Lua garbage collector in this case. Example: ```lua local c = require('net.box').connect(uri) local mp = c.eval('eval ...', {1, 2, 3}, {return_raw = true}) mp:decode() -- {1, 2, 3} ``` The option is ignored for methods that return nil (`begin`, `commit`, `rollback`, `upsert`, `prepare` will return nil even if `return_raw` is set) and for `index.count` (it returns a number). For `execute`, the option is applied only to data (`rows`) while metadata is decoded irrespective of the value of `return_raw`.
locker
added a commit
to locker/tarantool
that referenced
this issue
Dec 13, 2021
Closes tarantool#1629 Closes tarantool#3909 Closes tarantool#5316 Needed for tarantool#3349 Needed for tarantool#4861 @TarantoolBot document Title: Document msgpack.object The following new methods were added to the Lua `msgpack` module: - `msgpack.object()`: Encodes an arbitrary Lua object given in its only argument in MsgPack and returns the encoded MsgPack data encapsulated in a MsgPack object. - `msgpack.object_from_raw()`: Creates a MsgPack object from raw data given either as a string or as a pointer and size. - `msgpack.is_object()`: Checks is the given argument is a MsgPack object. Example: ```lua local buffer = require('buffer') local msgpack = require('msgpack') -- Create an object from a Lua object. local mp = msgpack.object(123) local mp = msgpack.object("foobar") local mp = msgpack.object({1, 2, 3}) local mp = msgpack.object({foo = 1, bar = 2}) local mp = msgpack.object(box.tuple.new(1, 2, 3)) -- Create an object from raw data given as a string. local data = msgpack.encode({1, 2, 3}) local mp = msgpack.object_from_raw(data) -- Create an object from raw data given as ptr and size. local buf = buffer.ibuf() msgpack.encode({1, 2, 3}, buf) local mp = msgpack.object_from_raw(buf.buf, buf:size()) -- Check object. msgpack.is_object(mp) -- true msgpack.is_object({}) -- false ``` A MsgPack object has the following methods: - `mp:decode()`: Decodes MsgPack and returns a Lua object. - `mp:iterator()`: Returns an iterator over MsgPack data. A MsgPack iterator has the following methods: - `it:decode_array_header()`: Decodes a MsgPack array header under the cursor and returns the number of elements in the array. After calling this function the iterator points to the first element of the array or to the value following the array if the array is empty. Raises an error if the type of the value under the iterator cursor is not `MP_ARRAY`. - `it:decode_map_header()`: Decodes a MsgPack map header under the cursor and returns the number of key value pairs in the map. After calling this function the iterator points to the first key stored in the map or to the value following the map if the map is empty. Raises an error if the type of the value under the iterator cursor is not `MP_MAP`. - `it:decode()`: Decodes a MsgPack value under the iterator cursor and advances the cursor. Returns a Lua object corresponding to the MsgPack value. Raises a Lua error if there's no data to decode. - `it:take()`: Returns a MsgPack value under the iterator cursor as a MsgPack object (without decoding). Raises a Lua error if there's no data to decode. This function doesn't copy MsgPack data - instead it takes a reference to the original object. - `it:skip()`: Advances the iterator cursor by skipping one MsgPack value under the cursor. Returns nothing. Raises a Lua error if there's not data to skip. Usage example: ```lua local msgpack = require('msgpack') local mp = msgpack.object({foo = 123, bar = {1, 2, 3}}) local it = mp:iterator() it:decode_map_header() -- returns 2 it:decode() -- returns 'foo' it:decode() -- returns 123 it:skip() -- returns none, skips 'bar' local mp2 = it:take() mp2:decode() -- returns {1, 2, 3} ``` For more examples, see `test/app-luatest/msgpack_test.lua`. A MsgPack object can be passed to MsgPack encoder with the effect being the same as passing the original Lua object: ```lua local msgpack = require('msgpack') local mp = msgpack.object(123) msgpack.object({mp, mp}):decode() -- returns {123, 123} msgpack.decode(msgpack.encode({mp, mp})) -- returns {123, 123} ``` In particular, this means that if a MsgPack object stores an array, it can be inserted into a database space: ```lua box.space.my_space:insert(msgpack.object({1, 2, 3})) ```
locker
added a commit
to locker/tarantool
that referenced
this issue
Dec 13, 2021
Closes tarantool#4861 @TarantoolBot document Title: Document return_raw net.box option If the `return_raw` flag is set, net.box will return response data wrapped in a msgpack object (see `msgpack.object()`) instead of decoding it to Lua. This can be useful when a response is supposed to be passed through without decoding or with partial decoding - creating a msgpack object can reduce pressure on the Lua garbage collector in this case. Example: ```lua local c = require('net.box').connect(uri) local mp = c.eval('eval ...', {1, 2, 3}, {return_raw = true}) mp:decode() -- {1, 2, 3} ``` The option is ignored for methods that return nil (`begin`, `commit`, `rollback`, `upsert`, `prepare` will return nil even if `return_raw` is set) and for `index.count` (it returns a number). For `execute`, the option is applied only to data (`rows`) while metadata is decoded irrespective of the value of `return_raw`.
kyukhin
pushed a commit
that referenced
this issue
Dec 20, 2021
Closes #1629 Closes #3909 Closes #5316 Needed for #3349 Needed for #4861 @TarantoolBot document Title: Document msgpack.object The following new methods were added to the Lua `msgpack` module: - `msgpack.object()`: Encodes an arbitrary Lua object given in its only argument in MsgPack and returns the encoded MsgPack data encapsulated in a MsgPack object. - `msgpack.object_from_raw()`: Creates a MsgPack object from raw data given either as a string or as a pointer and size. - `msgpack.is_object()`: Checks is the given argument is a MsgPack object. Example: ```lua local buffer = require('buffer') local msgpack = require('msgpack') -- Create an object from a Lua object. local mp = msgpack.object(123) local mp = msgpack.object("foobar") local mp = msgpack.object({1, 2, 3}) local mp = msgpack.object({foo = 1, bar = 2}) local mp = msgpack.object(box.tuple.new(1, 2, 3)) -- Create an object from raw data given as a string. local data = msgpack.encode({1, 2, 3}) local mp = msgpack.object_from_raw(data) -- Create an object from raw data given as ptr and size. local buf = buffer.ibuf() msgpack.encode({1, 2, 3}, buf) local mp = msgpack.object_from_raw(buf.buf, buf:size()) -- Check object. msgpack.is_object(mp) -- true msgpack.is_object({}) -- false ``` A MsgPack object has the following methods: - `mp:decode()`: Decodes MsgPack and returns a Lua object. - `mp:iterator()`: Returns an iterator over MsgPack data. A MsgPack iterator has the following methods: - `it:decode_array_header()`: Decodes a MsgPack array header under the cursor and returns the number of elements in the array. After calling this function the iterator points to the first element of the array or to the value following the array if the array is empty. Raises an error if the type of the value under the iterator cursor is not `MP_ARRAY`. - `it:decode_map_header()`: Decodes a MsgPack map header under the cursor and returns the number of key value pairs in the map. After calling this function the iterator points to the first key stored in the map or to the value following the map if the map is empty. Raises an error if the type of the value under the iterator cursor is not `MP_MAP`. - `it:decode()`: Decodes a MsgPack value under the iterator cursor and advances the cursor. Returns a Lua object corresponding to the MsgPack value. Raises a Lua error if there's no data to decode. - `it:take()`: Returns a MsgPack value under the iterator cursor as a MsgPack object (without decoding). Raises a Lua error if there's no data to decode. This function doesn't copy MsgPack data - instead it takes a reference to the original object. - `it:skip()`: Advances the iterator cursor by skipping one MsgPack value under the cursor. Returns nothing. Raises a Lua error if there's not data to skip. Usage example: ```lua local msgpack = require('msgpack') local mp = msgpack.object({foo = 123, bar = {1, 2, 3}}) local it = mp:iterator() it:decode_map_header() -- returns 2 it:decode() -- returns 'foo' it:decode() -- returns 123 it:skip() -- returns none, skips 'bar' local mp2 = it:take() mp2:decode() -- returns {1, 2, 3} ``` For more examples, see `test/app-luatest/msgpack_test.lua`. A MsgPack object can be passed to MsgPack encoder with the effect being the same as passing the original Lua object: ```lua local msgpack = require('msgpack') local mp = msgpack.object(123) msgpack.object({mp, mp}):decode() -- returns {123, 123} msgpack.decode(msgpack.encode({mp, mp})) -- returns {123, 123} ``` In particular, this means that if a MsgPack object stores an array, it can be inserted into a database space: ```lua box.space.my_space:insert(msgpack.object({1, 2, 3})) ```
kyukhin
pushed a commit
that referenced
this issue
Dec 20, 2021
Closes #4861 @TarantoolBot document Title: Document return_raw net.box option If the `return_raw` flag is set, net.box will return response data wrapped in a msgpack object (see `msgpack.object()`) instead of decoding it to Lua. This can be useful when a response is supposed to be passed through without decoding or with partial decoding - creating a msgpack object can reduce pressure on the Lua garbage collector in this case. Example: ```lua local c = require('net.box').connect(uri) local mp = c.eval('eval ...', {1, 2, 3}, {return_raw = true}) mp:decode() -- {1, 2, 3} ``` The option is ignored for methods that return nil (`begin`, `commit`, `rollback`, `upsert`, `prepare` will return nil even if `return_raw` is set) and for `index.count` (it returns a number). For `execute`, the option is applied only to data (`rows`) while metadata is decoded irrespective of the value of `return_raw`.
yanshtunder
pushed a commit
that referenced
this issue
Feb 8, 2022
Closes #1629 Closes #3909 Closes #5316 Needed for #3349 Needed for #4861 @TarantoolBot document Title: Document msgpack.object The following new methods were added to the Lua `msgpack` module: - `msgpack.object()`: Encodes an arbitrary Lua object given in its only argument in MsgPack and returns the encoded MsgPack data encapsulated in a MsgPack object. - `msgpack.object_from_raw()`: Creates a MsgPack object from raw data given either as a string or as a pointer and size. - `msgpack.is_object()`: Checks is the given argument is a MsgPack object. Example: ```lua local buffer = require('buffer') local msgpack = require('msgpack') -- Create an object from a Lua object. local mp = msgpack.object(123) local mp = msgpack.object("foobar") local mp = msgpack.object({1, 2, 3}) local mp = msgpack.object({foo = 1, bar = 2}) local mp = msgpack.object(box.tuple.new(1, 2, 3)) -- Create an object from raw data given as a string. local data = msgpack.encode({1, 2, 3}) local mp = msgpack.object_from_raw(data) -- Create an object from raw data given as ptr and size. local buf = buffer.ibuf() msgpack.encode({1, 2, 3}, buf) local mp = msgpack.object_from_raw(buf.buf, buf:size()) -- Check object. msgpack.is_object(mp) -- true msgpack.is_object({}) -- false ``` A MsgPack object has the following methods: - `mp:decode()`: Decodes MsgPack and returns a Lua object. - `mp:iterator()`: Returns an iterator over MsgPack data. A MsgPack iterator has the following methods: - `it:decode_array_header()`: Decodes a MsgPack array header under the cursor and returns the number of elements in the array. After calling this function the iterator points to the first element of the array or to the value following the array if the array is empty. Raises an error if the type of the value under the iterator cursor is not `MP_ARRAY`. - `it:decode_map_header()`: Decodes a MsgPack map header under the cursor and returns the number of key value pairs in the map. After calling this function the iterator points to the first key stored in the map or to the value following the map if the map is empty. Raises an error if the type of the value under the iterator cursor is not `MP_MAP`. - `it:decode()`: Decodes a MsgPack value under the iterator cursor and advances the cursor. Returns a Lua object corresponding to the MsgPack value. Raises a Lua error if there's no data to decode. - `it:take()`: Returns a MsgPack value under the iterator cursor as a MsgPack object (without decoding). Raises a Lua error if there's no data to decode. This function doesn't copy MsgPack data - instead it takes a reference to the original object. - `it:skip()`: Advances the iterator cursor by skipping one MsgPack value under the cursor. Returns nothing. Raises a Lua error if there's not data to skip. Usage example: ```lua local msgpack = require('msgpack') local mp = msgpack.object({foo = 123, bar = {1, 2, 3}}) local it = mp:iterator() it:decode_map_header() -- returns 2 it:decode() -- returns 'foo' it:decode() -- returns 123 it:skip() -- returns none, skips 'bar' local mp2 = it:take() mp2:decode() -- returns {1, 2, 3} ``` For more examples, see `test/app-luatest/msgpack_test.lua`. A MsgPack object can be passed to MsgPack encoder with the effect being the same as passing the original Lua object: ```lua local msgpack = require('msgpack') local mp = msgpack.object(123) msgpack.object({mp, mp}):decode() -- returns {123, 123} msgpack.decode(msgpack.encode({mp, mp})) -- returns {123, 123} ``` In particular, this means that if a MsgPack object stores an array, it can be inserted into a database space: ```lua box.space.my_space:insert(msgpack.object({1, 2, 3})) ```
yanshtunder
pushed a commit
that referenced
this issue
Feb 8, 2022
Closes #4861 @TarantoolBot document Title: Document return_raw net.box option If the `return_raw` flag is set, net.box will return response data wrapped in a msgpack object (see `msgpack.object()`) instead of decoding it to Lua. This can be useful when a response is supposed to be passed through without decoding or with partial decoding - creating a msgpack object can reduce pressure on the Lua garbage collector in this case. Example: ```lua local c = require('net.box').connect(uri) local mp = c.eval('eval ...', {1, 2, 3}, {return_raw = true}) mp:decode() -- {1, 2, 3} ``` The option is ignored for methods that return nil (`begin`, `commit`, `rollback`, `upsert`, `prepare` will return nil even if `return_raw` is set) and for `index.count` (it returns a number). For `execute`, the option is applied only to data (`rows`) while metadata is decoded irrespective of the value of `return_raw`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Raw idea (pointed by @karlovnv): we can wrap a return value using a special cdata type to let serializer extract a raw data from it. We need to look where GC may collect this cdata object to ensure that an access to a freed memory is not possible.
See also #4641 about C functions.
Extracted from #3349.
The text was updated successfully, but these errors were encountered: