Release 2.11.0 · tarantool/tarantool

Date: 2023-05-24

Tag: 2.11.0

Overview

2.11.0 is the first stable release in the long-term support (LTS)
version 2.11 release series.

The label "stable" means there are 2.11.x-based applications running in
production for quite a while without known crashes, incorrect results or
other showstopper bugs.

This release introduces 58 new features and resolves 33 bugs since
the 2.10.5 version. There can be bugs in less common areas. If you find any,
feel free to report an issue on GitHub.

Notable changes are:

Features for application developers
- Limitation of fiber execution time slice
- Linearizable isolation level
- Result set pagination
- Per-module logging
- Streaming in HTTP client
- Encoding HTTP query parameters
- Native foreign keys in the SQL engine
- Interactive debugger for Lua code
Enhancements for administration
- Replica join retry
- New bootstrap strategy
- Strict fencing in RAFT

Compatibility

Tarantool 2.11 is backward compatible with Tarantool 2.10 in the binary data
layout, client-server protocol, and replication protocol.

Please upgrade using the box.schema.upgrade() procedure to unlock
all the new features of the 2.x series.

Functionality added or changed

Core

The cpu_misses entry in fiber.top() output is deprecated (gh-5869).
Implemented fiber.top() for ARM64 (gh-4573).
Introduced the mechanism for catching fibers running without yielding for too
long. Now box operations, such as select and replace, will throw an error
if the fiber execution time since yield exceeds the limit. The limit can also
be checked from the application code with fiber.check_slice(). The default
limit is controlled by the new compat option fiber_slice_default. The old
default is no limit. The new default is one second. You can overwrite it with
fiber.set_slice() (gh-6085).
Now if a join fails with a non-critical error, such as ER_READONLY,
ER_ACCESS_DENIED, or a network-related error, the instance tries
to find a new master to join to and tries again (gh-6126).
Renamed replication states when a replica is joining. Now when querying
box.info.replication[id].upstream.status during join, you will
see either wait_snapshot or fetch_snapshot instead of
initial_join (gh-6126).
fiber_set_cancellable() C API function is deprecated and now does
nothing (gh-7166).
Introduced a new transaction isolation level linearizable. Transactions
started with box.begin{txn_isolation = "linearizable"} always see the latest
data confirmed by the quorum (gh-6707).
The box error C API (box_error_set(), box_error_last(), and so on) can now
be used in threads started by user modules with the pthread library (gh-7814).
box.info() can now be called before box.cfg() (gh-7255).
Introduced pagination support for memtx and vinyl tree indexes. It is now
possible to resume pairs and select from the position where the last
call stopped (gh-7639).
Now the log message contains the name of a Lua module from which the logging
function was called (gh-3211).
Now the log level can be set for specific modules using log.cfg{modules = {...}}
or box.cfg{log_modules = {...}} (gh-3211).
Reduced recovery time from a snapshot by up to 2x on the systems with a hard
disk drive (gh-8108).
Disabled automatic invocation of box.schema.upgrade on box.cfg for
read-write instances that don't set up replication. Now, box.schema.upgrade
may only be called manually by the admin (gh-8207).
Added the new function box.read_view.list() that returns a list of all
active database read views. The list includes both system read views (created
to make a checkpoint or join a replica) and read views created by application
code (available only in Enterprise Edition) (gh-8260).
The zstd version was updated to pre-1.5.5 (gh-8391).
Now the fiber_channel:close() closes the channel gracefully: closing it
for writing leaving the possibility to read existing events from it.
Previously, channel:close() was closing the channel completely and
discarding all unread events.

A new compat option fiber_channel_close_mode is added for switching to
the new behavior (gh-7746).

Vinyl

[Breaking change] Added support of transaction isolation levels for the
Vinyl engine. The txn_isolation option passed to box.begin() now has the
same effect for Vinyl and memtx. Note, this effectively changes the default
isolation level of Vinyl transactions from 'read-committed' to 'best-effort',
which may cause more conflicts (gh-5522).

Replication

Introduced the new configuration option bootstrap_strategy. The default
value of this option - "auto" - brings the new behavior of replica on:
- replica set bootstrap
- replica join to an existing replica set
- recovery
- replication reconfiguration.
To return to the old behavior, set the option to "legacy".

The new value "auto" will be in effect only when no value for
replication_connect_quorum is passed. If a value is present,
bootstrap_strategy is automatically set to "legacy" to preserve backward
compatibility.

Note that if you leave the options untouched (that is, bootstrap_strategy
defaults to "auto"), the following behavior will noticeably change: during
the recovery from local files and during replication reconfiguration
box.cfg{replication = ...} will not fail even if some (or all) of the remote
peers listed in box.cfg.replication are unavailable. Instead, the node will
try to connect to them for a period of replication_connect_timeout and then
transition to box.info.status == "running" as soon as it syncs with all the
reached peers (gh-5272).
[Breaking change] Joining a new replica to a working replica set.
If neither of the configuration options bootstrap_strategy and
replication_connect_quorum is passed explicitly, or if bootstrap_strategy
is set to "auto", bootstrapping a new replica in an existing replica set will
only succeed if all the replica set members are listed in the replica's
box.cfg.replication. For example, when joining a fresh replica to a replica
set of 3 nodes, all 3 node URIs must be present in the replica's
box.cfg.replication parameter. When joining 2 new replicas to a single
master, both replicas must have each other's URIs (alongside with master's
URI) in their box.cfg.replication (gh-5272).
A new compat option box_cfg_replication_sync_timeout was added to
control the default value of replication_sync_timeout option of box.cfg.
The old default is 300 seconds, and new default is 0. The noticeable difference
in the new behavior is that box.cfg{replication = ""} call now returns
before the node is synced with remote instances. If you need the node to be
writable once box.cfg returns, you can achieve it with new behavior by
calling box.ctl.wait_rw() after box.cfg{replication=...}.

By default, the option value is "old" and will be switched to "new" in the
next major release. To switch to the new behavior, set the option to "new"
before the initial box.cfg{} call in your application (gh-5272).

Raft

Introduced strict fencing, which tries its best to allow at most one
leader in cluster in any moment in time. This is achieved by setting
connection death timeout on the current leader to half the time compared to
followers (assuming the replication_timeout is the same on every replica)
(gh-7110).

LuaJIT

Backported patches from vanilla LuaJIT trunk (gh-7230). In the scope of this
activity, the following features is completed:

assert() now accepts any type of error object (from Lua 5.3).
Enabled LuaJIT CMake option LUAJIT_ENABLE_CHECKHOOK for checking instruction
and line hooks for compiled code (gh-7762).

Lua

Introduced the Tarantool compatibility module compat. It is used for
transparent management of behavior changes. compat stores options that
reflect behavior changes. Possible option values are old, new, and
default. By default, compat contains options for certain Tarantool
changes that may break compatibility with previous versions. Users can also
add their own compatibility options in runtime (gh-7000).
It is now possible to run scripts or load modules before the main script by
specifying them in the TT_PRELOAD environment variable. For example:
```
$ TT_PRELOAD=/path/to/foo.lua tarantool main.lua
```
(gh-7714).
Embedded the tarantool/checks module for function input validation
(gh-7726).
Introduced the -b and
-j flags that can be passed to
LuaJIT runtime for debugging and runtime configuration purposes (gh-5541).
Added the ability to override a built-in module by an external one (gh-7774).
Default module search paths now include the main script directory (gh-8182).

Console

Prevent mixing of background print and log output with the current user's
input in the interactive console (gh-7169).

Http client

Now the HTTP client is able to encode Lua objects automatically when it is
possible. The encoding format depends on the request content type.
The following content types are supported by default: application/json,
application/yaml, application/msgpack. Users can define encoding rules
for other content types by writing their own encoding functions (gh-6833).
Added a new method response:decode(). It decodes an HTTP response body to
a Lua object. The decoding result depends on the response content type. The
following content types are supported by default: application/json,
application/yaml, application/msgpack. Users can define decoding rules for
other content types by writing their own decoding functions (gh-6833).
Added a new option params to add query parameters to URI using a Lua table.
These parameters are encoded to a query string and passed to the HTTP request
(gh-6832).
Query parameters passed into a URI using the params option are now
percent-encoded automatically. Parameters are encoded with uri.QUERY_PART
when GET, DELETE, or HEAD HTTP methods are used, and with
uri.FORM_URLENCODED in other cases (gh-7931).
Introduced stream input/output interface for http.client (gh-7845).

Msgpack

Added the __index metamethod and the get method to msgpack.object.
They both perform indexation of MsgPack data stored in the object similar to
tuple from box.tuple: __index resolves collisions in favor of
msgpack.object methods, whereas get ignores methods (gh-7898).

Uri

Added a new method uri.values() for representing multivalue parameters (gh-6832).

> params = {q1 = uri.values("v1", "v2")}}
> uri.parse({"/tmp/unix.sock", params = params)
---
- host: unix/
  service: /tmp/unix.sock
  unix: /tmp/unix.sock
  params:
    q1:
    - v1
    - v2
...

Added functions uri.escape() and uri.unescape() for percent-encoding
and decoding URI parts (gh-3682).

SQL

Introduced a new function type SQL_EXPR (gh-6986).
SQL foreign key constraints and check constraints were replaced by tuple
constraints (gh-6986).
Introduced the new keyword SEQSCAN for SQL SELECT queries. You may now
use a scanning SQL SELECT query without the SEQSCAN keyword only if the
sql_seq_scan session setting is set to true. A new compat option
sql_seq_scan_default is added for managing the default value of
sql_seq_scan (gh-7747).

Box

Exported box_schema_version to public API and Lua via
box.info.schema_version (gh-7904).
Exported IPROTO constants and features to Lua (gh-7894).
It is now possible to register triggers for various recovery stages: use
box.ctl.on_recovery_state() before the initial box.cfg() call.
The trigger has one parameter – a string that shows the reached recovery stage:
snapshot_recovered, wal_recovered, indexes_built, or synced (gh-3159).
Exported current session identifier to C API via box_session_id (gh-7895).
Environment variables (TT_LOG, TT_LOG_LEVEL, and other) are taken into account
if logging is configured using log.cfg before box.cfg is called (gh-6011).
Added the IPROTO_AUTH_TYPE key to the IPROTO_ID response. The key contains
the name of the authentication method that is currently used on the server
for generating user authentication data (gh-7989).
Added an API for sending arbitrary IPROTO packets from server to client:
box.iproto.send in Lua and box_iproto_send in C (gh-7897).
It is now possible to set IPROTO request handler callbacks from Lua (using
box.iproto.override) and from C (using box_iproto_override) (gh-7901).
Introduced transaction options in box.atomic() by analogy with box.begin()
(gh-7202).
Added the support for downgrading system spaces to make them compatible with
older Tarantool versions (gh-7718).
The feedback daemon now collects metrics if the required version of the
metrics module is installed (gh-8192). feedback_version is updated
to 8.
Add fiber_set_ctx and fiber_get_ctx C API functions to pass data to fibers without
yielding immediately (gh-7669).

Console

Introduced a way to configure the printed End Of Stream (EOS) symbol in the
Lua console. Changed the default printed EOS from ';' to '' (gh-7031).

Debugger

Introduced a new console debugger luadebug.lua for debugging external and
builtin Lua modules.

Note: the debugger REPL is not yet compatible with Tarantool console.
This means that this code will hang in the console:
tarantool> dbg = require 'luadebug'
---
...

tarantool> dbg()
---
Users should call debugger activation only in their instrumented code, not
from the interactive console.

Introduced a new Lua API tarantool.debug.getsources() which allows
seeing sources of builtin modules in any external debugger.
Swapped 'up' and 'down' commands in the debugger to make them behave similar
to those in gdb and lldb.
Added the support for breakpoints to the builtin console debugger
luadebug.lua.
Made is easier to debug files with the same name (such as init.lua)
by handling partial path lookup in breakpoints:
```
break B/init.lua:10
break A/init.lua:20
break ./main.lua:30
break ../a/b/c/leaf.lua:40
```

Introduced the -d command-line option which runs the debugger console
instead of the standard interactive console (gh-7456).

$ tarantool -d debug-target.lua
Tarantool debugger 2.11.0-entrypoint-852-g9e6ed28ae
type 'help' for interactive help
luadebug: Loaded for 2.11.0-entrypoint-852-g9e6ed28ae
break via debug-target.lua => debug-target.lua:1 in chunk at debug-target.lua:0
   1 => local date = require 'datetime'
   4
luadebug>

This is a more convenient way to initiate a debugger session instead
of an older, more invasive approach of instrumenting the code with a
require 'luadebug'() call.

Doxygen

Doxygen module API documentation is now published automatically on every merge
to the master branch.

Yaml

Now yaml.encode can encode multiline strings in literal-scalar style for
better readability. A new compat option yaml_pretty_multiline is added
for switching to the new behavior (gh-3012).

Bugs fixed

Core

Added the fiber_join_timeout symbol to exports (gh-7125).
Allowed spurious wakeup of a fiber that is waiting for WAL write completion
(gh-6506).
Now box_latch_lock guarantees the order in which it is acquired by
fibers requesting it (gh-7166).
A proper error in now raised on incorrect syslog log configuration through
log.cfg (gh-7447).
Fixed a bug with Tarantool C API freeing fiber region allocations it does not
allocate. This could lead to use-after-free in client code which allocates
memory on fiber region. (gh-5665).
Errors thrown on specifying invalid index parts or format fields are now more
verbose. They include the bad index part or field number (gh-7933).
Fixed a potential crash when SIGTERM was received before box.cfg
execution is completed (gh-7743).
Fixed a bug when direct assignments of box.cfg parameters (such as
box.cfg.background = true) were silently ignored. Now such assignments
result in errors. The correct way to set box.cfg parameters is this:
box.cfg{ background=true } (gh-7350).
Fixed various bugs related to unsafe (i.e., coming from an unknown source)
decoding and validating of MsgPack extensions (ghs-73).

Memtx

Deprecated the GT iterator type for HASH indexes (gh-7231).

Replication

Fixed replicaset bootstrap getting stuck on some nodes with ER_READONLY when
there are connectivity problems (gh-7737).
Fixed a bug when a replicaset state machine that tracks the number of
appliers in different states could become inconsistent during
reconfiguration (gh-7590).
Fixed a bug related to box.info.replication[...].upstream being stuck in the "connecting"
state for several minutes after a replica DNS record change (gh-7294).

Raft

Fixed a bug when a replicaset could be split into parts if an old leader
started a new synchronous txn shortly before a new leader was going to be
elected. This was possible if the old leader hadn't learned the new term yet
(gh-7253).

Lua

Fixed a bug that prevented using C module API methods luaL_iscallable(),
luaL_checkcdata(), and luaL_setcdatagc() with the upvalue indexes
(gh-8249).

Json

A new compat option json_escape_forward_slash was added. This option
configures whether the internal JSON encoder escapes the forward slash
character (old behavior) or not (new behavior). This option affects the
json.encode() Lua function and the JSON logger (gh-6200).

SQL

Now SQL queries with subqueries and quoted names return the correct
column names in projection (gh-7063).
Now the ROUND() functions works correctly on arguments with big precision
(gh-6650).
Fixed assertion in the % (modulo) operation when the left value is negative
and the result is 0 (gh-6575).
Now NaN is always considered NULL (gh-6572).
[Breaking change in the SQL engine] Dropped the session setting
sql_defer_foreign_keys and rules reference trigger action,
constraint check time, and match type (gh-6986).

Box

Fixed an incorrect error message on granting privileges to the admin user.
Such attempts now fail with proper error messages such as "User 'admin'
already has read access on universe" (gh-7226).
Fixed accidental exposure of the _collation space to public (ghs-5).
Now box.execute cannot be called before box.cfg (gh-4726).
Fixed the error message for wrong options provided to a function definition
(gh-7972).
Fixed a bug where box.cfg.force_recovery doesn't work when there are
no user spaces in a snapshot (gh-7974).

LuaJIT

The JIT engine was disabled by default on macOS platforms to improve
the user experience. If necessary, you can enable it with jit.on (gh-8252).

Console

Fixed console ignoring -i flag in case stdin is not a tty (gh-5064).

Lib

Fixed http.client to properly parse HTTP status header such as HTTP/2 200
when the HTTP version does not have a minor part (gh-7319).

Tarantoolctl

Added a description for the --format flag of the tarantoolctl cat command.

URI

Optimized addition of URI parameters and their values (gh-7155).

Build

Fixed libunwind.h search in testing files (gh-6877).

Http client

Fixed a bug where a response body cannot be decoded (gh-8363).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

2.11.0

Overview

Compatibility

Functionality added or changed

Core

Vinyl

Replication

Raft

LuaJIT

Lua

Console

Http client

Msgpack

Uri

SQL

Box

Console

Debugger

Doxygen

Yaml

Bugs fixed

Core

Memtx

Replication

Raft

Lua

Json

SQL

Box

LuaJIT

Console

Lib

Tarantoolctl

URI

Build

Http client