WL#9384: Prepare each DML statement once

Alter SQL DML processing so that each statement is prepared once.
This affects prepared statements and stored routine statements, which
before have been prepared for every execution.

Lifecycle management of Item objects is changed as follows:
 - Item::fix_fields() is never called more than once.
 - Item_cleanup() is called after each preparation and execution
   to clean out resources allocated on the runtime memory context.
 - Item::bind_fields() must be called before every execution of preparable
   statements to "bind" Item_field and related objects with execution
   resources such as tables (TABLE objects).
 - Item destructor must be called to free any resources allocated during
   preparation.

Temporary tables are refactored so that they can persist for the whole
lifecycle of preparable statements. They are changed as follows:
 - create_tmp_table() allocates a TABLE, TABLE_SHARE and TABLE_LIST object
   for a temporary table. Notice that multiple TABLE objects may be created
   later for the same temporary table.
 - close_tmp_table() closes the temporary table, deletes the contents of
   the temporary table if this is the last open handle to a temporary table,
   and closes the storage engine handler.
   However, the basic metadata associated with the temporary table are not
   removed.
 - instantiate_tmp_table() opens a storage handler for a closed temporary
   table and creates the contents of the temporary table.
 - free_tmp_table() releases metadata for a temporary table.
   For tables with multiple TABLE objects, the last call releases
   all metadata objects and the mem_root the table is created on.

The resolved type of dynamic parameters is now derived from the context,
ie. the function in which the parameter is used and other expressions
passed to the function. E.g, if we have an expression "5.67 + ?", the
resolved type of the parameter is decimal, like the type of the
first argument for the addition function. Some parameters will have
a fixed type, like arguments to the concatenation operator that
are always character (or binary) string.
We are still able to accept almost all types of actual parameter
arguments and provide the same semantics as before. Some of this
is done by performing an implicit reprepare. E.g in the addition
example above where the resolved type of parameter is decimal and an
actual argument of type floating point is supplied, the query is
implicitly reprepared with type of parameters converted to double.

Another important change for dynamic parameters, and for some other
expressions that are constant for the duration of a statement
execution, is to set the flag INNER_TABLE_BIT for those expressions
instead of having them being fully constant as they used to be.
Some of the other expressions are system variables, user variables
that are not assigned to during execution and current timestamp.

It is no longer possible to use dynamic parameters in the ORDER BY
clause to select column numbers.

Since there is only one preparation, we no longer have to "rollback"
item allocations during preparation. In particular Item::transform()
is adjusted to comply with this.

Lifecycle of cursor objects is changed so that cursors are attached
to the preparable statements they are associated with. Temporary tables
used with cursors are implemented on dedicated mem_roots. They cannot
use the execution mem_root since they may be accessed from multiple
statements, and using the mem_root of the preparable statement might
cause infinite memory allocation.

Query_result objects for data change statements have been significantly
refactored to clarify what happens in the functions prepare(),
optimize() and start_execution().

UDF functions are handled specially: Due to the INIT function possibly
mixing resolving and resource allocation, we were forced to implement
a forced preparation at the start of each execution of a preparable
statement that references UDF functions.

Previously, all privilege checking was performed during preparation of
a statement. Since privileges are dynamic, we have implemented
privilege checking at the start of each statement execution, through
the virtual function Sql_cmd_dml::check_privileges().
Privilege checking is still performed during preparation, implemented
in the virtual function Sql_cmd_dml::precheck().

For more details, please see the worklog description.

Change-Id: I8801cafef236a307ca293a961bd3ff79d3eefdee

client/mysql.cc

@@ -1,5 +1,5 @@
 /*
-   Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
+   Copyright (c) 2000, 2020, Oracle and/or its affiliates.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License, version 2.0,
@@ -3433,66 +3433,7 @@ static int com_ego(String *buffer, char *line) {
   return result;
 }
 
-static const char *fieldtype2str(enum enum_field_types type) {
-  switch (type) {
-    case MYSQL_TYPE_BIT:
-      return "BIT";
-    case MYSQL_TYPE_BLOB:
-      return "BLOB";
-    case MYSQL_TYPE_DATE:
-      return "DATE";
-    case MYSQL_TYPE_DATETIME:
-      return "DATETIME";
-    case MYSQL_TYPE_NEWDECIMAL:
-      return "NEWDECIMAL";
-    case MYSQL_TYPE_DECIMAL:
-      return "DECIMAL";
-    case MYSQL_TYPE_DOUBLE:
-      return "DOUBLE";
-    case MYSQL_TYPE_ENUM:
-      return "ENUM";
-    case MYSQL_TYPE_FLOAT:
-      return "FLOAT";
-    case MYSQL_TYPE_GEOMETRY:
-      return "GEOMETRY";
-    case MYSQL_TYPE_INT24:
-      return "INT24";
-    case MYSQL_TYPE_JSON:
-      return "JSON";
-    case MYSQL_TYPE_LONG:
-      return "LONG";
-    case MYSQL_TYPE_LONGLONG:
-      return "LONGLONG";
-    case MYSQL_TYPE_LONG_BLOB:
-      return "LONG_BLOB";
-    case MYSQL_TYPE_MEDIUM_BLOB:
-      return "MEDIUM_BLOB";
-    case MYSQL_TYPE_NEWDATE:
-      return "NEWDATE";
-    case MYSQL_TYPE_NULL:
-      return "NULL";
-    case MYSQL_TYPE_SET:
-      return "SET";
-    case MYSQL_TYPE_SHORT:
-      return "SHORT";
-    case MYSQL_TYPE_STRING:
-      return "STRING";
-    case MYSQL_TYPE_TIME:
-      return "TIME";
-    case MYSQL_TYPE_TIMESTAMP:
-      return "TIMESTAMP";
-    case MYSQL_TYPE_TINY:
-      return "TINY";
-    case MYSQL_TYPE_TINY_BLOB:
-      return "TINY_BLOB";
-    case MYSQL_TYPE_VAR_STRING:
-      return "VAR_STRING";
-    case MYSQL_TYPE_YEAR:
-      return "YEAR";
-    default:
-      return "?-unknown-?";
-  }
-}
+const char *fieldtype2str(enum enum_field_types type);
 
 static char *fieldflags2str(uint f) {
   static char buf[1024];

client/mysqltest.cc

@@ -1,4 +1,4 @@
-// Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
+// Copyright (c) 2000, 2020, Oracle and/or its affiliates.
 //
 // This program is free software; you can redistribute it and/or modify
 // it under the terms of the GNU General Public License, version 2.0,
@@ -8365,11 +8365,6 @@ static void run_query_stmt(MYSQL *mysql, struct st_command *command,
 
         // Free normal result set with meta data
         mysql_free_result_wrapper(res);
-
-        // Clear prepare warnings if there are execute warnings,
-        // since they are probably duplicated.
-        if (ds_execute_warnings.length || mysql->warning_count)
-          dynstr_set(&ds_prepare_warnings, nullptr);
       } else {
         // This is a query without resultset
       }

include/field_types.h

@@ -1,4 +1,4 @@
-/* Copyright (c) 2014, 2019, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2014, 2020, Oracle and/or its affiliates.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License, version 2.0,
@@ -75,6 +75,8 @@ enum enum_field_types
   MYSQL_TYPE_DATETIME2,   /**< Internal to MySQL. Not used in protocol */
   MYSQL_TYPE_TIME2,       /**< Internal to MySQL. Not used in protocol */
   MYSQL_TYPE_TYPED_ARRAY, /**< Used for replication only */
+  MYSQL_TYPE_INVALID = 243,
+  MYSQL_TYPE_BOOL = 244, /**< Currently just a placeholder */
   MYSQL_TYPE_JSON = 245,
   MYSQL_TYPE_NEWDECIMAL = 246,
   MYSQL_TYPE_ENUM = 247,

include/my_inttypes.h

@@ -1,5 +1,5 @@
 /*
-   Copyright (c) 2016, 2019, Oracle and/or its affiliates. All rights reserved.
+   Copyright (c) 2016, 2020, Oracle and/or its affiliates.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License, version 2.0,

include/mysql.h.pp

@@ -23,6 +23,8 @@
   MYSQL_TYPE_DATETIME2,
   MYSQL_TYPE_TIME2,
   MYSQL_TYPE_TYPED_ARRAY,
+  MYSQL_TYPE_INVALID = 243,
+  MYSQL_TYPE_BOOL = 244,
   MYSQL_TYPE_JSON = 245,
   MYSQL_TYPE_NEWDECIMAL = 246,
   MYSQL_TYPE_ENUM = 247,
@@ -134,7 +136,7 @@
 };
 struct Vio;
 typedef struct NET {
-  struct Vio * vio;
+  struct Vio *vio;
   unsigned char *buff, *buff_end, *write_pos, *read_pos;
   my_socket fd;
   unsigned long remain_in_buf, length, buf_length, where_b;
@@ -184,7 +186,7 @@
   SESSION_TRACK_TRANSACTION_CHARACTERISTICS,
   SESSION_TRACK_TRANSACTION_STATE
 };
-bool my_net_init(struct NET *net, struct Vio * vio);
+bool my_net_init(struct NET *net, struct Vio *vio);
 void my_net_local_init(struct NET *net);
 void net_end(struct NET *net);
 void net_clear(struct NET *net, bool check_buffer);
@@ -274,7 +276,7 @@
 void my_thread_end(void);
 unsigned long net_field_length(unsigned char **packet);
 unsigned long net_field_length_checked(unsigned char **packet,
-                                               unsigned long max_length);
+                                       unsigned long max_length);
 uint64_t net_field_length_ll(unsigned char **packet);
 unsigned char *net_store_length(unsigned char *pkg, unsigned long long length);
 unsigned int net_length_size(unsigned long long num);
@@ -579,26 +581,26 @@
 bool mysql_eof(MYSQL_RES *res);
 MYSQL_FIELD * mysql_fetch_field_direct(MYSQL_RES *res,
                                               unsigned int fieldnr);
-MYSQL_FIELD * mysql_fetch_fields(MYSQL_RES *res);
+MYSQL_FIELD *mysql_fetch_fields(MYSQL_RES *res);
 MYSQL_ROW_OFFSET mysql_row_tell(MYSQL_RES *res);
 MYSQL_FIELD_OFFSET mysql_field_tell(MYSQL_RES *res);
 enum enum_resultset_metadata mysql_result_metadata(MYSQL_RES *result);
 unsigned int mysql_field_count(MYSQL *mysql);
 uint64_t mysql_affected_rows(MYSQL *mysql);
 uint64_t mysql_insert_id(MYSQL *mysql);
 unsigned int mysql_errno(MYSQL *mysql);
-const char * mysql_error(MYSQL *mysql);
-const char * mysql_sqlstate(MYSQL *mysql);
+const char *mysql_error(MYSQL *mysql);
+const char *mysql_sqlstate(MYSQL *mysql);
 unsigned int mysql_warning_count(MYSQL *mysql);
-const char * mysql_info(MYSQL *mysql);
+const char *mysql_info(MYSQL *mysql);
 unsigned long mysql_thread_id(MYSQL *mysql);
-const char * mysql_character_set_name(MYSQL *mysql);
+const char *mysql_character_set_name(MYSQL *mysql);
 int mysql_set_character_set(MYSQL *mysql, const char *csname);
-MYSQL * mysql_init(MYSQL *mysql);
+MYSQL *mysql_init(MYSQL *mysql);
 bool mysql_ssl_set(MYSQL *mysql, const char *key, const char *cert,
                            const char *ca, const char *capath,
                            const char *cipher);
-const char * mysql_get_ssl_cipher(MYSQL *mysql);
+const char *mysql_get_ssl_cipher(MYSQL *mysql);
 bool mysql_change_user(MYSQL *mysql, const char *user,
                                const char *passwd, const char *db);
 MYSQL * mysql_real_connect(MYSQL *mysql, const char *host,
@@ -610,8 +612,8 @@
 int mysql_query(MYSQL *mysql, const char *q);
 int mysql_send_query(MYSQL *mysql, const char *q, unsigned long length);
 int mysql_real_query(MYSQL *mysql, const char *q, unsigned long length);
-MYSQL_RES * mysql_store_result(MYSQL *mysql);
-MYSQL_RES * mysql_use_result(MYSQL *mysql);
+MYSQL_RES *mysql_store_result(MYSQL *mysql);
+MYSQL_RES *mysql_use_result(MYSQL *mysql);
 enum net_async_status mysql_real_connect_nonblocking(
     MYSQL *mysql, const char *host, const char *user, const char *passwd,
     const char *db, unsigned int port, const char *unix_socket,
@@ -629,11 +631,11 @@
 void mysql_get_character_set_info(MYSQL *mysql,
                                           MY_CHARSET_INFO *charset);
 int mysql_session_track_get_first(MYSQL *mysql,
-                                          enum enum_session_state_type type,
-                                          const char **data, size_t *length);
+                                  enum enum_session_state_type type,
+                                  const char **data, size_t *length);
 int mysql_session_track_get_next(MYSQL *mysql,
-                                         enum enum_session_state_type type,
-                                         const char **data, size_t *length);
+                                 enum enum_session_state_type type,
+                                 const char **data, size_t *length);
 void mysql_set_local_infile_handler(
     MYSQL *mysql, int (*local_infile_init)(void **, const char *, void *),
     int (*local_infile_read)(void *, char *, unsigned int),
@@ -648,18 +650,18 @@
 int mysql_set_server_option(MYSQL *mysql,
                                     enum enum_mysql_set_option option);
 int mysql_ping(MYSQL *mysql);
-const char * mysql_stat(MYSQL *mysql);
-const char * mysql_get_server_info(MYSQL *mysql);
-const char * mysql_get_client_info(void);
+const char *mysql_stat(MYSQL *mysql);
+const char *mysql_get_server_info(MYSQL *mysql);
+const char *mysql_get_client_info(void);
 unsigned long mysql_get_client_version(void);
-const char * mysql_get_host_info(MYSQL *mysql);
+const char *mysql_get_host_info(MYSQL *mysql);
 unsigned long mysql_get_server_version(MYSQL *mysql);
 unsigned int mysql_get_proto_info(MYSQL *mysql);
-MYSQL_RES * mysql_list_dbs(MYSQL *mysql, const char *wild);
-MYSQL_RES * mysql_list_tables(MYSQL *mysql, const char *wild);
-MYSQL_RES * mysql_list_processes(MYSQL *mysql);
+MYSQL_RES *mysql_list_dbs(MYSQL *mysql, const char *wild);
+MYSQL_RES *mysql_list_tables(MYSQL *mysql, const char *wild);
+MYSQL_RES *mysql_list_processes(MYSQL *mysql);
 int mysql_options(MYSQL *mysql, enum mysql_option option,
-                          const void *arg);
+                           const void *arg);
 int mysql_options4(MYSQL *mysql, enum mysql_option option,
                            const void *arg1, const void *arg2);
 int mysql_get_option(MYSQL *mysql, enum mysql_option option,
@@ -673,15 +675,15 @@
                                             MYSQL_FIELD_OFFSET offset);
 MYSQL_ROW mysql_fetch_row(MYSQL_RES *result);
 enum net_async_status mysql_fetch_row_nonblocking(MYSQL_RES *res,
-                                                          MYSQL_ROW *row);
-unsigned long * mysql_fetch_lengths(MYSQL_RES *result);
-MYSQL_FIELD * mysql_fetch_field(MYSQL_RES *result);
+                                                  MYSQL_ROW *row);
+unsigned long *mysql_fetch_lengths(MYSQL_RES *result);
+MYSQL_FIELD *mysql_fetch_field(MYSQL_RES *result);
 MYSQL_RES * mysql_list_fields(MYSQL *mysql, const char *table,
                                      const char *wild);
 unsigned long mysql_escape_string(char *to, const char *from,
-                                          unsigned long from_length);
+                                  unsigned long from_length);
 unsigned long mysql_hex_string(char *to, const char *from,
-                                       unsigned long from_length);
+                               unsigned long from_length);
 unsigned long mysql_real_escape_string(MYSQL *mysql, char *to,
                                                const char *from,
                                                unsigned long length);
@@ -759,13 +761,13 @@
   STMT_ATTR_CURSOR_TYPE,
   STMT_ATTR_PREFETCH_ROWS
 };
-MYSQL_STMT * mysql_stmt_init(MYSQL *mysql);
+MYSQL_STMT *mysql_stmt_init(MYSQL *mysql);
 int mysql_stmt_prepare(MYSQL_STMT *stmt, const char *query,
-                               unsigned long length);
+                       unsigned long length);
 int mysql_stmt_execute(MYSQL_STMT *stmt);
 int mysql_stmt_fetch(MYSQL_STMT *stmt);
 int mysql_stmt_fetch_column(MYSQL_STMT *stmt, MYSQL_BIND *bind_arg,
-                                    unsigned int column, unsigned long offset);
+                            unsigned int column, unsigned long offset);
 int mysql_stmt_store_result(MYSQL_STMT *stmt);
 unsigned long mysql_stmt_param_count(MYSQL_STMT *stmt);
 bool mysql_stmt_attr_set(MYSQL_STMT *stmt,
@@ -782,11 +784,11 @@
 bool mysql_stmt_send_long_data(MYSQL_STMT *stmt,
                                        unsigned int param_number,
                                        const char *data, unsigned long length);
-MYSQL_RES * mysql_stmt_result_metadata(MYSQL_STMT *stmt);
-MYSQL_RES * mysql_stmt_param_metadata(MYSQL_STMT *stmt);
+MYSQL_RES *mysql_stmt_result_metadata(MYSQL_STMT *stmt);
+MYSQL_RES *mysql_stmt_param_metadata(MYSQL_STMT *stmt);
 unsigned int mysql_stmt_errno(MYSQL_STMT *stmt);
-const char * mysql_stmt_error(MYSQL_STMT *stmt);
-const char * mysql_stmt_sqlstate(MYSQL_STMT *stmt);
+const char *mysql_stmt_error(MYSQL_STMT *stmt);
+const char *mysql_stmt_sqlstate(MYSQL_STMT *stmt);
 MYSQL_ROW_OFFSET mysql_stmt_row_seek(MYSQL_STMT *stmt,
                                              MYSQL_ROW_OFFSET offset);
 MYSQL_ROW_OFFSET mysql_stmt_row_tell(MYSQL_STMT *stmt);

include/mysql/psi/mysql_file.h

@@ -1213,7 +1213,7 @@ static inline int inline_mysql_file_rename(
   PSI_file_locker_state state;
   locker = PSI_FILE_CALL(get_thread_file_name_locker)(
       &state, key, PSI_FILE_RENAME, from, &locker);
-  if (likely(locker != NULL)) {
+  if (likely(locker != nullptr)) {
     PSI_FILE_CALL(start_file_rename_wait)
     (locker, (size_t)0, from, to, src_file, src_line);
     result = my_rename(from, to, flags);

include/mysql_com.h

@@ -840,13 +840,25 @@ struct Vio;
 #define MYSQL_VIO struct Vio *
 #endif
 
-#define MAX_TINYINT_WIDTH 3     /**< Max width for a TINY w.o. sign */
-#define MAX_SMALLINT_WIDTH 5    /**< Max width for a SHORT w.o. sign */
-#define MAX_MEDIUMINT_WIDTH 8   /**< Max width for a INT24 w.o. sign */
-#define MAX_INT_WIDTH 10        /**< Max width for a LONG w.o. sign */
-#define MAX_BIGINT_WIDTH 20     /**< Max width for a LONGLONG */
-#define MAX_CHAR_WIDTH 255      /**< Max length for a CHAR colum */
-#define MAX_BLOB_WIDTH 16777216 /**< Default width for blob */
+#define MAX_TINYINT_WIDTH 3   /**< Max width for a TINY w.o. sign */
+#define MAX_SMALLINT_WIDTH 5  /**< Max width for a SHORT w.o. sign */
+#define MAX_MEDIUMINT_WIDTH 8 /**< Max width for a INT24 w.o. sign */
+#define MAX_INT_WIDTH 10      /**< Max width for a LONG w.o. sign */
+#define MAX_BIGINT_WIDTH 20   /**< Max width for a LONGLONG */
+/// Max width for a CHAR column, in number of characters
+#define MAX_CHAR_WIDTH 255U
+/// Max width for a VARCHAR column, in number of bytes
+#define MAX_VARCHAR_WIDTH 65535U
+/// Max width for a tiny blob, in bytes
+#define MAX_TINY_BLOB_WIDTH 255U
+/// Max width for a short blob (aka BLOB), in bytes
+#define MAX_SHORT_BLOB_WIDTH 65535U
+/// Max width for a medium blob, in bytes
+#define MAX_MEDIUM_BLOB_WIDTH 16777215U
+/// Max width for a long blob, in bytes
+#define MAX_LONG_BLOB_WIDTH 4294967295U
+/// Default width for blob  @todo - replace this with the specific types above?
+#define MAX_BLOB_WIDTH 16777216
 
 typedef struct NET {
   MYSQL_VIO vio;

libmysql/libmysql.cc

@@ -1,4 +1,4 @@
-/* Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.
+/* Copyright (c) 2000, 2020, Oracle and/or its affiliates.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License, version 2.0,
@@ -3453,6 +3453,7 @@ static void fetch_result_with_conversion(MYSQL_BIND *param, MYSQL_FIELD *field,
   uint field_is_unsigned = field->flags & UNSIGNED_FLAG;
 
   switch (field_type) {
+    case MYSQL_TYPE_BOOL:
     case MYSQL_TYPE_TINY: {
       uchar value = **row;
       /* sic: we need to cast to 'signed char' as 'char' may be unsigned */

mysql-test/include/mix1.inc

@@ -486,20 +486,16 @@ primary key (id),
 unique key (c)
 ) engine=innodb;
 
---disable_ps_protocol # Different number of warnings until WL#6570.
 insert into t1 (id, c) values
 (NULL, 'a'),
 (NULL, 'a')
 on duplicate key update id = values(id), counter = counter + 1;
---enable_ps_protocol
 
 select * from t1;
 
---disable_ps_protocol # Different number of warnings until WL#6570.
 insert into t1 (id, c) values
 (NULL, 'b')
 on duplicate key update id = values(id), counter = counter + 1;
---enable_ps_protocol
 
 select * from t1;
 
@@ -509,17 +505,13 @@ insert into t1 (id, c) values (NULL, 'a');
 
 select * from t1;
 
---disable_ps_protocol # Different number of warnings until WL#6570.
 insert into t1 (id, c) values (NULL, 'b'), (NULL, 'b')
 on duplicate key update id = values(id), c = values(c), counter = counter + 1;
---enable_ps_protocol
 
 select * from t1;
 
---disable_ps_protocol # Different number of warnings until WL#6570.
 insert into t1 (id, c) values (NULL, 'a')
 on duplicate key update id = values(id), c = values(c), counter = counter + 1;
---enable_ps_protocol
 
 select * from t1;