Fix all warnings given by UBSAN

The easiest way to compile and test the server with UBSAN is to run:
./BUILD/compile-pentium64-ubsan
and then run mysql-test-run.
After this commit, one should be able to run this without any UBSAN
warnings. There is still a few compiler warnings that should be fixed
at some point, but these do not expose any real bugs.

The 'special' cases where we disable, suppress or circumvent UBSAN are:
- ref10 source (as here we intentionally do some shifts that UBSAN
  complains about.
- x86 version of optimized int#korr() methods. UBSAN do not like unaligned
  memory access of integers.  Fixed by using byte_order_generic.h when
  compiling with UBSAN
- We use smaller thread stack with ASAN and UBSAN, which forced me to
  disable a few tests that prints the thread stack size.
- Verifying class types does not work for shared libraries. I added
  suppression in mysql-test-run.pl for this case.
- Added '#ifdef WITH_UBSAN' when using integer arithmetic where it is
  safe to have overflows (two cases, in item_func.cc).

Things fixed:
- Don't left shift signed values
  (byte_order_generic.h, mysqltest.c, item_sum.cc and many more)
- Don't assign not non existing values to enum variables.
- Ensure that bool and enum values are properly initialized in
  constructors.  This was needed as UBSAN checks that these types has
  correct values when one copies an object.
  (gcalc_tools.h, ha_partition.cc, item_sum.cc, partition_element.h ...)
- Ensure we do not called handler functions on unallocated objects or
  deleted objects.
  (events.cc, sql_acl.cc).
- Fixed bugs in Item_sp::Item_sp() where we did not call constructor
  on Query_arena object.
- Fixed several cast of objects to an incompatible class!
  (Item.cc, Item_buff.cc, item_timefunc.cc, opt_subselect.cc, sql_acl.cc,
   sql_select.cc ...)
- Ensure we do not do integer arithmetic that causes over or underflows.
  This includes also ++ and -- of integers.
  (Item_func.cc, Item_strfunc.cc, item_timefunc.cc, sql_base.cc ...)
- Added JSON_VALUE_UNITIALIZED to json_value_types and ensure that
  value_type is initialized to this instead of to -1, which is not a valid
  enum value for json_value_types.
- Ensure we do not call memcpy() when second argument could be null.
- Fixed that Item_func_str::make_empty_result() creates an empty string
  instead of a null string (safer as it ensures we do not do arithmetic
  on null strings).

Other things:

- Changed struct st_position to an OBJECT and added an initialization
  function to it to ensure that we do not copy or use uninitialized
  members. The change to a class was also motived that we used "struct
  st_position" and POSITION randomly trough the code which was
  confusing.
- Notably big rewrite in sql_acl.cc to avoid using deleted objects.
- Changed in sql_partition to use '^' instead of '-'. This is safe as
  the operator is either 0 or 0x8000000000000000ULL.
- Added check for select_nr < INT_MAX in JOIN::build_explain() to
  avoid bug when get_select() could return NULL.
- Reordered elements in POSITION for better alignment.
- Changed sql_test.cc::print_plan() to use pointers instead of objects.
- Fixed bug in find_set() where could could execute '1 << -1'.
- Added variable have_sanitizer, used by mtr.  (This variable was before
  only in 10.5 and up).  It can now have one of two values:
  ASAN or UBSAN.
- Moved ~Archive_share() from ha_archive.cc to ha_archive.h and marked
  it virtual. This was an effort to get UBSAN to work with loaded storage
  engines. I kept the change as the new place is better.
- Added in CONNECT engine COLBLK::SetName(), to get around a wrong cast
  in tabutil.cpp.
- Added HAVE_REPLICATION around usage of rgi_slave, to get embedded
  server to compile with UBSAN. (Patch from Marko).
- Added #ifdef for powerpc64 to avoid a bug in old gcc versions related
  to integer arithmetic.

Changes that should not be needed but had to be done to suppress warnings
from UBSAN:

- Added static_cast<<uint16_t>> around shift to get rid of a LOT of
  compiler warnings when using UBSAN.
- Had to change some '/' of 2 base integers to shift to get rid of
  some compile time warnings.

Reviewed by:
- Json changes: Alexey Botchkov
- Charset changes in ctype-uca.c: Alexander Barkov
- InnoDB changes & Embedded server: Marko Mäkelä
- sql_acl.cc changes: Vicențiu Ciorbaru
- build_explain() changes: Sergey Petrunia

BUILD/SETUP.sh

@@ -141,7 +141,7 @@ elif [ "x$warning_mode" = "xmaintainer" ]; then
   debug_extra_cflags="-g3"
 else
 # Both C and C++ warnings
-  warnings="-Wall -Wextra -Wunused -Wwrite-strings -Wno-uninitialized -Wno-strict-aliasing -Wimplicit-fallthrough=2"
+  warnings="-Wall -Wextra -Wunused -Wwrite-strings -Wno-uninitialized -Wno-strict-aliasing -Wimplicit-fallthrough=2 -Wformat-security -Wvla"
 
 # For more warnings, uncomment the following line
 # warnings="$warnings -Wshadow"

BUILD/compile-pentium64-ubsan

@@ -0,0 +1,29 @@
+#! /bin/sh
+# Copyright (c) 2018, MariaDB Corporation.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1335  USA
+
+# Compilation with UBSAN, The Undefined Behavior Sanitizer
+# We have to use -Wno-uninitialized and -Wno-unitialized we get a lot of false
+# positive warnings for this when compiling with -fsanitize=undefined.
+# We also have to compile without Spider as linking with Spider library does
+# not work. (errno: 11, undefined symbol: _ZTI12ha_partition)
+
+path=`dirname $0`
+. "$path/SETUP.sh"
+
+extra_flags="$pentium64_cflags $debug_cflags -fsanitize=undefined -DWITH_UBSAN -Wno-conversion -Wno-uninitialized"
+extra_configs="$pentium_configs $debug_configs -DWITH_UBSAN=ON -DMYSQL_MAINTAINER_MODE=NO --without-spider --without-tokudb"
+
+. "$path/FINISH.sh"

client/mysqltest.cc

@@ -10905,7 +10905,7 @@ int get_next_bit(REP_SET *set,uint lastpos)
 
   start=set->bits+ ((lastpos+1) / WORD_BIT);
   end=set->bits + set->size_of_bits;
-  bits=start[0] & ~((1 << ((lastpos+1) % WORD_BIT)) -1);
+  bits=start[0] & ~((1U << ((lastpos+1) % WORD_BIT)) -1);
 
   while (! bits && ++start < end)
     bits=start[0];

cmake/configure.pl

@@ -93,6 +93,11 @@ sub check_compiler
   {
     $option = substr($option, 2);
   }
+  elsif (substr ($option, 0, 2) eq "-D")
+  {
+    # Must be cmake config option
+    $option = substr($option, 1);
+  }
   else
   {
     # This must be environment variable
@@ -119,6 +124,11 @@ sub check_compiler
     $just_print=1;
     next;
   }
+  if ($option =~ /D.*=/)
+  {
+    $cmakeargs = $cmakeargs." -".$option;
+     next;
+  }
   if($option =~ /with-plugins=/)
   {
     my @plugins= split(/,/, substr($option,13));

include/byte_order_generic.h

@@ -28,10 +28,10 @@
 				  (((uint32) (uchar) (A)[2]) << 16) |\
 				  (((uint32) (uchar) (A)[1]) << 8) | \
 				  ((uint32) (uchar) (A)[0])))
-#define sint4korr(A)	(int32) (((int32) ((uchar) (A)[0])) |\
-				(((int32) ((uchar) (A)[1]) << 8)) |\
-				(((int32) ((uchar) (A)[2]) << 16)) |\
-				(((int32) ((int16) (A)[3]) << 24)))
+#define sint4korr(A)	(int32) (((uint32) ((uchar) (A)[0])) |\
+				(((uint32) ((uchar) (A)[1]) << 8)) |\
+				(((uint32) ((uchar) (A)[2]) << 16)) |\
+				(((uint32) ((uchar) (A)[3]) << 24)))
 #define sint8korr(A)	(longlong) uint8korr(A)
 #define uint2korr(A)	(uint16) (((uint16) ((uchar) (A)[0])) |\
 				  ((uint16) ((uchar) (A)[1]) << 8))

include/byte_order_generic_x86.h

@@ -17,6 +17,7 @@
 /*
   Optimized function-like macros for the x86 architecture (_WIN32 included).
 */
+
 #define sint2korr(A)	(*((const int16 *) (A)))
 #define sint3korr(A)	((int32) ((((uchar) (A)[2]) & 128) ? \
 				  (((uint32) 255L << 24) | \

include/json_lib.h

@@ -172,6 +172,7 @@ enum json_states {
 
 enum json_value_types
 {
+  JSON_VALUE_UNINITALIZED=0,
   JSON_VALUE_OBJECT=1,
   JSON_VALUE_ARRAY=2,
   JSON_VALUE_STRING=3,

include/my_byteorder.h

@@ -31,10 +31,10 @@
   format (low byte first). There are 'korr' (assume 'corrector') variants
   for integer types, but 'get' (assume 'getter') for floating point types.
 */
-#if defined(__i386__) || defined(_WIN32)
+#if (defined(__i386__) || defined(_WIN32)) && !defined(WITH_UBSAN)
 #define MY_BYTE_ORDER_ARCH_OPTIMIZED
 #include "byte_order_generic_x86.h"
-#elif defined(__x86_64__)
+#elif defined(__x86_64__) && !defined(WITH_UBSAN)
 #include "byte_order_generic_x86_64.h"
 #else
 #include "byte_order_generic.h"

include/my_compare.h

@@ -1,5 +1,5 @@
 /* Copyright (c) 2011, Oracle and/or its affiliates.
-   Copyright (c) Monty Program Ab; 1991-2011
+   Copyright (c) 1991, 2020, MariaDB Corporation.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -95,15 +95,16 @@ static inline uchar get_rec_bits(const uchar *ptr, uchar ofs, uint len)
 {
   uint16 val= ptr[0];
   if (ofs + len > 8)
-    val|= (uint16)(ptr[1]) << 8;
-  return (val >> ofs) & ((1 << len) - 1);
+    val|= (uint16)(((uint) ptr[1]) << 8);
+  return (uchar) ((val >> ofs) & ((1 << len) - 1));
 }
 
 static inline void set_rec_bits(uint16 bits, uchar *ptr, uchar ofs, uint len)
 {
-  ptr[0]= (ptr[0] & ~(((1 << len) - 1) << ofs)) | (bits << ofs);
+  ptr[0]= (uchar) ((ptr[0] & ~(((1 << len) - 1) << ofs)) | (bits << ofs));
   if (ofs + len > 8)
-    ptr[1]= (ptr[1] & ~((1 << (len - 8 + ofs)) - 1)) | (bits >> (8 - ofs));
+    ptr[1]= (uchar) ((ptr[1] & ~((1 << (len - 8 + ofs)) - 1)) |
+                     bits >> (8 - ofs));
 }
 
 #define clr_rec_bits(bit_ptr, bit_ofs, bit_len) \

include/myisampack.h

@@ -30,7 +30,7 @@
 #define mi_uint1korr(A) ((uint8)(*A))
 
 #define mi_sint2korr(A) ((int16) (((int16) (((const uchar*) (A))[1])) |\
-                                  ((int16) ((int16) ((const char*) (A))[0]) << 8)))
+                                  ((int16) ((uint16) ((const uchar*) (A))[0]) << 8)))
 #define mi_sint3korr(A) ((int32) (((((const uchar*) (A))[0]) & 128) ? \
                                   (((uint32) 255L << 24) | \
                                    (((uint32) ((const uchar*) (A))[0]) << 16) |\
@@ -39,10 +39,10 @@
                                   (((uint32) ((const uchar*) (A))[0]) << 16) |\
                                   (((uint32) ((const uchar*) (A))[1]) << 8) | \
                                   ((uint32) ((const uchar*) (A))[2])))
-#define mi_sint4korr(A) ((int32) (((int32) (((const uchar*) (A))[3])) |\
-                                  ((int32) (((const uchar*) (A))[2]) << 8) |\
-                                  ((int32) (((const uchar*) (A))[1]) << 16) |\
-                                  ((int32) ((int16) ((const char*) (A))[0]) << 24)))
+#define mi_sint4korr(A) ((int32) (((uint32) (((const uchar*) (A))[3])) |\
+                                  ((uint32) (((const uchar*) (A))[2]) << 8) |\
+                                  ((uint32) (((const uchar*) (A))[1]) << 16) |\
+                                  ((uint32) (((const uchar*) (A))[0]) << 24)))
 #define mi_sint8korr(A) ((longlong) mi_uint8korr(A))
 #define mi_uint2korr(A) ((uint16) (((uint16) (((const uchar*) (A))[1])) |\
                                    ((uint16) (((const uchar*) (A))[0]) << 8)))

mysql-test/include/not_asan.inc

@@ -0,0 +1,8 @@
+# This file should only be used with test that finds bugs in ASan that can not
+# be overcome. In normal cases one should fix the bug server/test case or in
+# the worst case add a (temporary?) suppression in asan.supp or lsan.supp
+
+if (`select count(*) from information_schema.system_variables where variable_name='have_sanitizer' and global_value="ASAN"`)
+{
+--skip Can't be run with ASan
+}

mysql-test/include/not_ubsan.inc

@@ -0,0 +1,8 @@
+# This file should only be used with test that finds bugs in ASan that can not
+# be overcome. In normal cases one should fix the bug server/test case or in
+# the worst case add a (temporary?) suppression in asan.supp or lsan.supp
+
+if (`select count(*) from information_schema.system_variables where variable_name='have_sanitizer' and global_value="UBSAN"`)
+{
+--skip Can't be run with UBSAN
+}

mysql-test/main/mysqld--help.test

@@ -2,6 +2,8 @@
 # mysqld --help
 #
 --source include/not_embedded.inc
+--source include/not_asan.inc
+--source include/not_ubsan.inc
 --source include/have_perfschema.inc
 --source include/have_profiling.inc
 --source include/platform.inc

mysql-test/mysql-test-run.pl

@@ -1731,7 +1731,6 @@ sub collect_mysqld_features {
 }
 
 
-
 sub collect_mysqld_features_from_running_server ()
 {
   my $mysql= mtr_exe_exists("$path_client_bindir/mysql");
@@ -4376,7 +4375,13 @@ ($$)
      qr/InnoDB: Table .*mysql.*innodb_table_stats.* not found./,
      qr/InnoDB: User stopword table .* does not exist./,
      qr/Dump thread [0-9]+ last sent to server [0-9]+ binlog file:pos .+/,
-     qr/Detected table cache mutex contention at instance .* waits. Additional table cache instance cannot be activated: consider raising table_open_cache_instances. Number of active instances/
+     qr/Detected table cache mutex contention at instance .* waits. Additional table cache instance cannot be activated: consider raising table_open_cache_instances. Number of active instances/,
+
+     # for UBSAN
+     qr/decimal\.c.*: runtime error: signed integer overflow/,
+     # Disable test for UBSAN on dynamically loaded objects
+     qr/runtime error: member call.*object.*'Handler_share'/,
+     qr/sql_type\.cc.* runtime error: member call.*object.* 'Type_collection'/,
     );
 
   my $matched_lines= [];

mysql-test/suite/plugins/t/multiauth.test

@@ -1,3 +1,4 @@
+--source include/not_ubsan.inc
 #
 # MDEV-11340 Allow multiple alternative authentication methods for the same user
 #

mysql-test/suite/sys_vars/inc/sysvars_server.inc

@@ -23,7 +23,7 @@ select VARIABLE_NAME,VARIABLE_SCOPE,VARIABLE_TYPE,VARIABLE_COMMENT,NUMERIC_MIN_V
           variable_name not like 'wsrep%' and
           variable_name not like 's3%' and
           variable_name not in (
-          'log_tc_size'
+          'log_tc_size','have_sanitizer'
         )
   order by variable_name;
 

mysql-test/suite/sys_vars/r/sysvars_server_embedded.result

@@ -9,7 +9,7 @@ where variable_name not like 'debug%' and
 variable_name not like 'wsrep%' and
 variable_name not like 's3%' and
 variable_name not in (
-'log_tc_size'
+'log_tc_size','have_sanitizer'
         )
 order by variable_name;
 VARIABLE_NAME	ALTER_ALGORITHM

mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result

@@ -9,7 +9,7 @@ where variable_name not like 'debug%' and
 variable_name not like 'wsrep%' and
 variable_name not like 's3%' and
 variable_name not in (
-'log_tc_size'
+'log_tc_size','have_sanitizer'
         )
 order by variable_name;
 VARIABLE_NAME	ALTER_ALGORITHM

mysql-test/suite/sys_vars/t/thread_stack_basic.test

@@ -1,6 +1,8 @@
 #
 # only global
 #
+--source include/not_asan.inc
+--source include/not_ubsan.inc
 --replace_result 392192 299008
 select @@global.thread_stack;
 --error ER_INCORRECT_GLOBAL_LOCAL_VAR

mysys/ma_dyncol.c

@@ -865,7 +865,7 @@ dynamic_column_uint_read(DYNAMIC_COLUMN_VALUE *store_it_here,
 
 static size_t dynamic_column_sint_bytes(longlong val)
 {
-  return dynamic_column_uint_bytes((val << 1) ^
+  return dynamic_column_uint_bytes((((ulonglong) val) << 1) ^
                                    (val < 0 ? 0xffffffffffffffffull : 0));
 }
 
@@ -883,8 +883,8 @@ static enum enum_dyncol_func_result
 dynamic_column_sint_store(DYNAMIC_COLUMN *str, longlong val)
 {
   return dynamic_column_uint_store(str,
-                                 (val << 1) ^
-                                 (val < 0 ? 0xffffffffffffffffULL : 0));
+                                   (((ulonglong) val) << 1) ^
+                                   (val < 0 ? 0xffffffffffffffffULL : 0));
 }
 
 

plugin/auth_ed25519/CMakeLists.txt

@@ -19,6 +19,10 @@ IF(MSVC)
   SET_SOURCE_FILES_PROPERTIES(${REF10_SOURCES} PROPERTY COMPILE_FLAGS "/wd4244 /wd4146")
 ENDIF()
 
+IF(CMAKE_C_COMPILER_ID MATCHES "GNU" AND CMAKE_C_COMPILER_VERSION LESS 11 AND CMAKE_C_COMPILER_VERSION GREATER 6)
+  SET_SOURCE_FILES_PROPERTIES(${REF10_SOURCES} PROPERTY COMPILE_FLAGS -fno-sanitize=shift)
+ENDIF()
+
 # server plugin *cannot* link with the library, it needs all sources to be
 # compiled with MYSQL_DYNAMIC_PLUGIN
 MYSQL_ADD_PLUGIN(auth_ed25519 server_ed25519.c ${REF10_SOURCES} MODULE_ONLY)

sql/compat56.h

@@ -30,8 +30,8 @@
 
 #define MY_PACKED_TIME_GET_INT_PART(x)     ((x) >> 24)
 #define MY_PACKED_TIME_GET_FRAC_PART(x)    ((x) % (1LL << 24))
-#define MY_PACKED_TIME_MAKE(i, f)          ((((longlong) (i)) << 24) + (f))
-#define MY_PACKED_TIME_MAKE_INT(i)         ((((longlong) (i)) << 24))
+#define MY_PACKED_TIME_MAKE(i, f)          ((((ulonglong) (i)) << 24) + (f))
+#define MY_PACKED_TIME_MAKE_INT(i)         ((((ulonglong) (i)) << 24))
 
 longlong TIME_to_longlong_datetime_packed(const MYSQL_TIME *);
 longlong TIME_to_longlong_time_packed(const MYSQL_TIME *);

sql/events.cc

@@ -658,8 +658,16 @@ Events::drop_schema_events(THD *thd, const char *db)
   */
   if (event_queue)
     event_queue->drop_schema_events(thd, &db_lex);
-  db_repository->drop_schema_events(thd, &db_lex);
-
+  if (db_repository)
+    db_repository->drop_schema_events(thd, &db_lex);
+  else
+  {
+    if ((db_repository= new Event_db_repository))
+    {
+      db_repository->drop_schema_events(thd, &db_lex);
+      delete db_repository;
+    }
+  }
   DBUG_VOID_RETURN;
 }
 

sql/field.h

@@ -4534,7 +4534,13 @@ class Field_bit :public Field {
   void move_field_offset(my_ptrdiff_t ptr_diff)
   {
     Field::move_field_offset(ptr_diff);
-    bit_ptr= ADD_TO_PTR(bit_ptr, ptr_diff, uchar*);
+
+    /*
+      clang does not like when things are added to a null pointer, even if
+      it is never referenced.
+    */
+    if (bit_ptr)
+      bit_ptr= ADD_TO_PTR(bit_ptr, ptr_diff, uchar*);
   }
   void hash(ulong *nr, ulong *nr2);
 

sql/gcalc_tools.h

@@ -184,7 +184,11 @@ class Gcalc_result_receiver
   double first_x, first_y, prev_x, prev_y;
   double shape_area;
 public:
-  Gcalc_result_receiver() : collection_result(FALSE), n_shapes(0), n_holes(0)
+Gcalc_result_receiver() :
+    n_points(0),
+    common_shapetype(Gcalc_function::shape_point),
+    collection_result(FALSE), n_shapes(0), n_holes(0),
+    cur_shape(Gcalc_function::shape_point), shape_pos(0)
     {}
   int start_shape(Gcalc_function::shape_type shape);
   int add_point(double x, double y);

sql/ha_partition.cc

@@ -6461,6 +6461,7 @@ int ha_partition::multi_range_read_init(RANGE_SEQ_IF *seq,
   DBUG_ENTER("ha_partition::multi_range_read_init");
   DBUG_PRINT("enter", ("partition this: %p", this));
 
+  eq_range= 0;
   m_seq_if= seq;
   m_seq= seq->init(seq_init_param, n_ranges, mrr_mode);
   if (unlikely((error= multi_range_key_create_key(seq, m_seq))))

sql/item.cc

@@ -2599,9 +2599,7 @@ Item_sp::Item_sp(THD *thd, Name_resolution_context *context_arg,
   dummy_table= (TABLE*) thd->calloc(sizeof(TABLE) + sizeof(TABLE_SHARE) +
                                     sizeof(Query_arena));
   dummy_table->s= (TABLE_SHARE*) (dummy_table + 1);
-  /* TODO(cvicentiu) Move this sp_query_arena in the class as a direct member.
-     Currently it can not be done due to header include dependencies. */
-  sp_query_arena= (Query_arena *) (dummy_table->s + 1);
+  sp_query_arena= new(dummy_table->s + 1) Query_arena();
   memset(&sp_mem_root, 0, sizeof(sp_mem_root));
 }
 
@@ -2612,7 +2610,7 @@ Item_sp::Item_sp(THD *thd, Item_sp *item):
   dummy_table= (TABLE*) thd->calloc(sizeof(TABLE)+ sizeof(TABLE_SHARE) +
                                     sizeof(Query_arena));
   dummy_table->s= (TABLE_SHARE*) (dummy_table+1);
-  sp_query_arena= (Query_arena *) (dummy_table->s + 1);
+  sp_query_arena= new(dummy_table->s + 1) Query_arena();
   memset(&sp_mem_root, 0, sizeof(sp_mem_root));
 }
 
@@ -6208,12 +6206,14 @@ Item *Item_field::replace_equal_field(THD *thd, uchar *arg)
                   item_equal->compare_type_handler()->cmp_type());
       return const_item2;
     }
-    Item_field *subst= 
-      (Item_field *)(item_equal->get_first(param->context_tab, this));
+    Item_ident *subst=
+      (Item_ident *) (item_equal->get_first(param->context_tab, this));
     if (subst)
-      subst= (Item_field *) (subst->real_item());
-    if (subst && !field->eq(subst->field))
-      return subst;
+    {
+      Item_field *subst2= (Item_field *) (subst->real_item());
+      if (subst2 && !field->eq(subst2->field))
+      return subst2;
+    }
   }
   return this;
 }