Fix all warnings given by UBSAN

The easiest way to compile and test the server with UBSAN is to run:
./BUILD/compile-pentium64-ubsan
and then run mysql-test-run.
After this commit, one should be able to run this without any UBSAN
warnings. There is still a few compiler warnings that should be fixed
at some point, but these do not expose any real bugs.

The 'special' cases where we disable, suppress or circumvent UBSAN are:
- ref10 source (as here we intentionally do some shifts that UBSAN
  complains about.
- x86 version of optimized int#korr() methods. UBSAN do not like unaligned
  memory access of integers.  Fixed by using byte_order_generic.h when
  compiling with UBSAN
- We use smaller thread stack with ASAN and UBSAN, which forced me to
  disable a few tests that prints the thread stack size.
- Verifying class types does not work for shared libraries. I added
  suppression in mysql-test-run.pl for this case.
- Added '#ifdef WITH_UBSAN' when using integer arithmetic where it is
  safe to have overflows (two cases, in item_func.cc).

Things fixed:
- Don't left shift signed values
  (byte_order_generic.h, mysqltest.c, item_sum.cc and many more)
- Don't assign not non existing values to enum variables.
- Ensure that bool and enum values are properly initialized in
  constructors.  This was needed as UBSAN checks that these types has
  correct values when one copies an object.
  (gcalc_tools.h, ha_partition.cc, item_sum.cc, partition_element.h ...)
- Ensure we do not called handler functions on unallocated objects or
  deleted objects.
  (events.cc, sql_acl.cc).
- Fixed bugs in Item_sp::Item_sp() where we did not call constructor
  on Query_arena object.
- Fixed several cast of objects to an incompatible class!
  (Item.cc, Item_buff.cc, item_timefunc.cc, opt_subselect.cc, sql_acl.cc,
   sql_select.cc ...)
- Ensure we do not do integer arithmetic that causes over or underflows.
  This includes also ++ and -- of integers.
  (Item_func.cc, Item_strfunc.cc, item_timefunc.cc, sql_base.cc ...)
- Added JSON_VALUE_UNITIALIZED to json_value_types and ensure that
  value_type is initialized to this instead of to -1, which is not a valid
  enum value for json_value_types.
- Ensure we do not call memcpy() when second argument could be null.
- Fixed that Item_func_str::make_empty_result() creates an empty string
  instead of a null string (safer as it ensures we do not do arithmetic
  on null strings).

Other things:

- Changed struct st_position to an OBJECT and added an initialization
  function to it to ensure that we do not copy or use uninitialized
  members. The change to a class was also motived that we used "struct
  st_position" and POSITION randomly trough the code which was
  confusing.
- Notably big rewrite in sql_acl.cc to avoid using deleted objects.
- Changed in sql_partition to use '^' instead of '-'. This is safe as
  the operator is either 0 or 0x8000000000000000ULL.
- Added check for select_nr < INT_MAX in JOIN::build_explain() to
  avoid bug when get_select() could return NULL.
- Reordered elements in POSITION for better alignment.
- Changed sql_test.cc::print_plan() to use pointers instead of objects.
- Fixed bug in find_set() where could could execute '1 << -1'.
- Added variable have_sanitizer, used by mtr.  (This variable was before
  only in 10.5 and up).  It can now have one of two values:
  ASAN or UBSAN.
- Moved ~Archive_share() from ha_archive.cc to ha_archive.h and marked
  it virtual. This was an effort to get UBSAN to work with loaded storage
  engines. I kept the change as the new place is better.
- Added in CONNECT engine COLBLK::SetName(), to get around a wrong cast
  in tabutil.cpp.
- Added HAVE_REPLICATION around usage of rgi_slave, to get embedded
  server to compile with UBSAN. (Patch from Marko).
- Added #ifdef for powerpc64 to avoid a bug in old gcc versions related
  to integer arithmetic.

Changes that should not be needed but had to be done to suppress warnings
from UBSAN:

- Added static_cast<<uint16_t>> around shift to get rid of a LOT of
  compiler warnings when using UBSAN.
- Had to change some '/' of 2 base integers to shift to get rid of
  some compile time warnings.

Reviewed by:
- Json changes: Alexey Botchkov
- Charset changes in ctype-uca.c: Alexander Barkov
- InnoDB changes & Embedded server: Marko Mäkelä
- sql_acl.cc changes: Vicențiu Ciorbaru
- build_explain() changes: Sergey Petrunia

Author: montywi

BUILD/SETUP.sh

@@ -141,7 +141,7 @@ elif [ "x$warning_mode" = "xmaintainer" ]; then
   debug_extra_cflags="-g3"
 else
 # Both C and C++ warnings
-  warnings="-Wall -Wextra -Wunused -Wwrite-strings -Wno-uninitialized -Wno-strict-aliasing -Wimplicit-fallthrough=2"
+  warnings="-Wall -Wextra -Wunused -Wwrite-strings -Wno-uninitialized -Wno-strict-aliasing -Wimplicit-fallthrough=2 -Wformat-security -Wvla"
 
 # For more warnings, uncomment the following line
 # warnings="$warnings -Wshadow"

BUILD/compile-pentium64-ubsan

@@ -0,0 +1,29 @@
+#! /bin/sh
+# Copyright (c) 2018, MariaDB Corporation.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; version 2 of the License.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1335  USA
+
+# Compilation with UBSAN, The Undefined Behavior Sanitizer
+# We have to use -Wno-uninitialized and -Wno-unitialized we get a lot of false
+# positive warnings for this when compiling with -fsanitize=undefined.
+# We also have to compile without Spider as linking with Spider library does
+# not work. (errno: 11, undefined symbol: _ZTI12ha_partition)
+
+path=`dirname $0`
+. "$path/SETUP.sh"
+
+extra_flags="$pentium64_cflags $debug_cflags -fsanitize=undefined -DWITH_UBSAN -Wno-conversion -Wno-uninitialized"
+extra_configs="$pentium_configs $debug_configs -DWITH_UBSAN=ON -DMYSQL_MAINTAINER_MODE=NO --without-spider --without-tokudb"
+
+. "$path/FINISH.sh"

client/mysqltest.cc

@@ -10905,7 +10905,7 @@ int get_next_bit(REP_SET *set,uint lastpos)
 
   start=set->bits+ ((lastpos+1) / WORD_BIT);
   end=set->bits + set->size_of_bits;
-  bits=start[0] & ~((1 << ((lastpos+1) % WORD_BIT)) -1);
+  bits=start[0] & ~((1U << ((lastpos+1) % WORD_BIT)) -1);
 
   while (! bits && ++start < end)
     bits=start[0];

cmake/configure.pl

@@ -93,6 +93,11 @@ sub check_compiler
   {
     $option = substr($option, 2);
   }
+  elsif (substr ($option, 0, 2) eq "-D")
+  {
+    # Must be cmake config option
+    $option = substr($option, 1);
+  }
   else
   {
     # This must be environment variable
@@ -119,6 +124,11 @@ sub check_compiler
     $just_print=1;
     next;
   }
+  if ($option =~ /D.*=/)
+  {
+    $cmakeargs = $cmakeargs." -".$option;
+     next;
+  }
   if($option =~ /with-plugins=/)
   {
     my @plugins= split(/,/, substr($option,13));

include/byte_order_generic.h

@@ -28,10 +28,10 @@
 				  (((uint32) (uchar) (A)[2]) << 16) |\
 				  (((uint32) (uchar) (A)[1]) << 8) | \
 				  ((uint32) (uchar) (A)[0])))
-#define sint4korr(A)	(int32) (((int32) ((uchar) (A)[0])) |\
-				(((int32) ((uchar) (A)[1]) << 8)) |\
-				(((int32) ((uchar) (A)[2]) << 16)) |\
-				(((int32) ((int16) (A)[3]) << 24)))
+#define sint4korr(A)	(int32) (((uint32) ((uchar) (A)[0])) |\
+				(((uint32) ((uchar) (A)[1]) << 8)) |\
+				(((uint32) ((uchar) (A)[2]) << 16)) |\
+				(((uint32) ((uchar) (A)[3]) << 24)))
 #define sint8korr(A)	(longlong) uint8korr(A)
 #define uint2korr(A)	(uint16) (((uint16) ((uchar) (A)[0])) |\
 				  ((uint16) ((uchar) (A)[1]) << 8))

include/byte_order_generic_x86.h

@@ -17,6 +17,7 @@
 /*
   Optimized function-like macros for the x86 architecture (_WIN32 included).
 */
+
 #define sint2korr(A)	(*((const int16 *) (A)))
 #define sint3korr(A)	((int32) ((((uchar) (A)[2]) & 128) ? \
 				  (((uint32) 255L << 24) | \

include/json_lib.h

@@ -172,6 +172,7 @@ enum json_states {
 
 enum json_value_types
 {
+  JSON_VALUE_UNINITALIZED=0,
   JSON_VALUE_OBJECT=1,
   JSON_VALUE_ARRAY=2,
   JSON_VALUE_STRING=3,

include/my_byteorder.h

@@ -31,10 +31,10 @@
   format (low byte first). There are 'korr' (assume 'corrector') variants
   for integer types, but 'get' (assume 'getter') for floating point types.
 */
-#if defined(__i386__) || defined(_WIN32)
+#if (defined(__i386__) || defined(_WIN32)) && !defined(WITH_UBSAN)
 #define MY_BYTE_ORDER_ARCH_OPTIMIZED
 #include "byte_order_generic_x86.h"
-#elif defined(__x86_64__)
+#elif defined(__x86_64__) && !defined(WITH_UBSAN)
 #include "byte_order_generic_x86_64.h"
 #else
 #include "byte_order_generic.h"

include/my_compare.h

@@ -1,5 +1,5 @@
 /* Copyright (c) 2011, Oracle and/or its affiliates.
-   Copyright (c) Monty Program Ab; 1991-2011
+   Copyright (c) 1991, 2020, MariaDB Corporation.
 
    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
@@ -95,15 +95,16 @@ static inline uchar get_rec_bits(const uchar *ptr, uchar ofs, uint len)
 {
   uint16 val= ptr[0];
   if (ofs + len > 8)
-    val|= (uint16)(ptr[1]) << 8;
-  return (val >> ofs) & ((1 << len) - 1);
+    val|= (uint16)(((uint) ptr[1]) << 8);
+  return (uchar) ((val >> ofs) & ((1 << len) - 1));
 }
 
 static inline void set_rec_bits(uint16 bits, uchar *ptr, uchar ofs, uint len)
 {
-  ptr[0]= (ptr[0] & ~(((1 << len) - 1) << ofs)) | (bits << ofs);
+  ptr[0]= (uchar) ((ptr[0] & ~(((1 << len) - 1) << ofs)) | (bits << ofs));
   if (ofs + len > 8)
-    ptr[1]= (ptr[1] & ~((1 << (len - 8 + ofs)) - 1)) | (bits >> (8 - ofs));
+    ptr[1]= (uchar) ((ptr[1] & ~((1 << (len - 8 + ofs)) - 1)) |
+                     bits >> (8 - ofs));
 }
 
 #define clr_rec_bits(bit_ptr, bit_ofs, bit_len) \

include/myisampack.h

@@ -30,7 +30,7 @@
 #define mi_uint1korr(A) ((uint8)(*A))
 
 #define mi_sint2korr(A) ((int16) (((int16) (((const uchar*) (A))[1])) |\
-                                  ((int16) ((int16) ((const char*) (A))[0]) << 8)))
+                                  ((int16) ((uint16) ((const uchar*) (A))[0]) << 8)))
 #define mi_sint3korr(A) ((int32) (((((const uchar*) (A))[0]) & 128) ? \
                                   (((uint32) 255L << 24) | \
                                    (((uint32) ((const uchar*) (A))[0]) << 16) |\
@@ -39,10 +39,10 @@
                                   (((uint32) ((const uchar*) (A))[0]) << 16) |\
                                   (((uint32) ((const uchar*) (A))[1]) << 8) | \
                                   ((uint32) ((const uchar*) (A))[2])))
-#define mi_sint4korr(A) ((int32) (((int32) (((const uchar*) (A))[3])) |\
-                                  ((int32) (((const uchar*) (A))[2]) << 8) |\
-                                  ((int32) (((const uchar*) (A))[1]) << 16) |\
-                                  ((int32) ((int16) ((const char*) (A))[0]) << 24)))
+#define mi_sint4korr(A) ((int32) (((uint32) (((const uchar*) (A))[3])) |\
+                                  ((uint32) (((const uchar*) (A))[2]) << 8) |\
+                                  ((uint32) (((const uchar*) (A))[1]) << 16) |\
+                                  ((uint32) (((const uchar*) (A))[0]) << 24)))
 #define mi_sint8korr(A) ((longlong) mi_uint8korr(A))
 #define mi_uint2korr(A) ((uint16) (((uint16) (((const uchar*) (A))[1])) |\
                                    ((uint16) (((const uchar*) (A))[0]) << 8)))