Merge branch '11.0' into 11.1

Author: vuvova

.github/pull_request_template.md

@@ -1,61 +1,47 @@
 <!--
 Thank you for contributing to the MariaDB Server repository!
 
-You can help us review your changes faster by filling this template <3
+You can help us review your changes faster by filling in this template <3
 
-If you have any questions related to MariaDB or you just want to
-hang out and meet other community members, please join us on
-https://mariadb.zulipchat.com/ .
+If you have any questions related to MariaDB or you just want to hang out and meet other community members, please join us on https://mariadb.zulipchat.com/ .
 -->
 
 <!--
-If you've already identified a https://jira.mariadb.org/ issue
-that seems to track this bug/feature, please add its number below.
+If you've already identified a https://jira.mariadb.org/ issue that seems to track this bug/feature, please add its number below.
 -->
-- [x] *The Jira issue number for this PR is: MDEV-_____*
+- [x] *The Jira issue number for this PR is: MDEV-______*
 
 <!--
 An amazing description should answer some questions like:
 1. What problem is the patch trying to solve?
-2. If some output changed, what was it looking like before
-   the change and how it's looking with this patch applied
-3. Do you think this patch might introduce side-effects in
-   other parts of the server?
+2. If some output changed that is not visible in a test case, what was it looking like before the change and how it's looking with this patch applied?
+3. Do you think this patch might introduce side-effects in other parts of the server?
 -->
 ## Description
 TODO: fill description here
 
 ## How can this PR be tested?
 
-TODO: modify the automated test suite to verify that the PR causes MariaDB to
-behave as intended. Consult the documentation on
-["Writing good test cases"](https://mariadb.org/get-involved/getting-started-for-developers/writing-good-test-cases-mariadb-server).
-In many cases, this will be as simple as modifying one `.test` and one `.result`
-file in the `mysql-test/` subdirectory. Without _automated_ tests, future regressions
-in the expected behavior can't be automatically detected and verified.
+TODO: modify the automated test suite to verify that the PR causes MariaDB to behave as intended.
+Consult the documentation on ["Writing good test cases"](https://mariadb.org/get-involved/getting-started-for-developers/writing-good-test-cases-mariadb-server).
+<!--
+In many cases, this will be as simple as modifying one `.test` and one `.result` file in the `mysql-test/` subdirectory.
+Without automated tests, future regressions in the expected behavior can't be automatically detected and verified.
+-->
 
-If the changes are not amenable to automated testing, please explain why not and
-carefully describe how to test manually.
+If the changes are not amenable to automated testing, please explain why not and carefully describe how to test manually.
 
 <!--
-Tick one of the following boxes [x] to help us understand
-if the base branch for the PR is correct
-(Currently the earliest maintained branch is 10.3)
+Tick one of the following boxes [x] to help us understand if the base branch for the PR is correct. (Currently the earliest maintained branch is 10.3)
 -->
 ## Basing the PR against the correct MariaDB version
-- [ ] *This is a new feature and the PR is based against the latest MariaDB development branch*
-- [ ] *This is a bug fix and the PR is based against the earliest maintained branch in which the bug can be reproduced*
+- [ ] *This is a new feature and the PR is based against the latest MariaDB development branch.*
+- [ ] *This is a bug fix and the PR is based against the earliest maintained branch in which the bug can be reproduced.*
 
 <!--
-You might consider answering some questions like:
-1. Does this affect the on-disk format used by MariaDB?
-2. Does this change any behavior experienced by a user
-   who upgrades from a version prior to this patch?
-3. Would a user be able to start MariaDB on a datadir
-   created prior to your fix?
+  All code merged into the MariaDB codebase must meet a quality standard and codying style.
+  Maintainers are happy to point out inconsistencies but in order to speed up the review and merge process we ask you to check the CODING standards.
 -->
-## Backward compatibility
-TODO: fill details here, if applicable, or remove the section
-
 ## PR quality check
 - [ ] I checked the [CODING_STANDARDS.md](https://github.com/MariaDB/server/blob/11.0/CODING_STANDARDS.md) file and my PR conforms to this where appropriate.
+- [ ] For any trivial modifications to the PR, I am ok with the reviewer making the changes themselves.

.gitignore

@@ -267,7 +267,6 @@ support-files/mysql.10.0.11.spec
 support-files/mysql.server
 support-files/mysql.service
 support-files/mysql.spec
-support-files/mysql-log-rotate
 support-files/mysqld.service
 support-files/mysqld_multi.server
 support-files/policy/selinux/mysqld-safe.pp

.gitlab-ci.yml

@@ -27,6 +27,7 @@ stages:
   - build
   - test
   - Salsa-CI
+  - sast
 
 default:
   # Base image for builds and tests unless otherwise defined
@@ -42,7 +43,7 @@ variables:
   CMAKE_FLAGS: "-DWITH_SSL=system -DPLUGIN_COLUMNSTORE=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_S3=NO -DPLUGIN_MROONGA=NO -DPLUGIN_CONNECT=NO -DPLUGIN_MROONGA=NO -DPLUGIN_TOKUDB=NO -DPLUGIN_PERFSCHEMA=NO -DWITH_WSREP=OFF"
   # Major version dictates which branches share the same ccache. E.g. 10.6-abc
   # and 10.6-xyz will have the same cache.
-  MARIADB_MAJOR_VERSION: "10.8"
+  MARIADB_MAJOR_VERSION: "11.0"
   # NOTE! Currently ccache is only used on the Centos8 build. As each job has
   # sufficiently different environments they are unable to benefit from each
   # other's ccaches. As each build generates about 1 GB of ccache, having
@@ -515,6 +516,70 @@ mini-benchmark:
       metrics:
         - metrics.txt
 
+cppcheck: 
+  stage: sast
+  needs: []
+  variables:
+    GIT_STRATEGY: fetch
+    GIT_SUBMODULE_STRATEGY: normal
+  script:
+    - yum install -y cppcheck diffutils
+    # --template: use a single-line template
+    # --force: check large directories without warning
+    # -i<directory>: ignore this directory when scanning
+    # -j: run multiple cppcheck threads
+    # Use newline to escape colon in yaml
+    - >
+      cppcheck --template="{file}:{line}: {severity}: {message}" --force
+      client dbug extra include libmariadb libmysqld libservices mysql-test mysys mysys_ssl pcre plugin
+      strings tests unittest vio wsrep-lib sql sql-common storage
+      -istorage/mroonga -istorage/tokudb -istorage/spider -istorage/rocksdb -iextra/ -ilibmariadb/ -istorage/columnstore
+      --output-file=cppcheck.txt -j $(nproc)
+    # Parallel jobs may output findings in an nondeterministic order. Sort to match ignorelist.
+    - cat cppcheck.txt | sort > cppcheck_sorted.txt
+    # Remove line numbers for diff
+    - sed 's/:[^:]*:/:/' cppcheck_sorted.txt > cppcheck_sorted_no_line_numbers.txt
+    # Only print new issues not found in ignore list
+    - echo "Problems found in ignore list that were not discovered by cppcheck (may have been fixed)."
+    - diff --changed-group-format='%>' --unchanged-group-format='' cppcheck_sorted_no_line_numbers.txt tests/code_quality/cppcheck_ignorelist.txt || true
+    - echo "Problems found by cppcheck that were not in ignore list."
+    - diff --changed-group-format='%<' --unchanged-group-format='' cppcheck_sorted_no_line_numbers.txt tests/code_quality/cppcheck_ignorelist.txt > lines_not_ignored.txt || true
+    - cat lines_not_ignored.txt && test ! -s lines_not_ignored.txt
+  artifacts:
+    when: always
+    paths:
+      - cppcheck_sorted.txt
+
+flawfinder:
+  stage: sast
+  needs: []
+  variables:
+    GIT_STRATEGY: fetch
+    GIT_SUBMODULE_STRATEGY: normal
+  script:
+    - yum install -y python3 python3-pip jq diffutils git
+    - pip install flawfinder
+    - flawfinder --falsepositive --quiet --html . > flawfinder-all-vulnerabilities.html
+    - cat flawfinder-all-vulnerabilities.html | grep "Hits ="
+    - flawfinder --falsepositive --quiet --minlevel=5 --sarif . > flawfinder-output.json
+    # FlawFinder's --sarif output will display all vulnerabilities despite having --minlevel=5 specified.
+    # Therefore, we postprocess the results with jq and filter out findings where the vulnerability level is less than 5.
+    # Also in the SARIF output format, the vulnerabilities are ranked as 0.2/0.4/0.6/0.8/1.0 which correspond to the --minlevel=1/2/3/4/5 of FlawFinder.
+    # Additionally, we sort the results because individual findings are consistent across different runs, but their ordering may not be.
+    # Vulnerabilities can also be ignored in-line (/* Flawfinder: ignore */), but this option was chosen as to not clutter the codebase.
+    - jq 'del(.runs[] | .tool | .driver | .rules) | del(.runs[] | .results[] | select(.rank < 1)) | del(.runs[] | .results[] | .locations[] | .physicalLocation | .region | .startLine) | .runs[0].results|=sort_by(.fingerprints)' flawfinder-output.json > flawfinder-min-level5.json
+    # Diff against known vulnerabilities, but ignore the line number.
+    - echo "Problems found in ignore list that were not discovered by flawfinder (may have been fixed)."
+    - diff --changed-group-format='%>' --unchanged-group-format='' flawfinder-min-level5.json tests/code_quality/flawfinder_ignorelist.json || true
+    - echo "Problems found by flawfinder that were not in ignore list."
+    - diff --changed-group-format='%<' --unchanged-group-format='' flawfinder-min-level5.json tests/code_quality/flawfinder_ignorelist.json > lines_not_ignored.txt || true
+    - cat lines_not_ignored.txt && test ! -s lines_not_ignored.txt
+  artifacts:
+    when: always
+    paths:
+      - flawfinder-all-vulnerabilities.html
+      - flawfinder-min-level5.json
+      
 # Once all RPM builds and tests have passed, also run the DEB builds and tests
 # @NOTE: This is likely to work well only on salsa.debian.org as the Gitlab.com
 # runners are too small for everything this stage does.

.gitmodules

@@ -1,25 +1,19 @@
 [submodule "libmariadb"]
 	path = libmariadb
 	url = https://github.com/MariaDB/mariadb-connector-c.git
-	ignore = all
 [submodule "storage/rocksdb/rocksdb"]
 	path = storage/rocksdb/rocksdb
 	url = https://github.com/facebook/rocksdb.git
-	ignore = all
 [submodule "wsrep-lib"]
 	path = wsrep-lib
 	url = https://github.com/codership/wsrep-lib.git
 	branch = master
-	ignore = all
 [submodule "extra/wolfssl/wolfssl"]
 	path = extra/wolfssl/wolfssl
 	url = https://github.com/wolfSSL/wolfssl.git
-	ignore = all
 [submodule "storage/maria/libmarias3"]
 	path = storage/maria/libmarias3
 	url = https://github.com/mariadb-corporation/libmarias3.git
-	ignore = all
 [submodule "storage/columnstore/columnstore"]
 	path = storage/columnstore/columnstore
 	url = https://github.com/mariadb-corporation/mariadb-columnstore-engine.git
-	ignore = all

BUILD/SETUP.sh

@@ -216,15 +216,14 @@ max_plugins="--with-plugins=max"
 max_no_embedded_configs="$max_plugins"
 max_no_qc_configs="$max_plugins --without-query-cache"
 max_configs="$max_plugins --with-embedded-server --with-libevent --with-plugin-rocksdb=dynamic --with-plugin-test_sql_discovery=DYNAMIC --with-plugin-file_key_management=DYNAMIC --with-plugin-hashicorp_key_management=DYNAMIC --with-plugin-auth_gssapi=DYNAMIC"
-all_configs="$max_plugins --with-embedded-server --with-innodb_plugin --with-libevent"
+all_configs="$max_configs"
 
 #
 # CPU and platform specific compilation flags.
 #
 alpha_cflags="$check_cpu_cflags -Wa,-m$cpu_flag"
-amd64_cflags="$check_cpu_cflags"
-amd64_cxxflags=""  # If dropping '--with-big-tables', add here  "-DBIG_TABLES"
 pentium_cflags="$check_cpu_cflags -m32"
+amd64_cflags="$check_cpu_cflags -m64"
 pentium64_cflags="$check_cpu_cflags -m64"
 ppc_cflags="$check_cpu_cflags"
 sparc_cflags=""

BUILD/compile-amd64-debug-all

@@ -1,7 +1,10 @@
 #! /bin/sh
+
 path=`dirname $0`
+set -- "$@" --with-debug=full
 . "$path/SETUP.sh"
+
 extra_flags="$amd64_cflags $debug_cflags"
 extra_configs="$amd64_configs $debug_configs $all_configs"
-
+CC="$CC --pipe"
 . "$path/FINISH.sh"

BUILD/compile-amd64-debug-max

@@ -1,7 +1,6 @@
 #! /bin/sh
 
-# Copyright (c) 2005, 2006 MySQL AB
-# Use is subject to license terms
+# Copyright (c) 2005, 2010, Oracle and/or its affiliates. All rights reserved.
 # 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -14,11 +13,13 @@
 # 
 # You should have received a copy of the GNU General Public License
 # along with this program; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1335  USA
+# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335  USA
 
 path=`dirname $0`
 . "$path/SETUP.sh"
+
 extra_flags="$amd64_cflags $debug_cflags"
 extra_configs="$amd64_configs $debug_configs $max_configs"
 
+CC="$CC --pipe"
 . "$path/FINISH.sh"

BUILD/compile-pentium64-debug-all

@@ -6,7 +6,5 @@ set -- "$@" --with-debug=full
 
 extra_flags="$pentium64_cflags $debug_cflags"
 extra_configs="$pentium_configs $debug_configs $all_configs"
-
-extra_configs="$extra_configs "
 CC="$CC --pipe"
 . "$path/FINISH.sh"

BUILD/compile-pentium64-debug-max

@@ -21,6 +21,5 @@ path=`dirname $0`
 extra_flags="$pentium64_cflags $debug_cflags"
 extra_configs="$pentium_configs $debug_configs $max_configs"
 
-extra_configs="$extra_configs "
 CC="$CC --pipe"
 . "$path/FINISH.sh"

CODING_STANDARDS.md

@@ -118,6 +118,23 @@ if (!condition)
 return success;
 ```
 
+#### Functions
+
+Consecutive functions should be separated with 2 empty lines in between
+
+```cpp
+void my_function_1()
+{
+  <logic>  
+}
+
+
+void my_function_2()
+{
+  <logic>  
+}
+```
+
 ### File names
 
 File names should be lower case with underscore word separators.
@@ -218,11 +235,28 @@ Variables should be declared at the start of it's context (start of function, in
 The benefits of this:
 - Code lines gets shorter
 - It is easier to see the stack space used by a function.
-- It is easer to find the declaration of the variable.
+- It is easier to find the declaration of the variable.
 - If one has to add an 'if (error) goto end' construct, one can do
   that without having to move variable declarations around.
 
 
+### Variable initializations
+Variables can be initialized using assignment operator or initializer and expression list.
+For Example:
+
+```cpp
+int milliseconds= 1000;
+char type= 't';
+```
+
+Or
+
+```cpp
+int milliseconds{1000};
+char type{'t'};
+```
+
+
 ### Constant integers
 
 Constant integers that are used to define elements such as buffer sizes should be defined rather than used directly.