Browse files

WL#6378 New data dictionary (umbrella).

A. Overview:
This big patch introduces data dictionary (DD) schema in MySQL
server. Some of the main benefits of this is,

* The .FRM files are gone, and all dictionary changes will
  now happen in the dictionary schema stored in InnoDB.

* Builds a base platform enabling us to improve performance of
  INFORMATION_SCHEMA queries, which can be implemented as a view.

* This patch enables a forthcoming incremental patch that would
  remove internal dictionary table of InnoDB. Enables,
   - There by eliminate inconsistencies between .FRM files and
     InnoDB tables.
   - Remove file system dependencies (lower-case-table-names).
   - Single repository of meta data for Server, SE and Plugins.
   - Eliminate meta data redundancy.

* Remove db.opt and rely on dictionary tables.

B. Work logs:
The WL#6378 is the umbrella WL tracking all dependent WL's. This
patch contains implementation of following dependent WL's,

* Worklogs that deal with just DD API framework:

  WL#6379 - Schema definitions for new DD.
            This important WL defines the central data dictionary
            table definitions.

  WL#6380 - Formulate framework for API for DD.

  WL#7284 - Implement common code for different DD APIs.
  WL#6385 - Define and Implement API for Schema.
  WL#6382 - Define and Implement API for Table objects.
  WL#6389 - Implementation of common View API.
  WL#6387 - Define and Implement API for Tablespaces.
  WL#7630 - Define and Implement API for Table Partition Info
  WL#8150 - Dictionary object cache.
  WL#7770 - Develop GUNIT test framework and guidelines for DD API.
  WL#7771 - Make sure errors are properly handled in DD API.

* Worklogs that change server code invoking DD API's:

  WL#6390 - Use new DD API for handling non-partitioned tables.
  WL#7836 - Use new DD API for handling partitioned tables.
  WL#6394 - Bootstrapping the new data dictionary.
  WL#7784 - Store temporary table meta data in memory.
  WL#8542 - Server shutdown procedure with new data dictionary
  WL#7593 - New data dictionary: don't hold LOCK_open while
            reading table definition.
  WL#7464 - InnoDB: provide a way to do non-locking reads
            This is pre-requisite for WL#6599.

C. Tips to DD API users:
* Refer code in sql/dd/dd_schema.* to quickly learn how to write
  code using DD API and update dictionary tables.

  Refer sql/dd/cache/dictionary_client.h to get overview on what
  is Dictionary_client and Auto_release interface.

  Interested can refer to sql/dd_table_share.{h,cc} which
  implements mapping between the TABLE_SHARE and the dd::Table DD

* Overview of the directory structure and source files introduced
  by this patch is as following,

  DD user exposed files:-

  sql/dd/        - Top level directory containing most of DD code.
  sql/dd/*.h     - Headers that you need to operate on DD.
  sql/dd/types/  - Contains headers to operate on single
                   dictionary object.
  sql/dd/dd_*.cc - Contains MySQL server and DD API framework
                   glue code. E.g., Code that updates to DD tables
                   upon table creation/alter table/drop table and
                   so on.
  sql/dd/cache/  - Contains implementation of DD Object cache.

  Implementation that is hidden to DD user:-

  sql/dd/impl/   - Contains implementation of DD API's.
  sql/dd/impl/cache/  - Contains implementation of DD object cache
                         E.g., dd::cache::Dictionary_client
  sql/dd/impl/types/  - Contains implementation of DD user object
                        operations. E.g., dd:Table, dd::View etc.
  sql/dd/impl/tables/ - Contains implementation of DD table
                        operations. E.g., dd::tables::* classes
                        that abstracts operations on a DD table.
  sql/dd/impl/raw/    - Contains implementation of generic
                        operations on all DD tables.

* The code related to .FRM handle is removed. So, the following
  files are removed,

D. New configuration variables and startup options:
* Introduced a new option "--install-server", behaving like
  "--bootstrap", but additionally creating the dictionary tables.
  Note that the --install-server is a temporary workaround until
  MTR use --initialize.

* The existing "--bootstrap" option behaves like before as far
  as possible, but note that the changes in plugin initialization
  means that e.g. InnoDB must be able to start, the DD must be
  able to start, etc., for bootstrap to enter the stage where SQL
  commands are actually executed.

* The mysql_install_db and the mysql_test_run scripts are modified
  to use the new "--install-server" option.

* The cmake script is also changed to
  use --install-server rather than --bootstrap.

* WL#8150 adds server options schema_definition_cache,
  tablespace_definition_cache and stored_program_definition_cache
  size. These define the DD object cache size for respective DD

E. Upcoming improvements
* WL#6599 New Data Dictionary and I_S integration
  Information schema queries will be execute faster as this WL
  would make information schema tables as a view on top of DD

* WL#7743 New data dictionary: changes to DDL-related parts of SE API
  - Support atomic/crash-safe DDL.
  - Support auxiliary columns (InnoDB-specific)
  - Support auxiliary tables (needed for InnoDB FTS)

* WL#6394 - Bootstrapping the new data dictionary.
  - Improvements in bootstrap procedure supporting removal of
    InnoDB dictionary.

* WL#7896 Use DD API to work with triggers.
  - Will remove use of .TRN and .TRG files and start using
    mysql.triggers DD table.

* Additional system tables are to be moved from MyISAM
  WL#7897 Use DD API to work with stored routines
  - Will remove mysql.proc MyISAM table and use mysql.routines
    InnoDB dictionary table.

  WL#7898 Use DD API to work with events.
  - Will remove mysql.event MyISAM table and use
    InnoDB dictionary table.

* WL#6391: Hide DD system tables from user.
  - Will hide direct access to DD system tables from user.

F. Permanent changes in behavior.
* Since the .frm files are gone, you will not be able to copy
  .FRMs/data files and move them around. We will address this
  when we finish the implementation of new import, based on
  serialized dictionary information, see WL#7069.

* Related to the above is the fact that --innodb-read-only
  option does not currently work. In 5.7, meta-data would be
  written to .FRM files and not be affected by the option, but
  now as meta-data is written to InnoDB tables, we must
  reconsider the semantics of this option.

* The setting of the system variable 'lower_case_table_names'
  determines the collation of e.g. the 'name' column in the table
  storing the meta data for tables. This collation is decided once
  and for all when the dictionary table is created. Thus, we do
  not support changing 'lower_case_table_names' afterwards.

* Character sets and collation tables are populated at initial
  boot, and for every subsequent startup, unless the server or
  the dictionary storage engine (InnoDB) is started in read
  only mode.

G. Intermediate limitations in functionality:
* Tests on upgrade/downgrade will not work, and will be
  temporarily disabled. QA will also have to temporarily postpone
  any upgrade tests, until we have completed the upgrade tool
  (WL#6392) from 5.7 to 5.8. We are working towards building a
  intermediate solution for system QA.

* Some InnoDB tests related to recovery of TRUNCATE are
  disabled. But n-test can be run as long as you are not trying to
  recover TRUNCATE. The tests will be enabled when we push the
  bundle of WLs(7141/7016/7743), which will move the storing of
  InnoDB DD to the common data dictionary, and make DDL atomic.
  Until which recovery after killing the server will also not

* MySQL Cluster cannot be used with this version of the server.

H. Disabled tests:
Apart from test that are disabled due to above functionality that
does not work now. Additionally we have temporarily disabled a
few other tests, which might be a bit challenging for bug fixing
in trunk in very restricted areas of the code. Refer
sql/dd/mtr_readme.txt to see which exact test cases are disabled.
  • Loading branch information...
Gopal Shankar
Gopal Shankar committed Nov 3, 2015
1 parent 246c3ea commit 31350e8ab15179acab5197fa29d12686b1efd6ef
Showing 814 changed files with 58,319 additions and 25,434 deletions.
@@ -978,12 +978,10 @@ class Process_writer
errno= 0;
info << "Creating system tables...";
- string create_db("CREATE DATABASE mysql;\n");
string use_db("USE mysql;\n");
// ssize_t write() may be declared with attribute warn_unused_result
- size_t w1= write(fh, create_db.c_str(), create_db.length());
- size_t w2= write(fh, use_db.c_str(), use_db.length());
- if (w1 != create_db.length() || w2 != use_db.length())
+ size_t w= write(fh, use_db.c_str(), use_db.length());
+ if (w != use_db.length())
info << "failed." << endl;
return false;
@@ -1042,17 +1040,17 @@ class Process_writer
<< endl;
string create_user_cmd;
- w1= write(fh, create_user_cmd.c_str(), create_user_cmd.length());
- if (w1 !=create_user_cmd.length() || errno != 0)
+ w= write(fh, create_user_cmd.c_str(), create_user_cmd.length());
+ if (w !=create_user_cmd.length() || errno != 0)
return false;
info << "Creating default proxy " << m_user->user << "@"
<< m_user->host
<< endl;
Proxy_user proxy_user(m_user->host, m_user->user);
string create_proxy_cmd;
- w1= write(fh, create_proxy_cmd.c_str(), create_proxy_cmd.length());
- if (w1 != create_proxy_cmd.length() || errno != 0)
+ w= write(fh, create_proxy_cmd.c_str(), create_proxy_cmd.length());
+ if (w != create_proxy_cmd.length() || errno != 0)
return false;
if (!opt_skipsys)
@@ -1656,7 +1654,7 @@ int main(int argc,char *argv[])
if (opt_def_extra_file != NULL)
- command_line.push_back(string("--bootstrap"));
+ command_line.push_back(string("--install-server"));
@@ -1,4 +1,4 @@
-# Copyright (c) 2009, 2014, Oracle and/or its affiliates. All rights reserved.
+# Copyright (c) 2009, 2014, 2015 Oracle and/or its affiliates. All rights reserved.
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
@@ -44,7 +44,7 @@ MACRO (INSTALL_DEBUG_SYMBOLS targets)
SET(comp Server)
SET(comp Debuginfo)
# No .pdb file for static libraries.
INSTALL(FILES ${pdb_location}
@@ -85,11 +85,19 @@ enum ha_rkey_function {
/* Key algorithm types */
enum ha_key_alg {
- HA_KEY_ALG_UNDEF= 0, /* Not specified (old file) */
- HA_KEY_ALG_BTREE= 1, /* B-tree, default one */
+ /**
+ Used for cases when key algorithm which is supported by SE can't be
+ described by one of other classes from this enum (@sa Federated,
+ PerfSchema SE, @sa dd::Index::IA_SE_SPECIFIC).
+ @note Assigned as default value for key algorithm by parser, replaced by
+ SEs default algorithm for keys in mysql_prepare_create_table().
+ */
+ HA_KEY_ALG_BTREE= 1, /* B-tree. */
HA_KEY_ALG_RTREE= 2, /* R-tree, for spatial searches */
- HA_KEY_ALG_HASH= 3, /* HASH keys (HEAP tables) */
+ HA_KEY_ALG_HASH= 3, /* HASH keys (HEAP, NDB). */
/* Storage media types */
@@ -197,7 +205,11 @@ enum ha_extra_function {
/** Do secondary sort by handler::ref (rowid) after key sort. */
+ /*
+ Skip Serializable isolation level on Views on DD tables.
+ This will make reads on DD Views non blocking */
/* Compatible option, to be deleted in 6.0 */
@@ -240,34 +252,50 @@ enum ha_base_keytype {
#define HA_MAX_KEYTYPE 31 /* Must be log2-1 */
- /* These flags kan be OR:ed to key-flag */
+ Flags for KEY::flags bitmap.
+ Also used for similar bitmaps in storage engines (HP_KEYDEF::flag,
+ MI_KEYDEF::flag, ...).
+/** Do not allow duplicate records. */
+#define HA_NOSAME 1
+/** Pack string key to previous key (optimization supported by MyISAM). */
+#define HA_PACK_KEY 2
+ Auto-increment key.
+ @note Not used by SQL-layer/ for KEY::flags. Only set by MyISAM and
+ Heap SEs in MI/HP_KEYDEF::flag.
+#define HA_AUTO_KEY 16
+/** Packing of all keys to previous key (optimization supported by MyISAM). */
+#define HA_BINARY_PACK_KEY 32
+/** Full-text key. */
+#define HA_FULLTEXT 128
+ Flag in MI_KEYDEF::flag which marks MyISAM's "uniques".
-#define HA_NOSAME 1 /* Set if not dupplicated records */
-#define HA_PACK_KEY 2 /* Pack string key to previous key */
-#define HA_AUTO_KEY 16
-#define HA_BINARY_PACK_KEY 32 /* Packing of all keys to prev key */
-#define HA_FULLTEXT 128 /* For full-text search */
-#define HA_UNIQUE_CHECK 256 /* Check the key for uniqueness */
-#define HA_SPATIAL 1024 /* For spatial search */
-#define HA_NULL_ARE_EQUAL 2048 /* NULL in key are cmp as equal */
-#define HA_GENERATED_KEY 8192 /* Automaticly generated key */
+ @note Internal to MyISAM. Current server doesn't use this feature.
+#define HA_UNIQUE_CHECK 256
+/** Spatial key. */
+#define HA_SPATIAL 1024
+ NULLs in key are compared as equal.
+ @note Used only for internal temporary tables created by optimizer.
+#define HA_NULL_ARE_EQUAL 2048
+/** Key was automatically created to support Foreign Key constraint. */
+#define HA_GENERATED_KEY 8192
/* The combination of the above can be used for key type comparison. */
- Key contains partial segments.
- This flag is internal to the MySQL server by design. It is not supposed
- neither to be saved in FRM-files, nor to be passed to storage engines.
- It is intended to pass information into internal static sort_keys(KEY *,
- KEY *) function.
- This flag can be calculated -- it's based on key lengths comparison.
-#define HA_KEY_HAS_PART_KEY_SEG 65536
Key was renamed (or is result of renaming a key).
@@ -283,15 +311,36 @@ enum ha_base_keytype {
/** Set if a key is on any virtual generated columns */
#define HA_VIRTUAL_GEN_KEY (1 << 18)
- /* Automatic bits in key-flag */
+ Bits in KEY::flags, MI/HP_KEYDEF::flag which are automatically
+ calculated based on other flags/members in these structures
+ (often from info about key parts).
+/** Some key part packs space. Internal to MyISAM. */
+/** Some key part has variable length. Internal to MyISAM and Heap engines. */
+#define HA_VAR_LENGTH_KEY 8
+/** Some key part is nullable. */
+#define HA_NULL_PART_KEY 64
+/** Internal bit used when sorting records. Internal to MyISAM. */
+#define HA_SORT_ALLOWS_SAME 512
+/** Key has comment. */
+#define HA_USES_COMMENT 4096
+/** Fulltext index uses [pre]parser */
+#define HA_USES_PARSER 16384
+/** Key uses KEY_BLOCK_SIZE option. */
+#define HA_USES_BLOCK_SIZE 32768
+ Key contains partial segments.
+ @note This flag is internal to SQL-layer by design. It is not supposed to
+ be used to storage engines. It is intended to pass information into
+ internal static sort_keys(KEY *, KEY *) function.
-#define HA_SPACE_PACK_USED 4 /* Test for if SPACE_PACK used */
-#define HA_VAR_LENGTH_KEY 8
-#define HA_NULL_PART_KEY 64
-#define HA_USES_COMMENT 4096
-#define HA_USES_PARSER 16384 /* Fulltext index uses [pre]parser */
-#define HA_USES_BLOCK_SIZE ((uint) 32768)
-#define HA_SORT_ALLOWS_SAME 512 /* Intern bit when sorting records */
+ This flag can be calculated -- it's based on key lengths comparison.
+#define HA_KEY_HAS_PART_KEY_SEG 65536
/* These flags can be added to key-seg-flag */
@@ -310,19 +359,111 @@ enum ha_base_keytype {
#define HA_BIT_PART 1024
- /* optionbits for database */
+ Flags for HA_CREATE_INFO::table_options and TABLE_SHARE::db_create_options
+ TABLE_SHARE::db_options_in_use bitmaps.
+ @note These bitmaps are used for storing information about some table
+ option values/attributes.
+ @note HA_CREATE_INFO::table_options and TABLE_SHARE::db_create_options
+ are basically the same bitmap at the time of table creation and
+ at the time of table opening/usage correspondingly.
+ @note TABLE_SHARE::db_options_in_use is normally copy of db_create_options
+ but can be overriden by SE. E.g. MyISAM does this at handler::open()
+ and hander::info() time.
+ Also used for similar bitmaps in MyISAM (MYISAM_SHARE/MI_ISAMINFO::options).
+ Indicates that storage engine needs to use packed row format.
+ Set for tables with ROW_FORMAT=DYNAMIC clause, for tables with BLOB fields,
+ and for tables with VARCHAR columns without ROW_FORMAT=FIXED.
+ This flag is respected by MyISAM only (it might also decide to use this
+ optimization for its own reasons). InnoDB relies on HA_CREATE_INFO::row_type
+ directly instead.
+ PACK_KEYS=1 option was specified.
+ PACK_KEYS=# option specifies whether key packing - optimization supported
+ by MyISAM, should be used.
+ * PACK_KEYS=1 means all keys should be packed,
+ * PACK_KEYS=0 (denoted by @sa HA_OPTION_NO_PACK_KEYS flag) means that key
+ packing should not be used at all.
+ * Not using this option or using PACK_KEYS=DEFAULT clause (denoted by
+ absence of both HA_OPTION_PACK_KEYS and HA_OPTION_NO_PACK_KEYS flags)
+ means that key packing will be used for long string columns.
+ Flag indicating that table is compressed. Used by MyISAM storage engine to
+ tell SQL-layer that tables is compressed. Not set or stored by SQL-layer,
+ MyISAM doesn't respect ROW_FORMAT=COMPRESSED clause and doesn't allow direct
+ creation of compressed tables. Existing tables can be compressed by external
+ tool. This tool marks such tables with HA_OPTION_COMPRESS_RECORD flag in
+ MYISAM_SHARE/MI_ISAMINFO::options. Then storage engine sets this flag in
+ TABLE_SHARE::db_options_in_use to let SQL-layer know about the fact. It is
+ never set in HA_CREATE_INFO::table_options/TABLE_SHARE::db_create_options.
-#define HA_OPTION_LONG_BLOB_PTR 8 /* new ISAM format */
+ Unused. Formerly HA_OPTION_LONG_BLOB_PTR - new ISAM format/portable
+ BLOB pointers.
+#define HA_OPTION_UNUSED1 8
+ Storage engine (MyISAM) internal flag for marking temporary tables.
+ Used in MYISAM_SHARE/MI_ISAMINFO::options, not used by SQL-layer/
+ in HA_CREATE_INFO::table_options/TABLE_SHARE::db_create_options.
+ CHECKSUM=1 option was specified.
+ This option enables live checksumming for MyISAM tables. Not used by InnoDB.
+ DELAY_KEY_WRITE=1 option was specified. This option enables MyISAM optimization
+ which postpones key updates until table is closed/expelled from the table cache.
-#define HA_OPTION_NO_PACK_KEYS 128 /* Reserved for MySQL */
+ PACK_KEYS=0 option was specified.
+ @sa HA_OPTION_PACK_KEYS for further description.
+ @note Unlike HA_OPTION_PACK_KEYS this flag is SQL-layer only.
+ Flag specific to table creation/HA_CREATE_INFO::table_options.
+ Indicates that storage engine instead of creating new table, needs
+ to discover existing one (which metadata was discovered from SE
+ earlier).
+ Not used in TABLE_SHARE::db_create_options/db_options_in_use.
+ Storage engine (MyISAM) internal flag for marking tables which
+ rely on SQL-layer because they have keys using fulltext parser plugin.
+ Used in MYISAM_SHARE/MI_ISAMINFO::options, not used by SQL-layer/
+ in HA_CREATE_INFO::table_options/TABLE_SHARE::db_create_options.
+/** Unused. Formerly HA_OPTION_NULL_FIELDS - reserved for Maria SE. */
+#define HA_OPTION_UNUSED2 1024
+/** Unused. Formerly HA_OPTION_PAGE_CHECKSUM - reserved for Maria SE. */
+#define HA_OPTION_UNUSED3 2048
/** STATS_PERSISTENT=1 has been specified in the SQL command (either CREATE
or ALTER TABLE). Table and index statistics that are collected by the
storage engine and used by the optimizer for query optimization will be
@@ -335,8 +476,31 @@ HA_OPTION_NO_STATS_PERSISTENT is set, this means that the setting is not
explicitly set at table level and the corresponding table will use whatever
is the global server default. */
-#define HA_OPTION_TEMP_COMPRESS_RECORD ((uint) 16384) /* set by isamchk */
-#define HA_OPTION_READ_ONLY_DATA ((uint) 32768) /* Set by isamchk */
+ MyISAM internal flag used by myisamchk external tool.
+ Not used by SQL-layer/in HA_CREATE_INFO::table_options and
+ TABLE_SHARE::db_create_options.
+ MyISAM internal flag which marks table as read-only.
+ Set by myisampack external tool.
+ Not used by SQL-layer/in HA_CREATE_INFO::table_options and
+ TABLE_SHARE::db_create_options.
+ CHECKSUM=0 option was specified. Only used by SQL-layer, in
+ HA_CREATE_INFO::table_options. Not persisted in data-dictionary.
+#define HA_OPTION_NO_CHECKSUM (1L << 17)
+ DELAY_KEY_WRITE=0 option was specified. Only used by SQL-layer, in
+ HA_CREATE_INFO::table_options. Not persisted in data-dictionary.
+#define HA_OPTION_NO_DELAY_KEY_WRITE (1L << 18)
/* Bits in flag to create() */
Oops, something went wrong.

0 comments on commit 31350e8

Please sign in to comment.