Temporary table names may incorrectly think they have a schema

[MSS970]

Cacti [1.3.0 Dev]
Windows 2019

The following issues have been identified in lib/boost.php (function boost_process_poller_output())

1. Line 728 to 730:
   $timestamp = db_fetch_cell('SELECT MIN(UNIX_TIMESTAMP(start_time))
   FROM poller_time
   WHERE end_time="0000-00-00"');

Should be fixed by:

$timestamp = db_fetch_cell('SELECT MIN(UNIX_TIMESTAMP(start_time))
	FROM poller_time
	WHERE end_time LIKE "0000-00-00%"');

2. Line 742 to 744:
   $temp_table = 'poller_output_boost_temp_' . $local_data_id . '' . mt_rand() . '' . microtime(true);

    db_execute("CREATE TEMPORARY TABLE $temp_table LIKE poller_output_boost");
   

Should be fixed by:
$temp_table = 'poller_output_boost_temp';

	db_execute("CREATE TABLE IF NOT EXISTS $temp_table LIKE poller_output_boost");

3. Line 792:
   db_execute("DROP TEMPORARY TABLE $temp_table");

Should be fixed by:
db_execute("TRUNCATE TABLE $temp_table");

[TheWitness]

1. This is valid syntax, no reason to change:

Line 728 to 730:
$timestamp = db_fetch_cell('SELECT MIN(UNIX_TIMESTAMP(start_time))
FROM poller_time
WHERE end_time="0000-00-00"');

2. This the current dev is as follows, which does not match your example:

$temp_table = 'poller_output_boost_temp_' . $local_data_id . '' . mt_rand() . '' . microtime(true);

So, as mentioned before, I don't know where you are getting your lib/boost.php from, but develop it does not match your example.

3. I don't think this matters one way or another. Truncate by the way, is very expensive in InnoDB.

So, changes 1 & 2 are not required, and 3 is optional and not selected.

[TheWitness]

In every case, the number of records deleted would be maybe 1-30 records, so with MySQL or MariaDB, truncate vs. delete is likely nano seconds in difference in execution time.

[TheWitness]

Lastly, since we 'always' drop that table in a session, and since we are using temporary tables, the entropy should be fine, and there should never be a 'duplicate' table. So, IF NOT EXISTS is redundant. It would be required if we were using a static table name of course, which we are not.

There is a very low probability that if you had multiple threads it might be able to happen, but since the threads each have their own MySQL/MariaDB session, even if the tables were the same name, they would not see each other. If that were to happen, it would be a MySQL/MariaDB bug.

[TheWitness]

By default, we use persistent connections, but since there are a few microseconds between connections, the microtime() should be different or the mt_rand() should be different. So, the probability of a name space collision in a session is likely 1e-100.

[MSS970]

@TheWitness ,
I hope you are well and thank you for the response.

1. Line 728 to 730:
   $timestamp = db_fetch_cell('SELECT MIN(UNIX_TIMESTAMP(start_time))
   FROM poller_time
   WHERE end_time="0000-00-00"');

The data in the table does not match the where condition, below is a sample of the data in the table:

id, pid, poller_id, start_time, end_time

'342455', '1215', '1', '2024-10-06 11:30:01', '0000-00-00 00:00:00'
I have attached a screenshot of the data for reference.

So either of the following conditions is required:
WHERE end_time LIKE "0000-00-00%"
or
WHERE end_time="0000-00-00 00:00:00"

I recommend WHERE end_time LIKE "0000-00-00%" to cover both cases "000-00-00" and "0000-00-00 00:00:00".

2. As for the $temp_table variable, it is in github.com/cacti/cacti ==>> Develop located at lib/boost.php line 742

$temp_table = 'poller_output_boost_temp_' . $local_data_id . '' . mt_rand() . '' . microtime(true);
This line does not provide valid table name as the microtime(true) provides a float value that contains a dot (.), hence I have tried in one of the attempts to replace the dot with underscore so that the table name becomes valid.
So, I have tried to fix the invalid output of the $temp_table line, then I replaced it with direct simple assignment to the variable:
$temp_table = 'poller_output_boost_temp';

Based on the above, there was a misunderstanding between you and me on your statement "as mentioned before, I don't know where you are getting your lib/boost.php from, but develop it does not match your example", as I was replacing the dot with underscore.

Alternatively, we can modify the line 742 as follows:
$temp_table = 'poller_output_boost_temp_' . $local_data_id . '' . mt_rand() . '' . floor(microtime(true));
So that the microtime is rounded to integer.

In this case CREATE TEMPORARY TABLE can be used at line 744
db_execute("CREATE TEMPORARY TABLE $temp_table LIKE poller_output_boost");

However, the above has generated an error:
CMDPHP ERROR: A DB Exec Failed!, Error: InnoDB: Tablespace innodb_system cannot contain TEMPORARY tables.

Therefore, I still believe the right solution is to create static table:
$temp_table = 'poller_output_boost_temp';
db_execute("CREATE TABLE IF NOT EXISTS $temp_table LIKE poller_output_boost");

3. Based on the above, I agree on the cost of the DB time, however, the solution at line 792 may be one of the following:
   a. db_execute("DROP TABLE $temp_table");
   b. db_execute("TRUNCATE TABLE $temp_table");
   c. db_execute("DELETE FROM TABLE $temp_table");

Appreciate your input and support.
Regards,
MSS

[TheWitness]

The timestamp issue must be specific to MySQL 8.x. I think this is all due to idiosyncrasies between MySQL 8.x++ and MariaDB 10.x++, as there are no issues in MariaDB. It's a bit disturbing. And damn, the microtime issue is legit. Bummer.

[TheWitness]

Please share your my.ini/my.cnf file please.

[TheWitness]

@MSS970, Finding some real issues with MySQL 8. Ugh!

[MSS970]

@TheWitness
Below is the my.ini of cacti:

# Other default tuning values
# MySQL Server Instance Configuration File
# ----------------------------------------------------------------------
# Generated by the MySQL Server Instance Configuration Wizard
#
#
# Installation Instructions
# ----------------------------------------------------------------------
#
# On Linux you can copy this file to /etc/my.cnf to set global options,
# mysql-data-dir/my.cnf to set server-specific options
# (@localstatedir@ for this installation) or to
# ~/.my.cnf to set user-specific options.
#
# On Windows you should keep this file in the installation directory 
# of your server (e.g. C:\Program Files\MySQL\MySQL Server X.Y). To
# make sure the server reads the config file use the startup option 
# "--defaults-file". 
#
# To run the server from the command line, execute this in a 
# command line shell, e.g.
# mysqld --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
#
# To install the server as a Windows service manually, execute this in a 
# command line shell, e.g.
# mysqld --install MySQLXY --defaults-file="C:\Program Files\MySQL\MySQL Server X.Y\my.ini"
#
# And then execute this in a command line shell to start the server, e.g.
# net start MySQLXY
#
#
# Guidelines for editing this file
# ----------------------------------------------------------------------
#
# In this file, you can use all long options that the program supports.
# If you want to know the options a program supports, start the program
# with the "--help" option.
#
# More detailed information about the individual options can also be
# found in the manual.
#
# For advice on how to change settings please see
# https://dev.mysql.com/doc/refman/8.0/en/server-configuration-defaults.html
#
#
# CLIENT SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by MySQL client applications.
# Note that only client applications shipped by MySQL are guaranteed
# to read this section. If you want your own MySQL client program to
# honor these values, you need to specify it as an option during the
# MySQL client library initialization.
#
[client]

# pipe=

# socket=MYSQL

port=3306

[mysql]
no-beep

# default-character-set=

# SERVER SECTION
# ----------------------------------------------------------------------
#
# The following options will be read by the MySQL Server. Make sure that
# you have installed the server correctly (see above) so it reads this 
# file.
#
# server_type=2
[mysqld]

# The next three options are mutually exclusive to SERVER_PORT below.
# skip-networking
# enable-named-pipe
# shared-memory

# shared-memory-base-name=MYSQL

# The Pipe the MySQL Server will use
# socket=MYSQL

# The TCP/IP Port the MySQL Server will listen on
port=3306

# Path to installation directory. All paths are usually resolved relative to this.
basedir="D:/cacti/mysql"

# Path to the database root
datadir="D:/cacti/mysql/db/data"

# The default character set that will be used when a new schema or table is
# created and no character set is defined
# character-set-server=

# The default authentication plugin to be used when connecting to the server
# default_authentication_plugin=mysql_native_password
authentication_policy=*,,

# The default storage engine that will be used when create new tables when
default_storage_engine=InnoDB

# Set the SQL mode to strict
sql_mode="STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_ZERO_DATE,NO_ZERO_IN_DATE,ERROR_FOR_DIVISION_BY_ZERO"

# General and Slow logging.
log-output=FILE

slow-query-log=1

slow_query_log_file="D:/cacti/mysql/log/CACTI1-slow.log"

# If a query takes longer than this many seconds, the server increments the Slow_queries status variable. 
# If the slow query log is enabled, the query is logged to the slow query log file. This value is measured in real time, 
# not CPU time, so a query that is under the threshold on a lightly loaded system might be above the threshold on a heavily loaded one. 
# The minimum and default values of long_query_time are 0 and 10, respectively. The maximum is 31536000, which is 365 days in seconds. 
# The value can be specified to a resolution of microseconds
long_query_time=15

# Error Logging.
#log-error="D:/cacti/mysql/log/CACTI1.err"

# ***** Group Replication Related *****
# Specifies the base name to use for binary log files. With binary logging
# enabled, the server logs all statements that change data to the binary
# log, which is used for backup and replication.


# ***** Group Replication Related *****
# Specifies the server ID. For servers that are used in a replication topology,
# you must specify a unique server ID for each replication server, in the
# range from 1 to 2^32 − 1. "Unique" means that each ID must be different
# from every other ID in use by any other source or replica.
server-id=1

# ***** Group Replication Related *****
# The host name or IP address of the replica to be reported to the source
# during replica registration. This value appears in the output of SHOW REPLICAS
# on the source server. Leave the value unset if you do not want the replica to
# register itself with the source.
# report_host=0.0

# NOTE: Modify this value after Server initialization won't take effect.
lower_case_table_names=1

# The maximum amount of concurrent sessions the MySQL server will
# allow. One of these connections will be reserved for a user with
# SUPER privileges to allow the administrator to login even if the
# connection limit has been reached.
max_connections = 10000

# The number of open tables for all threads. Increasing this value
# increases the number of file descriptors that mysqld requires.
# Therefore you have to make sure to set the amount of open files
# allowed to at least 4096 in the variable "open-files-limit" in
# section [mysqld_safe]
# Optimal value is:
# MAX(
#    (open_files_limit - 10 - max_connections) / 2,
#    400
#   )
#
table_open_cache=3000

# Maximum size for internal (in-memory) temporary tables. If a table
# grows larger than this value, it is automatically converted to disk
# based table This limitation is for a single table. There can be many
# of them.
tmp_table_size=690M

# How many threads we should keep in a cache for reuse. When a client
# disconnects, the client's threads are put in the cache if there aren't
# more than thread_cache_size threads from before.  This greatly reduces
# the amount of thread creations needed if you have a lot of new
# connections. (Normally this doesn't give a notable performance
# improvement if you have a good thread implementation.)
thread_cache_size=10

#*** MyISAM Specific options
# The maximum size of the temporary file MySQL is allowed to use while
# recreating the index (during REPAIR, ALTER TABLE or LOAD DATA INFILE.
# If the file-size would be bigger than this, the index will be created
# through the key cache (which is slower).
myisam_max_sort_file_size=100G

# The size of the buffer that is allocated when sorting MyISAM indexes
# during a REPAIR TABLE or when creating indexes with CREATE INDEX
# or ALTER TABLE.
myisam_sort_buffer_size=2G

# Size of the Key Buffer, used to cache index blocks for MyISAM tables.
# Do not set it larger than 30% of your available memory, as some memory
# is also required by the OS to cache rows. Even if you're not using
# MyISAM tables, you should still set it to 8-64M as it will also be
# used for internal temporary disk tables.
key_buffer_size=8M

# Size of the buffer used for doing full table scans of MyISAM tables.
# Allocated per thread, if a full scan is needed.
read_buffer_size=64K

read_rnd_buffer_size=256K

#*** INNODB Specific options ***
# The common part of the directory path for InnoDB system tablespace data files. The default value is the MySQL data directory. 
# The setting is concatenated with the innodb_data_file_path setting, unless that setting is defined with an absolute path.
# A trailing slash is required when specifying a value for innodb_data_home_dir. For example:
# innodb_data_home_dir = /path/to/myibdata/
innodb_data_home_dir="D:/cacti/mysql/db/innodb/"

# Defines the name, size, and attributes of InnoDB system tablespace data files. 
# If you do not specify a value for innodb_data_file_path, the default behavior is to create a single auto-extending data file, 
# slightly larger than 12MB, named ibdata1.
# The full syntax for a data file specification includes the file name, file size, autoextend attribute, and max attribute:
#
# file_name:file_size[:autoextend[:max:max_file_size]]
#
# File sizes are specified in kilobytes, megabytes, or gigabytes by appending K, M or G to the size value. 
# If specifying the data file size in kilobytes, do so in multiples of 1024. Otherwise, KB values are rounded to nearest megabyte (MB) boundary. 
# The sum of file sizes must be, at a minimum, slightly larger than 12MB.
# You can specify more than one data file using a semicolon-separated list.

#innodb_data_file_path=ibdata1:1G:autoextend:max:2G
innodb_data_file_path=ibdata1:1G;ibdata2:1G;ibdata3:1G:autoextend:max:2G
# Use this option if you have a MySQL server with InnoDB support enabled
# but you do not plan to use it. This will save memory and disk space
# and speed up some things.
# skip-innodb

# If set to 1, InnoDB will flush (fsync) the transaction logs to the
# disk at each commit, which offers full ACID behavior. If you are
# willing to compromise this safety, and you are running small
# transactions, you may set this to 0 or 2 to reduce disk I/O to the
# logs. Value 0 means that the log is only written to the log file and
# the log file flushed to disk approximately once per second. Value 2
# means the log is written to the log file at each commit, but the log
# file is only flushed to disk approximately once per second.
innodb_flush_log_at_trx_commit = 1

# The size of the buffer InnoDB uses for buffering log data. As soon as
# it is full, InnoDB will have to flush it to disk. As it is flushed
# once per second anyway, it does not make sense to have it very large
# (even with long transactions).
#innodb_log_buffer_size=16M
innodb_log_buffer_size=32M

# InnoDB, unlike MyISAM, uses a buffer pool to cache both indexes and
# row data. The bigger you set this the less disk I/O is needed to
# access data in tables. On a dedicated database server you may set this
# parameter up to 80% of the machine physical memory size. Do not set it
# too large, though, because competition of the physical memory may
# cause paging in the operating system.  Note that on 32bit systems you
# might be limited to 2-3.5G of user level memory per process, so do not
# set it too high.
innodb_buffer_pool_size=6G

# The path where InnoDB creates undo tablespaces. Typically used to place undo tablespaces on a different storage device.
# There is no default value (it is NULL). If the innodb_undo_directory variable is undefined, undo tablespaces are created in the data directory.
innodb_undo_directory="D:/cacti/mysql/db/undo"

# The directory path to the InnoDB redo log files
innodb_log_group_home_dir="D:/cacti/mysql/db/redo"

# Defines the amount of disk space occupied by redo log files
# The innodb_redo_log_capacity default value, which is 104857600 bytes (100MB). The maximum redo log capacity is 128GB.
# innodb_redo_log_capacity = 8589934592 is 8GB.
innodb_redo_log_capacity = 104857600

# Size of each log file in a log group. You should set the combined size
# of log files to about 25%-100% of your buffer pool size to avoid
# unneeded buffer pool flush activity on log file overwrite. However,
# note that a larger logfile size will increase the time needed for the
# recovery process.
# innodb_log_file_size=2G

innodb_doublewrite_dir="D:/cacti/mysql/db/innodb_dblwr"


innodb_temp_tablespaces_dir="D:/cacti/mysql/db/#innodb_temp"

# Number of threads allowed inside the InnoDB kernel. The optimal value
# depends highly on the application, hardware as well as the OS
# scheduler properties. A too high value may lead to thread thrashing.
innodb_thread_concurrency=10

# The increment size (in MB) for extending the size of an auto-extend InnoDB system tablespace file when it becomes full.
innodb_autoextend_increment=64

# The number of regions that the InnoDB buffer pool is divided into.
# For systems with buffer pools in the multi-gigabyte range, dividing the buffer pool into separate instances can improve concurrency,
# by reducing contention as different threads read and write to cached pages.
innodb_buffer_pool_instances=10

# Determines the number of threads that can enter InnoDB concurrently.
innodb_concurrency_tickets=5000

# Specifies how long in milliseconds (ms) a block inserted into the old sublist must stay there after its first access before
# it can be moved to the new sublist.
innodb_old_blocks_time=1000

# It specifies the maximum number of .ibd files that MySQL can keep open at one time. The minimum value is 10.
# Specifies the maximum number of files that InnoDB can have open at one time. The minimum value is 10. 
# If innodb_file_per_table is disabled, the default value is 300; otherwise, 
# the default value is 300 or the table_open_cache setting, whichever is higher.
innodb_open_files = 500

# When this variable is enabled, InnoDB updates statistics during metadata statements.
innodb_stats_on_metadata=0

# When innodb_file_per_table is enabled (the default in 5.6.6 and higher), InnoDB stores the data and indexes for each newly created table
# in a separate .ibd file, rather than in the system tablespace.
# innodb_file_per_table=1
innodb_file_per_table=0

# Use the following list of values: 0 for crc32, 1 for strict_crc32, 2 for innodb, 3 for strict_innodb, 4 for none, 5 for strict_none.
innodb_checksum_algorithm=0

# The number of outstanding connection requests MySQL can have.
# This option is useful when the main MySQL thread gets many connection requests in a very short time.
# It then takes some time (although very little) for the main thread to check the connection and start a new thread.
# The back_log value indicates how many requests can be stacked during this short time before MySQL momentarily
# stops answering new requests.
# You need to increase this only if you expect a large number of connections in a short period of time.
back_log=80

# If this is set to a nonzero value, all tables are closed every flush_time seconds to free up resources and
# synchronize unflushed data to disk.
# This option is best used only on systems with minimal resources.
flush_time=0

# The minimum size of the buffer that is used for plain index scans, range index scans, and joins that do not use
# indexes and thus perform full table scans.
join_buffer_size=256K

# The maximum size of one packet or any generated or intermediate string, or any parameter sent by the
# mysql_stmt_send_long_data() C API function.
max_allowed_packet=512M

# If more than this many successive connection requests from a host are interrupted without a successful connection,
# the server blocks that host from performing further connections.
max_connect_errors=100

# Changes the number of file descriptors available to mysqld.
# You should try increasing the value of this option if mysqld gives you the error "Too many open files".
# default value is 4000
open_files_limit=4000

# If you see many sort_merge_passes per second in SHOW GLOBAL STATUS output, you can consider increasing the
# sort_buffer_size value to speed up ORDER BY or GROUP BY operations that cannot be improved with query optimization
# or improved indexing.
sort_buffer_size = 3G

# The number of table definitions (from .frm files) that can be stored in the definition cache.
# If you use a large number of tables, you can create a large table definition cache to speed up opening of tables.
# The table definition cache takes less space and does not use file descriptors, unlike the normal table cache.
# The minimum and default values are both 400.
table_definition_cache=1400

# Specify the maximum size of a row-based binary log event, in bytes.
# Rows are grouped into events smaller than this size if possible. The value should be a multiple of 256.


# If the value of this variable is greater than 0, a replica synchronizes its master.info file to disk.
# (using fdatasync()) after every sync_source_info events.
sync_source_info=10000

# If the value of this variable is greater than 0, the MySQL server synchronizes its relay log to disk.
# (using fdatasync()) after every sync_relay_log writes to the relay log.
sync_relay_log=10000

# If the value of this variable is greater than 0, a replica synchronizes its relay-log.info file to disk.
# (using fdatasync()) after every sync_relay_log_info transactions.
sync_relay_log_info=10000

# Load mysql plugins at start."plugin_x ; plugin_y".
# plugin_load

# The TCP/IP Port the MySQL Server X Protocol will listen on.
loose_mysqlx_port=33060


default-time-zone=Asia/Dubai
character_set_server=utf8mb4
collation_server=utf8mb4_unicode_ci
max_heap_table_size = 2G
innodb_flush_log_at_timeout=3
innodb_read_io_threads=32
innodb_write_io_threads=16
innodb_io_capacity=5000
innodb_io_capacity_max=10000

log_timestamps=SYSTEM

tmpdir=D:/cacti/mysql/db/tmpdir
innodb_tmpdir=D:/cacti/mysql/db/tmpdir/innodb
replica_load_tmpdir=D:/cacti/mysql/db/tmpdir/replica


# binlog_expire_logs_seconds = 432000


max_binlog_size = 10M


# Secure File Priv.
secure-file-priv = "D:/cacti/mysql/db/uploads"
# secure-file-priv = ""


binlog_gtid_simple_recovery = false
skip-log-bin

[TheWitness]

@MSS970, what specific version of MySQL? I've got a Rocky9 setup, but want to match the release.

[MSS970]

@TheWitness , MySQL 8.0.32 on Windows 2019 server