fix - Bug/increase history list length

[chungeun-choi]

Overview

issue number #430

When using the pymysqlreplication package, there was a recurring issue with the history list length value under the following circumstances:

• MySQL version 8.x or higher.
• Continuous execution of DML (Data Manipulation Language) statements.

This problem led to a gradual slowing down of SELECT queries.

For detailed information, you can refer to the following reference:

https://minervadb.xyz/troubleshooting-innodb-history-length-with-hung-mysql-transaction/

As a result, I investigated the problem area and made necessary modifications. After making the changes, you were able to compare the performance before and after the fix, using monitoring tools such as Prometheus and MySQL Exporter.

Fixed

• Modified code

  def __get_table_information(self, schema, table):
          for i in range(1, 3):
              try:
                  if not self.__connected_ctl:
                      self.__connect_to_ctl()
  
                  cur = self._ctl_connection.cursor()
                  cur.execute("""
                      SELECT
                          COLUMN_NAME, COLLATION_NAME, CHARACTER_SET_NAME,
                          COLUMN_COMMENT, COLUMN_TYPE, COLUMN_KEY, ORDINAL_POSITION,
                          DATA_TYPE, CHARACTER_OCTET_LENGTH
                      FROM
                          information_schema.columns
                      WHERE
  			  table_schema = %s AND table_name = %s
                      ORDER BY ORDINAL_POSITION
                      """, (schema, table))
                  result = cur.fetchall()
                  cur.close()
  
                  return result
  								...
  								...

Performance Comparison After the Code Modifications

• Before modification
  
  

• After modification
  
  

[chungeun-choi]

We also found and added parts that can be solved by adding autocommit settings

    def __connect_to_ctl(self):
        if not self._ctl_connection_settings:
            self._ctl_connection_settings = dict(self.__connection_settings)
        self._ctl_connection_settings["db"] = "information_schema"
        self._ctl_connection_settings["cursorclass"] = DictCursor
        self._ctl_connection_settings["autocommit"] = True # Changed
        self._ctl_connection = self.pymysql_wrapper(**self._ctl_connection_settings)
        self._ctl_connection._get_table_information = self.__get_table_information
        self.__connected_ctl = True

[dongwook-chan]

This PR makes it feasible for major corporations handling extensive traffic to utilize this library. It addresses potential query response lags and reduces the strain on database servers. One of the standout features of the replication protocol is its minimal impact on database performance, and this PR ensures the library capitalizes on that strength.

[julien-duponchelle]

Great , thanks for the investigation . Small fix but big problem indeed