Json type 150 is not handled

[hgwat]

Received this error whilst listening to binlogs in AWS RDS:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "./main.py", line 25, in startDBListener
    "secret": gfire
  File "/usr/src/app/database_listener.py", line 47, in listen
    for row in binlogevent.rows:
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/row_event.py", line 427, in rows
    self._fetch_rows()
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/row_event.py", line 422, in _fetch_rows
    self.__rows.append(self._fetch_one_row())
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/row_event.py", line 511, in _fetch_one_row
    row["before_values"] = self._read_column_data(self.columns_present_bitmap)
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/row_event.py", line 181, in _read_column_data
    values[name] = self.packet.read_binary_json(column.length_size)
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/packet.py", line 352, in read_binary_json
    return self.read_binary_json_type(t, length)
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/packet.py", line 385, in read_binary_json_type
    raise ValueError('Json type %d is not handled' % t)
ValueError: Json type 150 is not handled

[sumanau7]

@hgwat Do you have a reproducible example ?

[SriSivaC]

Received this error whilst listening to binlogs in AWS RDS:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "./main.py", line 25, in startDBListener
    "secret": gfire
  File "/usr/src/app/database_listener.py", line 47, in listen
    for row in binlogevent.rows:
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/row_event.py", line 427, in rows
    self._fetch_rows()
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/row_event.py", line 422, in _fetch_rows
    self.__rows.append(self._fetch_one_row())
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/row_event.py", line 511, in _fetch_one_row
    row["before_values"] = self._read_column_data(self.columns_present_bitmap)
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/row_event.py", line 181, in _read_column_data
    values[name] = self.packet.read_binary_json(column.length_size)
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/packet.py", line 352, in read_binary_json
    return self.read_binary_json_type(t, length)
  File "/usr/local/lib/python3.7/site-packages/pymysqlreplication/packet.py", line 385, in read_binary_json_type
    raise ValueError('Json type %d is not handled' % t)
ValueError: Json type 150 is not handled

Hi @hgwat , I am python developer and am interested to help.

[hgwat]

Hi @SriSivaC , I think it may be an issue with JSON serialising date formats. Here's a related issue on stackoverflow: https://stackoverflow.com/questions/11875770/how-to-overcome-datetime-datetime-not-json-serializable. Do you know if datetime serialisation is handled in the above error?

[KiranPlastiq]

Hi @hgwat @SriSivaC I don't think it's date time serialization issue. I also hit this issue but only for some records in my table. All other records flowing through well... This also happens only for JSON field. Non-JSON fields are still getting streamed.

I am trying to get more examples so that i can pinpoint the issue. As of now it happened only for a couple of records and there is not enough logging in those records.

[laminko]

I am also having same issue but with different type:

2021-08-08 15:08:33 MainProcess ERROR: Stack trace: Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/chameleon/lib/python3.8/site-packages/pg_chameleon/lib/global_lib.py", line 496, in read_replica
    self.mysql_source.read_replica()
  File "/home/ubuntu/miniconda3/envs/chameleon/lib/python3.8/site-packages/pg_chameleon/lib/mysql_lib.py", line 1453, in read_replica
    replica_data=self.__read_replica_stream(batch_data)
  File "/home/ubuntu/miniconda3/envs/chameleon/lib/python3.8/site-packages/pg_chameleon/lib/mysql_lib.py", line 1311, in __read_replica_stream
    for row in binlogevent.rows:
  File "/home/ubuntu/miniconda3/envs/chameleon/lib/python3.8/site-packages/pymysqlreplication/row_event.py", line 443, in rows
    self._fetch_rows()
  File "/home/ubuntu/miniconda3/envs/chameleon/lib/python3.8/site-packages/pymysqlreplication/row_event.py", line 438, in _fetch_rows
    self.__rows.append(self._fetch_one_row())
  File "/home/ubuntu/miniconda3/envs/chameleon/lib/python3.8/site-packages/pymysqlreplication/row_event.py", line 529, in _fetch_one_row
    row["after_values"] = self._read_column_data(self.columns_present_bitmap2)
  File "/home/ubuntu/miniconda3/envs/chameleon/lib/python3.8/site-packages/pymysqlreplication/row_event.py", line 191, in _read_column_data
    values[name] = self.packet.read_binary_json(column.length_size)
  File "/home/ubuntu/miniconda3/envs/chameleon/lib/python3.8/site-packages/pymysqlreplication/packet.py", line 350, in read_binary_json
    return self.read_binary_json_type(t, length)
  File "/home/ubuntu/miniconda3/envs/chameleon/lib/python3.8/site-packages/pymysqlreplication/packet.py", line 383, in read_binary_json_type
    raise ValueError('Json type %d is not handled' % t)
ValueError: Json type 23 is not handled

[KiranPlastiq]

@laminko My issue has been resolved after removing some tables. One of the table contained a JSON column in the MySQL database and I used to get this error when the content of that column going to extend a certain limit such as more than 1 MB like that but don't exactly know the size where it throws the error. But, once after I removed those tables from streaming, haven't hit this issue later. So, seems that even though JSON field can store more data, the replication package has some limit

[laminko]

@laminko My issue has been resolved after removing some tables. One of the table contained a JSON column in the MySQL database and I used to get this error when the content of that column going to extend a certain limit such as more than 1 MB like that but don't exactly know the size where it throws the error. But, once after I removed those tables from streaming, haven't hit this issue later. So, seems that even though JSON field can store more data, the replication package has some limit

Yeah, agree. I noticed the issue happened on the table with json columns.

[oseemann]

Hey I can reproduce this class of issues by inserting a large json document (>64kb).

In this case it's creating AssertionError: Result length not requested length: Expected=2. Actual=0. Position: 500054. Data Length: 500054, but (with a different payload than the one below) I've also seen large json docs trigger the "Json type %d is not handled" errors.

Tested with:

• MySQL 5.7.36-log
• mysql-replication 0.27
• pymysql 0.9.3
• Python 3.7.10

Example code:

CREATE TABLE t1 (a INT, b JSON);

import json
import pymysql

conn = pymysql.connect(host='localhost', port=3306, user='myuser', password='mypassword',
                       database='mydb', cursorclass=pymysql.cursors.DictCursor)

large_array = json.dumps([i for i in range(100000)])

cur = conn.cursor()
sql = "INSERT INTO `t1` (`a`, `b`) VALUES (%s, %s)"
cur.execute(sql, (1, large_array))
conn.commit()

The issue does not occur when the following patch is applied:

diff --git a/pymysqlreplication/packet.py b/pymysqlreplication/packet.py
index a5a7c0f..5277216 100644
--- a/pymysqlreplication/packet.py
+++ b/pymysqlreplication/packet.py
@@ -396,9 +396,9 @@ class BinLogPacketWrapper(object):
         elif t == JSONB_TYPE_UINT16:
             return self.read_uint32() if large else self.read_uint16()
         elif t == JSONB_TYPE_INT32:
-            return self.read_int64() if large else self.read_int32()
+            return self.read_int32()
         elif t == JSONB_TYPE_UINT32:
-            return self.read_uint64() if large else self.read_uint32()
+            return self.read_uint32()

         raise ValueError('Json type %d is not handled' % t)

This code was last changed in 0415f4a. I noticed that the "if large" is needed for the int16/uint16 case, but I don't think the read_uint64() is required when large.

I'll start looking at 👇 those, trying to understand what's going on
https://github.com/mysql/mysql-server/blob/5.7/sql/json_binary.cc#L416
https://github.com/mysql/mysql-server/blob/5.7/sql/json_binary.cc#L183

But maybe you have an idea as well, pinging @baloo and @mascah who also touched this function.