[Bug] python client批量数据写入数据丢失问题

[firebinhuo]

Search before asking

• I searched in the issues and found nothing similar.

Version

OS: Ubuntu-server20.04.6
IOTDB version: 2.0.1-beta

Describe the bug and provide the minimal reproduce step

np_tablet = NumpyTablet(dev_id, [measurement], [TSDataType.FLOAT], np_values, np_tt)，
批量写入数据大小为80000的数据包，前5次正常，第六次写入数据后数据长度变为465328，丢失部分数据；即使将80000再进行分批次插入，也会出现上述问题

What did you expect to see?

show me the reason, or better solution

What did you see instead?

no idea

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[github-actions]

Hi, this is your first issue in IoTDB project. Thanks for your report. Welcome to join the community!

[HTHou]

可以提供一下完整复现代码吗

[firebinhuo]

可以提供一下完整复现代码吗

 存储数据点,16个通道数据
self.data_list = [[] for _ in range(16)]
 存储时间戳
self.tt_list = [[] for _ in range(16)]

self.mqtt2iot.insert_tablet(f"root.mqtt.{str(msg.topic).split('/')[2]}.ch{index}", 'voltage', self.data_list[index], self.tt_list[index])
其中，len(self,data_list[index])，len(self,tt_list[index])为80000,

`def insert_tablet(self, dev_id, measurement, data: list, tt: list):

    np_values = [
        np.array(data, TSDataType.FLOAT.np_dtype())
    ]

    np_tt = np.array(tt, TSDataType.INT64.np_dtype())

    bk_size = 500

    if len(data) < bk_size:
        bk_size = len(data)
    for i in range(0,len(tt),bk_size):
        tem_value = np_values[0][i:i+bk_size]
        tem_tt = np_tt[i:i+bk_size]
        np_tablet = NumpyTablet(dev_id, [measurement], [TSDataType.FLOAT], [tem_value], tem_tt)
        try:
            self.session.insert_tablet(np_tablet)
        except Exception as e:
            print(e)`

[HTHou]

使用下面的代码没有复现

session.open(False)
for j in range(6):
    data = np.full(80000, 1.1, dtype=float)
    np_values = [
        np.array(data, TSDataType.FLOAT.np_dtype())
    ]
    tt = np.arange(j * 80000, (j + 1) * 80000)
    np_tt = np.array(tt, TSDataType.INT64.np_dtype())

    bk_size = 500

    if len(data) < bk_size:
        bk_size = len(data)
    for i in range(0, len(tt), bk_size):
        tem_value = np_values[0][i:i + bk_size]
        tem_tt = np_tt[i:i + bk_size]
        np_tablet = NumpyTablet("root.sg.d1", ["measurement"], [TSDataType.FLOAT],
                                [tem_value], tem_tt)
        try:
            session.insert_tablet(np_tablet)
        except Exception as e:
            print(e)
session.close()

查询count显示数据量没问题

IoTDB> select count(*) from root.**
+-----------------------------+
|count(root.sg.d1.measurement)|
+-----------------------------+
|                       480000|
+-----------------------------+
Total line number = 1

建议检查一下写入的单个序列是否存在相同的时间戳，时间戳在一个时间序列中相当于主键，写入相同的时间戳相当于update，不会增加count。

[firebinhuo]

可以提供一下完整复现代码吗

def insert_tablet(self, dev_id, measurement, data: list, tt: list):

    np_values = [
        np.array(data, TSDataType.FLOAT.np_dtype())
    ]

    np_tt = np.array(tt, TSDataType.INT64.np_dtype())

    bk_size = 500

    if len(data) < bk_size:
        bk_size = len(data)
    for i in range(0,len(tt),bk_size):
        tem_value = np_values[0][i:i+bk_size]
        tem_tt = np_tt[i:i+bk_size]
        np_tablet = NumpyTablet(dev_id, [measurement], [TSDataType.FLOAT], [tem_value], tem_tt)
        try:
            self.session.insert_tablet(np_tablet)
        except Exception as e:
            print(e)

self.mqtt2iot.insert_tablet(f"root.mqtt.{str(msg.topic).split('/')[2]}.ch{index}", 'voltage', self.data_list[index], self.tt_list[index])

[firebinhuo]

大佬，非常感谢！