Loss of 'float' type values precision in the Kafka sink using 'canal-json' protocol

[andriy-buryy]

What did you do?

1. Create TiCDC changefeed:

curl -X POST http://127.0.0.1:8300/api/v1/changefeeds \
  -H 'Content-Type: application/json' \
  -d '{
    "changefeed_id": "test-changefeed",
    "sink_uri": "kafka://kafka-0:9092/test-topic?protocol=canal-json&kafka-version=3.2.0",
    "filter_rules": ["test.test_table"]
  }'

2. Create table test_table

$ mysql -u root -h 127.0.0.1 -P 4000 test

CREATE TABLE test_table (
    id     int unsigned auto_increment primary key,
    name   varchar(64) not null,
    price  float
);

4. Insert record

INSERT INTO test_table (name, price) VALUES ('Product 1', 123.99);

5. Check data in the TiDB:

SELECT * FROM test_table;
+----+-----------+--------+
| id | name      | price  |
+----+-----------+--------+
|  1 | Product 1 | 123.99 |
+----+-----------+--------+

6. Check the INSERT event message value in the Kafka topic:

{
  "id": 0,
  "database": "test",
  "table": "test_table",
  "pkNames": [
    "id"
  ],
  "isDdl": false,
  "type": "INSERT",
  "es": 1678364595958,
  "ts": 1678364596360,
  "sql": "",
  "sqlType": {
    "id": 4,
    "name": 12,
    "price": 7
  },
  "mysqlType": {
    "id": "int unsigned",
    "name": "varchar",
    "price": "float"
  },
  "old": null,
  "data": [
    {
      "id": "1",
      "name": "Product 1",
      "price": "123.98999786376953"
    }
  ]
}

What did you expect to see?

The price field value in the Kafka message should be 123.99 (the same as in the database).

What did you see instead?

The price field value in the Kafka message is 123.98999786376953 (not the same as in the database).

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

Release Version: v6.6.0
Edition: Community
Git Commit Hash: f4ca0821fb96a2bdd37d2fb97eb26c07fc58d4e4
Git Branch: heads/refs/tags/v6.6.0
UTC Build Time: 2023-02-17 14:49:02
GoVersion: go1.19.5
Race Enabled: false
TiKV Min Version: 6.2.0-alpha
Check Table Before Drop: false
Store: tikv

Upstream TiKV version (execute tikv-server --version):

TiKV 
Release Version:   6.6.0
Edition:           Community
Git Commit Hash:   58d231bccb4fb188bb697905ee4faf086c4ac931
Git Commit Branch: heads/refs/tags/v6.6.0
UTC Build Time:    2023-02-18 08:44:45
Rust Version:      rustc 1.67.0-nightly (96ddd32c4 2022-11-14)
Enable Features:   pprof-fp jemalloc mem-profiling portable sse test-engine-kv-rocksdb test-engine-raft-raft-engine cloud-aws cloud-gcp cloud-azure
Profile:           dist_release

TiCDC version (execute cdc version):

Release Version: v6.6.0
Git Commit Hash: feae75ed4c816da5df52e931cb936a97f619ae0e
Git Branch: heads/refs/tags/v6.6.0
UTC Build Time: 2023-02-15 15:38:00
Go Version: go version go1.19.5 linux/amd64
Failpoint Build: false

[overvenus]

It might caused by the float rounding issue (lost precise, float64 to float32) , see https://go.dev/play/p/jwvumoA8aaF

[sdojjy]

ticdc mounter decode float as float64

tiflow/cdc/entry/mounter.go

Lines 458 to 465 in 2abe021

	case mysql.TypeFloat, mysql.TypeDouble:
	v := datum.GetFloat64()
	if math.IsNaN(v) || math.IsInf(v, 1) || math.IsInf(v, -1) {
	warn = fmt.Sprintf("the value is invalid in column: %f", v)
	v = 0
	}
	const sizeOfV = unsafe.Sizeof(v)
	return v, int(sizeOfV), warn, nil