Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Choose the binary data type encoding in the sink-connector #340

Closed
aadant opened this issue Oct 25, 2023 · 3 comments
Closed

Choose the binary data type encoding in the sink-connector #340

aadant opened this issue Oct 25, 2023 · 3 comments
Assignees
Labels
enhancement New feature or request GA-1 All the issues that are issues in release(Scheduled Dec 2023) high-priority qa-verified label to mark issues that were verified by QA

Comments

@aadant
Copy link
Collaborator

aadant commented Oct 25, 2023

The default encoding for binary is currently hex but base64 returns more compact strings.

this should be specified in #323

A default encoding would be exposed in the config.yml

@aadant
Copy link
Collaborator Author

aadant commented Oct 26, 2023

in the config.yml we would like to expose clickhouser.server.binary_datatype_encoding=base64

@subkanthi
Copy link
Collaborator

(https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-binary-handling-mode)

[binary.handling.mode](https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-binary-handling-mode)

bytes

Specifies how binary columns, for example, blob, binary, varbinary, should be represented in change events. Possible settings:

bytes represents binary data as a byte array.

base64 represents binary data as a base64-encoded String.

base64-url-safe represents binary data as a base64-url-safe-encoded String.

hex represents binary data as a hex-encoded (base16) String.

@subkanthi
Copy link
Collaborator

mysql> INSERT INTO `binary_types_BLOB` VALUES ('BLOB',0xD3D530AC844A2F7347B20DFFEAAC218026E01F838380DE086BB0BA6105E73382DC8346A210D7968DFFFD5438D4B7D3415B422464815F5A129BD6D86DA649200FD6FD2BDB4A072561934EF681BE5D5AF07DDCDF89B3B8352DBFDF5EF7846FA3232F5AF99D2D964955B4384B757B6D444C9CA85BE8E3424AA454CA3728BF7405B4DB62ABDBFE4EC4BD6CF5CD575D251D6035E93113B1F0A89CB1DDE1652D55714E34F3E5C728C8D95A0DF7830DD4E3428875E4DC84B0A3E007E4E3D3E822F072BF12DD56EF756F46CF780107FF06EDB31D19FE9C523437BE917C3199F24F32F3460188ED9AA6CF416B44360CEC6151183B95E2DF536DE7E18B558BBAA90404B493,NULL);
Query OK, 1 row affected (0.00 sec)

4af5a6d291d9 :) select * from binary_types_BLOB;

SELECT *
FROM binary_types_BLOB

Query id: 89ca990b-a2cd-40be-b6b6-e3e4a2d91a50

┌─Type─┬─Value──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─Null_Value─┬────────────_version─┬─is_deleted─┐
│ BLOB │ d3d530ac844a2f7347b20dffeaac218026e01f838380de086bb0ba6105e73382dc8346a210d7968dfffd5438d4b7d3415b422464815f5a129bd6d86da649200fd6fd2bdb4a072561934ef681be5d5af07ddcdf89b3b8352dbfdf5ef7846fa3232f5af99d2d964955b4384b757b6d444c9ca85be8e3424aa454ca3728bf7405b4db62abdbfe4ec4bd6cf5cd575d251d6035e93113b1f0a89cb1dde1652d55714e34f3e5c728c8d95a0df7830dd4e3428875e4dc84b0a3e007e4e3d3e822f072bf12dd56ef756f46cf780107ff06edb31d19fe9c523437be917c3199f24f32f3460188ed9aa6cf416b44360cec6151183b95e2df536de7e18b558bbaa90404b493 │ ᴺᵁᴸᴸ       │ 1732799481902006367 │          0 │
└──────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────┴─────────────────────┴────────────┘

With binary.handling.mode: base64

95f3bfcd2cd2 :) select * from binary_types_BLOB;

SELECT *
FROM binary_types_BLOB

Query id: 5dc15609-87f6-4420-ad98-42f586bc3170

┌─Type─┬─Value──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬─Null_Value─┬────────────_version─┬─is_deleted─┐
│ BLOB │ 09UwrIRKL3NHsg3/6qwhgCbgH4ODgN4Ia7C6YQXnM4Lcg0aiENeWjf/9VDjUt9NBW0IkZIFfWhKb1thtpkkgD9b9K9tKByVhk072gb5dWvB93N+Js7g1Lb/fXveEb6MjL1r5nS2WSVW0OEt1e21ETJyoW+jjQkqkVMo3KL90BbTbYqvb/k7EvWz1zVddJR1gNekxE7HwqJyx3eFlLVVxTjTz5ccoyNlaDfeDDdTjQoh15NyEsKPgB+Tj0+gi8HK/Et1W73VvRs94AQf/Bu2zHRn+nFI0N76RfDGZ8k8y80YBiO2aps9Ba0Q2DOxhURg7leLfU23n4YtVi7qpBAS0kw== │ ᴺᵁᴸᴸ       │ 1732801767797686330 │          0 │
└──────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────┴─────────────────────┴────────────┘

@subkanthi subkanthi added the GA-1 All the issues that are issues in release(Scheduled Dec 2023) label Dec 18, 2023
@Selfeer Selfeer added the qa-verified label to mark issues that were verified by QA label Feb 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request GA-1 All the issues that are issues in release(Scheduled Dec 2023) high-priority qa-verified label to mark issues that were verified by QA
Projects
None yet
Development

No branches or pull requests

4 participants