Issue #8154 : Support binary data in bulk load transforms - #8162
Conversation
Serialize Hop Binary fields in each bulk loader using the format that engine's protocol already accepts (hex for text COPY/LOAD DATA, native bytes for SQL Server and Vertica).
|
Can we add some IT tests for the databases in the integration tests? Other than that, looks like a solid addition! |
|
I couldn't immediately see a non-spammy way of doing it. It's tricky from an IT test-project perspective but I'll try to use |
|
Ah, I see the issue, as we can't include binary data in our source code. |
|
It's not a bad idea but as you can see we now already have a bunch of prrojects that need the bulk loader test. So we'd be copying files all over the place to make it work properly in the docker containers. Anyway, I don't think generating the binary data is as big of an issue as I thought it was yesterday. I'll figure it out. |
Serialize Binary data-set CSV fields as lowercase hex so the four existing IT projects can load known bytes and compare what the engine stored, including NUL and 0xFF.
|
Added support for binary data in data sets as hex, so a combination of ideas. |
fixes #8154
Hop Binary stream fields (hash keys, raw bytes, BLOBs) can now be bulk loaded on every transform whose engine accepts binary in its bulk protocol. There is no new dialog option: the stream type drives the encoding, the same way PostgreSQL already serializes booleans as
t/f.The previous failure (
PGBulkLoader doesn't handle the type Binary) is gone. Text-based loaders callgetBinary()and write hex in the format that engine actually reads:\xdeadbeef(byteahex)SET col = UNHEX(@col)HEXTORAWin the SQL*Loader control file (replaces the brokenstartlob/byte[].toString()path)BINARY_FORMAT = 'HEX'onCOPY\x(VARBYTE)BLOBBYTE/VARBYTESTRING)byte[]BINARY/VARBINARYnow usegetBinary()Out of scope: Doris (it stream-loads a payload field you already built) and the MySQL/SQL Server actions (they load an existing file, not Hop rows).
Tests
Unit tests cover the hex helpers and write paths for PostgreSQL, MySQL, Oracle, Snowflake, Redshift, MonetDB, TeraFast, CrateDB, and Vertica.
Existing integration tests were run successfully for projects database, mssql, vertica, and monetdb. Project cratedb is disabled. None of those suites currently load Binary/
BYTEA/VARBINARYcolumns.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
mvn clean install apache-rat:checkto make sure basic checks pass. A more thorough check will be performed on your pull request automatically.git rebase -i.addresses #123), if applicable.To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.