Skip to content

Issue #8154 : Support binary data in bulk load transforms - #8162

Merged
hansva merged 2 commits into
apache:mainfrom
mattcasters:issue-8154
Aug 29, 2026
Merged

Issue #8154 : Support binary data in bulk load transforms#8162
hansva merged 2 commits into
apache:mainfrom
mattcasters:issue-8154

Conversation

@mattcasters

Copy link
Copy Markdown
Contributor

fixes #8154

Hop Binary stream fields (hash keys, raw bytes, BLOBs) can now be bulk loaded on every transform whose engine accepts binary in its bulk protocol. There is no new dialog option: the stream type drives the encoding, the same way PostgreSQL already serializes booleans as t/f.

The previous failure (PGBulkLoader doesn't handle the type Binary) is gone. Text-based loaders call getBinary() and write hex in the format that engine actually reads:

Loader Encoding
PostgreSQL / Greenplum \xdeadbeef (bytea hex)
MySQL lowercase hex + SET col = UNHEX(@col)
Oracle hex in the data file + HEXTORAW in the SQL*Loader control file (replaces the broken startlob / byte[].toString() path)
Snowflake hex + BINARY_FORMAT = 'HEX' on COPY
Redshift hex without \x (VARBYTE)
MonetDB hex for BLOB
TeraFast hex for FastLoad BYTE/VARBYTE
CrateDB hex string (no native binary type; lands in STRING)
SQL Server unchanged — already sent as byte[]
Vertica unchanged native bytes; BINARY/VARBINARY now use getBinary()

Out of scope: Doris (it stream-loads a payload field you already built) and the MySQL/SQL Server actions (they load an existing file, not Hop rows).

Tests

Unit tests cover the hex helpers and write paths for PostgreSQL, MySQL, Oracle, Snowflake, Redshift, MonetDB, TeraFast, CrateDB, and Vertica.

Existing integration tests were run successfully for projects database, mssql, vertica, and monetdb. Project cratedb is disabled. None of those suites currently load Binary/BYTEA/VARBINARY columns.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Run mvn clean install apache-rat:check to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
  • If you have a group of commits related to the same change, please squash your commits into one and force push your branch using git rebase -i.
  • Mention the appropriate issue in your description (for example: addresses #123), if applicable.

To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

Serialize Hop Binary fields in each bulk loader using the format that
engine's protocol already accepts (hex for text COPY/LOAD DATA, native
bytes for SQL Server and Vertica).
@hansva

hansva commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Can we add some IT tests for the databases in the integration tests? Other than that, looks like a solid addition!

@mattcasters

Copy link
Copy Markdown
Contributor Author

I couldn't immediately see a non-spammy way of doing it. It's tricky from an IT test-project perspective but I'll try to use /dev/random to generate some sample data in the various IT projects. Another option would perhaps be to use a JavaScript or UDJC transform to generate, perhaps in combination with the new "Data Set Output" transform.

@hansva

hansva commented Aug 29, 2026

Copy link
Copy Markdown
Contributor

Ah, I see the issue, as we can't include binary data in our source code.
Maybe Base64 encoded text that we converto to binary strings?

@mattcasters
mattcasters marked this pull request as draft August 29, 2026 10:21
@mattcasters

Copy link
Copy Markdown
Contributor Author

It's not a bad idea but as you can see we now already have a bunch of prrojects that need the bulk loader test. So we'd be copying files all over the place to make it work properly in the docker containers. Anyway, I don't think generating the binary data is as big of an issue as I thought it was yesterday. I'll figure it out.

Serialize Binary data-set CSV fields as lowercase hex so the four
existing IT projects can load known bytes and compare what the engine
stored, including NUL and 0xFF.
@mattcasters
mattcasters marked this pull request as ready for review August 29, 2026 11:55
@mattcasters

Copy link
Copy Markdown
Contributor Author

Added support for binary data in data sets as hex, so a combination of ideas.

@hansva
hansva merged commit dcad630 into apache:main Aug 29, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Support binary data in bulk load transforms and actions

2 participants