Blob / binary columns

[Komnomnomnom]

I realise from the docs that binary column support is experimental. I'd like to use maxwell with an existing database which has some blob columns but maxwell crashes whenever a row with a blob column is modified.

MySQL 5.1.59
Maxwell 0.11.0-RC1

Steps to reproduce:

CREATE TABLE `maxwell_blob_test` (
    `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
    `blob` BLOB NOT NULL,
    PRIMARY KEY (`id`)
)
COLLATE='utf8_unicode_ci'
ENGINE=MyISAM
;

INSERT INTO `sellstuf1`.`maxwell_blob_test` (`blob`) VALUES (0x22);

Maxwell log

16:36:04,286 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-binlog.000002:65790]
16:42:58,092 DEBUG SchemaChange - SQL_PARSE <- "CREATE TABLE `maxwell_blob_test` (
    `id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
    `blob` BLOB NOT NULL,
    PRIMARY KEY (`id`)
)
COLLATE='utf8_unicode_ci'
ENGINE=MyISAM"
16:42:58,127 DEBUG SchemaChange - SQL_PARSE ->   (parse (statement (create_table (create_table_preamble CREATE TABLE (table_name (name (id `maxwell_blob_test`)))) (create_specifications ( (create_specification (column_definition (name (id `id`)) (data_type (signed_type INT (int_flags UNSIGNED))) (column_options (nullability NOT NULL)) (column_options AUTO_INCREMENT))) , (create_specification (column_definition (name (id `blob`)) (data_type (generic_type BLOB)) (column_options (nullability NOT NULL)))) , (create_specification (index_definition (index_type_pk PRIMARY KEY (index_column_list ( (index_columns (index_column (name (id `id`)))) ))))) )) (table_creation_options (creation_collation COLLATE = (string 'utf8_unicode_ci'))) (table_creation_options (creation_engine ENGINE = (string MyISAM))))) <EOF>)
16:42:58,129 INFO  MaxwellParser - storing schema @BinlogPosition[mysql-binlog.000002:124061] after applying "CREATE T ENGINE=MyISAM"unicode_ci'LL, NULL AUTO_INCREMENT,
16:42:58,252 DEBUG SchemaPosition - syncing binlog position: BinlogPosition[mysql-binlog.000002:124061]
16:42:58,254 DEBUG SchemaPosition - Writing binlog position to maxwell.positions: BinlogPosition[mysql-binlog.000002:124061]
java.lang.NullPointerException: charset
    at java.lang.String.<init>(String.java:449)
    at java.lang.String.<init>(String.java:503)
    at com.zendesk.maxwell.schema.columndef.StringColumnDef.asJSON(StringColumnDef.java:54)
    at com.zendesk.maxwell.MaxwellAbstractRowsEvent.jsonMaps(MaxwellAbstractRowsEvent.java:229)
    at com.zendesk.maxwell.MaxwellAbstractRowsEvent.toJSONStrings(MaxwellAbstractRowsEvent.java:251)
    at com.zendesk.maxwell.StdoutProducer.push(StdoutProducer.java:12)
    at com.zendesk.maxwell.MaxwellParser.run(MaxwellParser.java:111)
    at com.zendesk.maxwell.Maxwell.run(Maxwell.java:69)
    at com.zendesk.maxwell.Maxwell.main(Maxwell.java:75)
2015-09-10 16:43:43,071 FATAL Unable to register shutdown hook because JVM is shutting down.

[osheroff]

ok, I'll look into the crash. would you want the blob output as base64? That's sort of the best thing I can think of to do with it....

[Komnomnomnom]

base64 makes the most sense to me

[osheroff]

#97 that was easier than I thought it would be

[Komnomnomnom]

Great #97 fixes the crash and I can see the base64'd data coming through maxwell.

However, I notice the base64 encoded string is not padded and is a 'URL safe' version. Was this intentional? I think a padded base64 with standard encoding would be best for interoperability, unless there is a reason for choosing a URL safe version?

[osheroff]

I wanted to avoid the line chunking that happens in apache commons 1.4. I'll use a different method, one sec.

[osheroff]

updated apache commons to 1.5 for non-chunked base64, now it's padded.

[Komnomnomnom]

Awesome 👍, decoding's a lot more straightforward now (Python)

[osheroff]

cool, I will release this and others once I can get sign-off from a Zendesk folk (dumb coding standards) :)

[osheroff]

fixed in https://github.com/zendesk/maxwell/releases/tag/v0.10.1