java.lang.ClassCastException: java.sql.Timestamp cannot be cast to java.lang.Integer

[divo77]

HI,

We constantly have a problem with MAXWELL crashing with the error like:

10:00:02,010 WARN MaxwellReplicator - Started replication stream outside of transaction. This shouldn't normally happen.
java.lang.ClassCastException: java.sql.Timestamp cannot be cast to java.lang.Integer
at com.zendesk.maxwell.schema.columndef.IntColumnDef.toLong(IntColumnDef.java:35)
at com.zendesk.maxwell.schema.columndef.IntColumnDef.asJSON(IntColumnDef.java:51)
at com.zendesk.maxwell.schema.ColumnWithDefinition.asJSON(ColumnWithDefinition.java:34)
at com.zendesk.maxwell.replication.AbstractRowsEvent.jsonMaps(AbstractRowsEvent.java:207)
at com.zendesk.maxwell.replication.MaxwellReplicator.getTransactionRows(MaxwellReplicator.java:291)
at com.zendesk.maxwell.replication.MaxwellReplicator.getRow(MaxwellReplicator.java:438)
at com.zendesk.maxwell.replication.MaxwellReplicator.work(MaxwellReplicator.java:159)
at com.zendesk.maxwell.util.RunLoopProcess.runLoop(RunLoopProcess.java:35)
at com.zendesk.maxwell.Maxwell.start(Maxwell.java:163)
at com.zendesk.maxwell.Maxwell.main(Maxwell.java:184)

We are on 1.5.1 version Is there is a solution for that ?

[osheroff]

this means generally that Maxwell's view of the schema is out of sync with mysql's. Were there other errors before this one? What's the rest of your config like?

[divo77]

we had all the issues that bmichaud posted here. Something got wrong with schema so to prevent loosing transactions I have manually edited the schema table data which allowed me to get through most of them but it all ended up with a failure today. Do we have any ability to sync up the metadata in schema with a real metadata in mySql database on demand ? Let's say if I see something wrong with table TAB2 I run something and it Sync my TAB2 mySql metadata with schema ?

[divo77]

if you provide just an algorithm and DMLs thtat would be helpfull

[osheroff]

unfortunately there's no way to simply "sync on demand" ; there's no way of telling from a given binlog position (especially one in the past) what the associated schema might be; once you're in the business of trying to hand-edit schemas, all bets are off. I don't suggest that you go down this route, the best thing to do is to drop the maxwell database and start from the head.

Other than that, you can look into mysqlbinlog -vv to try to determine what the schema was at the time of the crash.

[osheroff]

I'm going to close this as it was all a result of trying to fix a position by hand after encountering a separate maxwell error. There's still a set of fixes owed somewhere around here on trying to recover somehow after a disaster, but it's unclear to me exactly what.