Error seen in chbench: Non-postive accumulation in ReduceHierarchical

[cirego]

I've been running the chbench load-test-postgres workflow on my workstation for 6 hours and I started getting the following messages from Materialized:

Oct 09 21:49:08.840 ERROR dataflow::render::reduce: [customer-data] Non-positive accumulation in ReduceHierarchical: key: (Row{[Int32(50915)]}, 246532442547802865)     value: Row{[Int32(24)]} count: -1                                      
Oct 09 21:49:08.842 ERROR dataflow::render::reduce: [customer-data] Non-positive accumulation in ReduceHierarchical: key: (Row{[Int32(86694)]}, 219537877572457763)     value: Row{[Int32(10)]} count: -1                                      
Oct 09 21:49:08.842 ERROR dataflow::render::reduce: [customer-data] Non-positive accumulation in ReduceHierarchical: key: (Row{[Int32(22227)]}, 221037575626643602)     value: Row{[Int32(11)]} count: -1                                      
Oct 09 21:49:08.842 ERROR dataflow::render::reduce: [customer-data] Non-positive accumulation in ReduceHierarchical: key: (Row{[Int32(13898)]}, 263029121143847094)     value: Row{[Int32(47)]} count: -1                                      
Oct 09 21:49:08.844 ERROR dataflow::render::reduce: [customer-data] Non-positive accumulation in ReduceHierarchical: key: (Row{[Int32(28067)]}, 258530026981289577)     value: Row{[Int32(32)]} count: -1                                      
Oct 09 21:49:08.844 ERROR dataflow::render::reduce: [customer-data] Non-positive accumulation in ReduceHierarchical: key: (Row{[Int32(93835)]}, 278026101685705484)     value: Row{[Int32(53)]} count: -1                                      
Oct 09 21:49:08.846 ERROR dataflow::render::reduce: [customer-data] Non-positive accumulation in ReduceHierarchical: key: (Row{[Int32(66783)]}, 285524591956634679)     value: Row{[Int32(62)]} count: -1                                      
Oct 09 21:49:09.121 ERROR dataflow::render::reduce: [customer-data] Non-positive accumulation in ReduceHierarchical: key: (Row{[Int32(75243)]}, 248032140601988704)     value: Row{[Int32(25)]} count: -1                                      
Oct 09 21:49:09.951 ERROR dataflow::render::reduce: [customer-data] Non-positive accumulation in ReduceHierarchical: key: (Row{[Int32(13963)]}, 242033348385245348)     value: Row{[Int32(29)]} count: -1                                      
Oct 09 21:49:09.951 ERROR dataflow::render::reduce: [customer-data] Non-positive accumulation in ReduceHierarchical: key: (Row{[Int32(62437)]}, 353011004394997434)     value: Row{[Int32(99)]} count: -1

[frankmcsherry]

This is only supposed to appear if the input data accumulate to negative counts for some records. At least, that is what it is warning about. When we saw this in the past it was defective input data; are the input data stream captured? If so, we can inspect them for this property.

[cuongdo]

It might be worth understanding if these occur with the MySQL chbench setup -- which we run nightly anyway. If it does not, there could be an issue with the Debezium / Postgres config.

[cirego]

It might be worth understanding if these occur with the MySQL chbench setup -- which we run nightly anyway. If it does not, there could be an issue with the Debezium / Postgres config.

Yes, agreed. @umanwizard mentioned that Postgres may require some tuning to work correctly with Debezium.

[cirego]

This was in fact a Debezium / Postgres issue. Closing this out -- we have #5262 open to track postgres /debezium.