gemini detected row difference, while error message shows rows are the same

[aleksbykov]

Job:
https://jenkins.scylladb.com/job/enterprise-2023.1/job/Reproducers/job/gemini-3h-ics-cdc-with-nemesis-test/12/
Gemini logs:
gemini-l0-c0-2400514c-9363-4f76-b02c-f54cbb1f4d42.log
gemini_result_0a96fe8c-6795-4ed6-9159-79d8c4afa6c2.log

[dkropachev]

Related issue - scylladb/scylladb#11103

[aleksbykov]

Also may be it time data representation.
in another job, i got similar issue of gemini:
gemini-l0-c0-31c486ef-4ed4-443e-a8b1-84e5b925cd0e.log
gemini_result_b3689814-7ae5-4073-8d51-d1afde711539.log

when i try to request the rows with python driver, i got next error:

Traceback (most recent call last):
  File "cassandra/obj_parser.pyx", line 72, in cassandra.obj_parser.TupleRowParser.unpack_row
  File "cassandra/deserializers.pxd", line 41, in cassandra.deserializers.from_binary
  File "cassandra/deserializers.pyx", line 173, in cassandra.deserializers.DesTimeType.deserialize
  File "cassandra/util.py", line 955, in cassandra.util.Time.__init__
  File "cassandra/util.py", line 1002, in cassandra.util.Time._from_timestamp
ValueError: value must be less than number of nanoseconds in a day (86400000000000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cassandra/row_parser.pyx", line 38, in cassandra.row_parser.make_recv_results_rows.recv_results_rows
  File "cassandra/obj_parser.pyx", line 31, in cassandra.obj_parser.ListParser.parse_rows
  File "cassandra/obj_parser.pyx", line 74, in cassandra.obj_parser.TupleRowParser.unpack_row
cassandra.DriverException: Failed decoding result column "col2" of type time: value must be less than number of nanoseconds in a day (86400000000000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cassandra/obj_parser.pyx", line 72, in cassandra.obj_parser.TupleRowParser.unpack_row
  File "cassandra/deserializers.pxd", line 41, in cassandra.deserializers.from_binary
  File "cassandra/deserializers.pyx", line 173, in cassandra.deserializers.DesTimeType.deserialize
  File "cassandra/util.py", line 955, in cassandra.util.Time.__init__
  File "cassandra/util.py", line 1002, in cassandra.util.Time._from_timestamp
ValueError: value must be less than number of nanoseconds in a day (86400000000000)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/abykov/Projects/scylladb-project/scylla-cluster-tests/check_gemini_query.py", line 9, in <module>
    print(list(session.execute(stm)))
  File "cassandra/cluster.py", line 2668, in cassandra.cluster.Session.execute
  File "cassandra/cluster.py", line 4953, in cassandra.cluster.ResponseFuture.result
  File "cassandra/connection.py", line 1263, in cassandra.connection.Connection.process_msg
  File "cassandra/protocol.py", line 1196, in cassandra.protocol._ProtocolHandler.decode_message
  File "cassandra/protocol.py", line 744, in cassandra.protocol.ResultMessage.recv_body
  File "cassandra/protocol.py", line 730, in cassandra.protocol.ResultMessage.recv
  File "cassandra/row_parser.pyx", line 46, in cassandra.row_parser.make_recv_results_rows.recv_results_rows
  File "cassandra/obj_parser.pyx", line 57, in cassandra.obj_parser.TupleRowParser.unpack_row
  File "cassandra/obj_parser.pyx", line 74, in cassandra.obj_parser.TupleRowParser.unpack_row
cassandra.DriverException: Failed decoding result column "col2" of type time: value must be less than number of nanoseconds in a day (86400000000000)

[aleksbykov]

Looks it is related to gemini seed 8 and could be related to datatype time

[illia-li]

From logs:

        \"col13\": map[*big.Int]int{
-               \u0026⟪0xc01dfd4b40⟫{abs: ⟪ptr:0xc020c7b198, len:1, cap:1⟫{2020941397193943905}}: 851432987,
-               \u0026⟪0xc01dfd4b60⟫{abs: ⟪ptr:0xc020c7b1a8, len:1, cap:1⟫{8382710959889238823}}: 50194853,

+               \u0026⟪0xc01e9c15a0⟫{abs: ⟪ptr:0xc01b9c0b78, len:1, cap:1⟫{2020941397193943905}}: 851432987,
+               \u0026⟪0xc01e9c16a0⟫{abs: ⟪ptr:0xc01b9c0b88, len:1, cap:1⟫{8382710959889238823}}: 50194853,

col13 - map type with a key - varint type.
At this moment this schema unsupported, because qocql makes map with pointer as a key.
Reasons explained here:
#405 (comment)