Missing Updates with Timestamp+incrementing mode in Postgres

[salemn]

Hello Everybody,

i'm deploying kafka-connect, in timestamp+incrementing mode, on our platform in order to get data transferred from our main database to secondary one. I'm using source and sink connectors. Both databases are Postgres databases

I'm facing a missing lines issues. the case is not frequent but happens from time to time.

Here is the situation:

postgres=# CREATE TABLE foo(id serial primary key, updated timestamp without time zone);
CREATE TABLE
postgres=# CREATE TRIGGER update_updated_column
postgres-#   BEFORE INSERT OR UPDATE
postgres-#   ON foo
postgres-#   FOR EACH ROW
postgres-#   EXECUTE PROCEDURE update_updated_column();
--
SESSION 1:
postgres=# begin;
BEGIN
postgres=# INSERT INTO foo(updated) VALUES (null);
INSERT 0 1
postgres=# SELECT * FROM foo;
 id |          updated           
----+----------------------------
  1 | 2016-12-01 08:53:54.099995
(1 ligne)
---

SESSION 2 :
postgres=# BEGIN;
BEGIN
postgres=# INSERT INTO foo(updated) VALUES (null);
INSERT 0 1
postgres=# SELECT * FROM foo;
 id |          updated           
----+----------------------------
  3 | 2016-12-01 08:54:46.420757
(1 ligne)

postgres=# COMMIT;
COMMIT
postgres=# SELECT * FROM foo;
 id |          updated           
----+----------------------------
  3 | 2016-12-01 08:54:46.420757
(1 ligne)
----

SESSION 1 :
postgres=# COMMIT;
COMMIT
postgres=# SELECT * FROM foo;
 id |          updated           
----+----------------------------
  1 | 2016-12-01 08:53:54.099995
  3 | 2016-12-01 08:54:46.420757
(2 lignes)

Line 1 is missing in my target database.

Is there any way to use to source connector to avoird such situation?

Best Regards
Naoufel

[shikhar]

salemn is the missing row present in the Kafka topic written to by the source connector? (to isolate whether the issue is on the source or the sink side of things)

[salemn]

Shikhar Bhushan (@shikhar) i think that the issue is in the source connector as The row is missing in the topic.

[gfodor]

Is there any chance that some of the records that you are seeing get dropped have approximately the same timestamp as other records? The PR here:

#49

Basically resolves an issue where the queries being sent to postgresql were querying at millisecond (vs microsecond) resolution, so my suspicion is that if you have records whose timestamps fall within the same millisecond that some could get missed since they would be mischaracterized as falling earlier than the last committed offset timestamp if consumed across commits. (Right now this is just a theory, I haven't confirmed it)

[gilbsgilbs]

Greg Fodor (@gfodor) I think the issue is a basic race conditon.

1. Transaction TR0 begins, inserts row R0 (at updated = T0), and hangs without committing.
2. Transaction TR1 begins, inserts row R1 (at updated = T1 > T0), and commits.
3. Source connector polls. It sees R1, but not R0 (because it's not committed yet). Last updated timestamp is now at T1. Source connector will assume that nothing could have happened before T1.
4. TR0 commits eventually. Recall that R0 has an older updated timestamp than R1.
5. Source connector polls, but misses R0 because T0 < T1.

[ewencp]

salemn Did you set timestamp.delay.interval.ms? Without a delay there is a possibility that transactions may still be open and not visible to queriers despite timestamps & rows that are "newer" are visible. You'd want to set this value to be conservatively longer than the longest time you expect a transaction to take. This is just a fundamental limitation of working through the JDBC interface.

[willyhoang]

If I'm using a source connector with incrementing mode (with only an incrementing id and no timestamp), is there an equivalent config that we could use to resolve this race condition?

[SemanticBeeng]

Can this #172 (comment) be happening when using a "mode": "timestamp+incrementing" and "timestamp.delay.interval.ms": 3000 (assumption is that transactions end faster than 1 sec)? Ewen Cheslack-Postava (@ewencp)

[gilbsgilbs]

SemanticBeeng (@SemanticBeeng) Assuming that the delay is always higher than the slowest transaction, that is perfectly fine. However, a tiny delay such as 3 seconds will be difficult to keep in the long run. One day or another, you'll have some misses and need to implement a backup mechanism or increase the delay. I don't know what is your use-case, but I would say that most of the time, if it's really important for you to not miss anything, you'll need to increase the delay drastically, trading off the lag. That's the main reason why we gave up on using kafka-connect-jdbc.

[song860918]

Willy Hoang (@willyhoang) Did you get the incrementing mode problem figured out? I'm running into the same issue.

[makearl]

For incrementing mode, it looks like this becomes a problem once the connector finishes backfilling and catches up to realtime. Should there be a similar configuration item to "timestamp.delay.interval.ms" for incrementing mode?

[dss010101]

Gilbert Gilb's (@gilbsgilbs) what are you using instead of kafka-connect-jdbc and why wouldn't any other odbc driver exhibit the same problem?

[vibtrip]

Hi, did the original author of this issue get the issue resolved? I am facing the same issue with SQL Server. Basically, it seems like even in "timestamp+incrementing" mode, the connect is doing a strictly "incrementing". No updates involving earlier incrementing IDs show up but new rows inserted with higher IDs do show up. salemn

[vibtrip]

Ewen Cheslack-Postava (@ewencp) was timestamp+incrementing in the original issue here resolved with the delay?? Is 5 seconds deays good enough for polling every second?

[gilbsgilbs]

@dizzy0ny We ended up using nothing and abandoned the project. Any ODBC driver will suffer from the same issues because it tries to solve a problem with wrong tools. There's a reason for DBMS replication systems being based on transaction ids: it's the only way to guarantee consistency and real-time. Not using XIDs (one way or the other) implies a trade-off between consistency (missed rows due to "transactions shadowing" for instance) and real-time (requires a huge lag to ensure all transactions are over). This might be OK in some (maybe most) cases, but I really reckon the doc should be more honest about this. Anyhow, some tools that let you leverage logical replication to solve the same problem have emerged recently. Take a look at Debezium or Bottledwater (which seems abandoned) or even AWS DMS. They might be a better fit.

bt2025 (@vibtrip) I don't think your problem is related to the OP's. The issue is about some marginal missed updates. Please open another issue.

[dss010101]

Gilbert Gilb's (@gilbsgilbs) the only way i can think of to solve this may be to violate ACID and use read uncommitted. that presents a whole lot of other issues however.
However, increasing the timestamp.delay.interval.ms doesn't necessarily mean you are incurring lag does it? i think it simply mean the query will look back further..meaning it may return more data than it should which may have been processed already. That's what i thought and could be wrong.