Queries break collection sync when multiple optimistic updates are queued

[obeattie]

I've made a repro of an issue I've been seeing in my app where collections with functional where queries can stop syncing when multiple optimistic updates are queued simultaneously. The page needs to be fully reloaded when this happens, so it's a fairly serious issue for me.

Reproduction

https://stackblitz.com/edit/tanstack-db-init-ewe1mn4x Click the "Insert" button to see the issue:

• The collection without functional where queries continues displaying updates each second. ✓
• The collection with a functional where query throws errors which you can see in the console, and stops syncing. ✗

Logic

Both share the same logic:

1. Insert two items simultaneously
2. Wait for server sync before returning from onInsert
3. The (simulated) server then sends updates to each item once per second. Only the collection that has a functional where query registered breaks.

I've had difficulty tracing the internal flow here, so any help/advice if I'm doing something wrong in how I'm calling the APId would be appreciated. 🙏

[samwillis]

Interestingly it's not related to the fn.where, I changed the other query to do this and it broke the "plain" side:

  const { data, isReady } = useLiveQuery((q) =>
    q.from({ collection }).where(({ collection }) => gt(collection.n, -1))
  );

The error is being throws from inside the live query collection as it tries to materialise the results. It's interesting that it's getting duplicate inserts. I will keep looking into this.

[obeattie]

Aha! I hadn't actually tried that as in my case I'm just using the functional version to do some filtering that isn't possible using the built-in expressions. Thanks for taking a look at this.

[samwillis]

Looks like this is fixed in a more recent db version, your repo was on react-db v0.1.21. I updated it to v0.1.26, with a small change to the code (syncData is internal and under collection._state.syncedData now), and it all works fine.

We've fixed a few bugs related to emitting change events to the live queries and I suspect this was a symptom of one of those bugs - I suspect it was this one: #631

Could you upgrade your app to the latest and let me know if you still see this in the actual app?

[obeattie]

Whoops, sorry I was not using the latest version.

Annoyingly, that fixes the repro as you say, but I am still getting the same error in my app. Let me see if I can work out why that is and whether I can get the repro to do the same thing again.

[obeattie]

Did a bit more sleuthing, and put together a refined repro: https://stackblitz.com/edit/tanstack-db-init-dfgqvbxt

It turns out that in my real app, the server response includes a few extra properties that aren’t present on the original object created in the optimistic client-side transaction 😅 — so that’s reflected in the updated repro.

Better validation on my part (ie. z.strictObject) would have enabled me to understand this quicker. That said, I think there is a bug here: this only happens when using a filtered query, the error raised is extemely cryptic, and it anyway seems reasonable that the server should be able to return different data to what the client has in its optimistic cache.

[samwillis]

Thanks! interesting... just checked and its not where clauses, its anything using the build so even just a from also breaks (q) => q.from({ collection })

I strongly suspect it's a variation of the previous bug, I'm sure I can find it quite quickly.

[samwillis]

This is another reconciliation bug, the source collection is emitting multiple duplicate insert message into the query pipeline. I am going to try and fix in the short term (immediately), but really want to work on the refactor to simplify this whole process: #634

This new test case will make the refactor more resilient when we get to it.

[samwillis]

This is fixes by #650

The reproduction but updated to the preview package from that PR https://stackblitz.com/edit/tanstack-db-init-jm98exso?file=src%2FApp.tsx