Persistent live queries, surviving to offline periods

[messagenius-admin]

Current Limitation

To ensure data remains synchronized on my client, I must perform several queries and subscribe to live queries whenever the app returns online after being offline even for a short period.

Feature / Enhancement Description

I would like to implement persistent live queries that survive periods of offline operation. When the app comes back online, the server should restore existing subscriptions and push all the events that occurred during the offline period to the client.

Example Use Case

// Initialize the LiveQuery client
const client = new Parse.LiveQueryClient({ ... });

// Open the connection
client.open();

// Create a query for the Books class
const BooksQuery = new Parse.Query('Books');

// Define persistent subscription options
const BooksSubscriptionOptions = {
  persistent: true,           // Enable persistence for this subscription
  persistentTTL: 3600,        // Time to live in seconds (1 hour)
  persistentKey: "ALL_BOOKS"  // Unique identifier for this subscription
};

// Subscribe to the query with persistence options
// If subscription was previously initiated by this device, with same persistentKey, and still within TTL, restore it and trigger relevant events heppened during offline period.
const BooksSubscription = client.subscribe(BooksQuery, BooksSubscriptionOptions); 

BooksSubscription.on('create', myLocalStorage.addBook);

Alternatives / Workarounds

Currently, I need to perform a query.find() operation to fetch all books since my last connection:

// When coming back online
const lastConnectionTime = getLastConnectionTimestamp();
const BooksQuery = new Parse.Query('Books');
const subscription = client.subscribe(BooksQuery); 

BooksQuery.greaterThan('updatedAt', lastConnectionTime);
BooksQuery.find().then(books => {
    books.forEach(book => myLocalStorage.updateBook(book));
  });

Imagine having Books, Authors, Categories, Tags, Orders, FavoriteBooks, etc... This approach is inefficient as it requires a REST call for each class.

3rd Party References

Firebase Realtime Database and Realm provide similar offline persistence capabilities:

• Firebase automatically synchronizes local changes with the server when connectivity is restored
• MongoDB Realm Sync provides seamless offline-to-online synchronization

Implementing this feature would bring Parse Platform more in line with these competing solutions.

[parse-github-assistant]

🚀 Thanks for opening this issue!

[mtrezza]

Any suggestions how this could be implemented, on a high level? For example the client sending a lastSyncDate on reconnect and the server then pushing all objects where the updatedAt date is greater?

[messagenius-admin]

We can reach this goal with minimal changes to the server and client SDKs.

Current behavior:

• Client connects to LQ Server, server assigns a Client ID
• The client subscribes to Query
• After each CRUD event, Parse checks if any relevant subscription is affected
• Parse sends out events to all affected subscriptions
• If the event fails to be routed, for example, when the WS connection is lost, this event is lost
• If the WS connection is lost for a longer period, the subscription gets eliminated, and events are not sent anymore

Additionally, there is no clear SDK method to quickly determine if the WebSocket connection is active and reliable. Furthermore, when the WS connection is lost and restored, there is no documented strategy to identify whether there was a disconnection period that may have resulted in lost data, and therefore, if a query is needed.

Desired behavior for Subscriptions marked as persistent:

• Subscription doesn't get eliminated unless the given TTL is passed
• Events are either cached or queued, so that if the connection is not available, sending out the event is retried after successful reconnection
• Cache/queue is purged when TTL passes or when the subscription is closed explicitly

This approach minimizes changes on the client side and lets the server maintain responsibility for keeping clients in sync—even across offline periods. There is no need, in my view, to use a lastSyncDate parameter.

If we want to achieve even more reliability, we should keep the event in queue until an ACK is received.
Similar concepts can be found in MQTT with QoS 1 or 2, or simply in socket.io built-in ACK feature

[mtrezza]

Events are either cached or queued, so that if the connection is not available, sending out the event is retried after successful reconnection

Do we really want to cache every missed event and then replay them sequentially when the client reconnects? That includes obsolete events, for example, a field value that changes N times would cache N events, but N-1 events end up irrelevant. This approach adds noise but it also risks serious scalability issues. Multiply many clients by many events, and you get hyperbolic cache growth, potentially triggering huge fluctuations in data storage, cache writes, and network traffic. Imagine a whole network region goes down temporarily with many clients, that could become a meltdown scenario for the server side infrastructure. Also, with a TTL that you mentioned - that means the client receives N-x events with x being the events discarded by TTL. If a use case allows for that, why not always discard the obsolete N-1 events.

I would argue for a mechanism that just takes into account a lastSyncDate (or a similar approach) which means:

• zero caching server side
• on reconnect only sent to client last changes -> less work for server and client

Clearly, there is a theoretical use case for any approach, but from a practical POV I think not caching anything by discarding obsolete events is useful for more applications than caching everything.

If the use case requires recording of all value changes, then maybe the better approach is to store these value changes in a class and subscribe to it. For example a switch that is flipped many times and we need to know all the times it flipped, regardless of its final state - that should not be handled by LiveQuery by caching flips, but written to a class for persistent storage. Same for processing queue-like behavior, I don't see that as a LiveQuery use case, because there are more efficient ways to do that.

[messagenius-admin]

There are advantages and trade-offs with each approach. Let’s break them down.

1. Caching each event

All event-driven protocols (MQTT, PubSub, XMPP) rely on some form of spool/queue. These are logic-agnostic but storage-sensitive. This means fewer logic-related failures, and one table can store all events—one query is enough on reconnect, regardless of subscription count.

PROS: Low logic overhead; Single source of truth
CONS: High storage usage; Risk of obsolete data

Obsolete data can be mitigated by keeping only the latest version of the object

## 2. Send modified objects on reconnect
On reconnect, send all objects modified since lastSyncDate.
This timestamp should be managed server-side for simplicity and to avoid client-side clock drift and time issues.
However, this model does not support Delete or Leave events—since the object no longer matches any query, you have no way to recover or signal the deletion.

PROS: No event storage required; No redundancy
CONS: Fails to capture Delete/Leave; A query for each class is required (even if nothing changed)

3. Cache only event metadata (no payload)

Instead of storing full event data, cache only the Class Name, Event type, and Object ID.
This reduces storage but forces per-object queries during resync.

PROS: Lightweight storage; Still tracks all changes
CONS: Many queries required

Extra Considerations - regardless of the chosen solution

• Implementing an ACK system to ensure delivery, similar to MQTT QoS it's still something to be considered
• We need to ensure compatibility with relational data and complex operations (__op: AddUnique, Increment, etc)

[mtrezza]

Fails to capture Delete

Yes, not sure now that could be addressed at first glance. Having to query all subscribed classes is still negligible compared to building up and managing a cache backlog.

What still puzzles me is the TTL option. If an app can deal with lost events, why would it need a full object history in the first place? The lost events are unrecoverable, for example there's no way of recovering a field's previous values beyond what's cached.

In any case, LiveQuery could offer the options so the developer can set the caching strategy according to their needs.

• enable or disable TTL
• cache all events incl. field value diff, or only meta data like the obj ID, or only meta data like the class name without object references, for even less storage needs