RFC: Explicit Cross-Collection Transactions for TanStack Optimistic

[KyleAMathews]

Summary

This RFC proposes elevating transactions to first-class citizens in TanStack Optimistic by introducing explicit transaction support through a new mutation factory API. This change addresses limitations in the current implementation that prevent developers from implementing common patterns like cross-collection updates and multi-step operations. By making transactions explicit and providing direct control over their lifecycle, we enable more complex workflows while maintaining the optimistic UI experience that makes the library valuable. The proposed API is designed to be intuitive and flexible, supporting both simple cross-collection operations and complex multi-step workflows.

Background

TanStack Optimistic is a library for building optimistic UI experiences, where user interface updates happen immediately while data changes are persisted asynchronously in the background. This approach provides a responsive user experience while ensuring data consistency.

The current architecture of TanStack Optimistic includes several key components:

1. Collections: Represent sets of data with the same structure, similar to database tables. Each collection has its own state management and transaction handling.

2. TransactionManager: Each collection has its own transaction manager that handles the lifecycle of transactions for that collection. The transaction manager is responsible for creating, persisting, and tracking transactions.

3. Transactions: Currently an internal implementation detail, transactions represent groups of mutations (inserts, updates, deletes) that are processed together. Transactions have a lifecycle (pending, persisting, completed, failed) and are used to track the status of optimistic updates.

4. MutationFn: Each collection can define a mutation function that is responsible for persisting changes to the backend. This function is called when a transaction is ready to be persisted.

5. Sync: Collections also have a sync mechanism that handles incoming changes from the backend, ensuring that the local state stays in sync with the server state.

In the current implementation, transactions are created implicitly when mutation operations (insert, update, delete) are called on a collection. Mutations that occur within the same event loop tick are grouped into a single transaction. The transaction is then processed by the collection's transaction manager, which calls the collection's mutation function to persist the changes.

This approach works well for simple cases where mutations are isolated to a single collection. However, it doesn't address more complex scenarios where related data needs to be updated across multiple collections as part of a single logical operation.

Other libraries and frameworks have addressed similar challenges:

• Database ORMs: Libraries like Knex.js, Prisma, and Drizzle ORM provide explicit transaction APIs that allow grouping multiple operations across tables into a single atomic unit.
• Client-side data stores: Libraries like Dexie.js (for IndexedDB) and Replicache provide transaction capabilities for local data stores.
• State management libraries: Libraries like Redux and MobX support middleware and actions that can group related state changes.

These examples demonstrate the value of explicit transaction control for ensuring data consistency across related entities and providing a better developer experience for complex workflows.

Problem

The current transaction system in TanStack Optimistic has several limitations that prevent developers from implementing common patterns found in real-world applications:

1. Transactions are hidden implementation details: Transactions exist in the library but are not exposed as first-class objects that developers can interact with directly. This limits the ability to build more complex workflows on top of the transaction system.

2. No cross-collection transactions: Each collection manages its own transactions independently, making it impossible to atomically update related data across multiple collections. This is a common requirement in real-world applications, such as adding a user to a team which requires updating both the users collection and the team_members collection.

3. Mutation functions are tied to collections: The mutationFn is defined at the collection level, which doesn't accommodate scenarios where a single API endpoint handles changes to multiple related entities. For example, a POST to /api/users might create both a user record and team membership records.

4. No explicit transaction lifecycle control: Developers cannot explicitly control when a transaction is created, accumulated, committed, or rolled back. The transaction lifecycle is managed automatically by collections, with mutations within a single tick being grouped together.

5. Limited flexibility for multi-step operations: Some workflows require accumulating changes over time before committing them, such as multi-step forms or wizard interfaces. The current system doesn't provide a way to build up a transaction across multiple user interactions.

These limitations force developers to create workarounds or avoid using TanStack Optimistic for more complex scenarios, reducing the library's utility in real-world applications.

Proposal

We propose making transactions first-class citizens in TanStack Optimistic by introducing explicit transaction support through a new mutation factory API. This will allow developers to create, control, and persist transactions that span multiple collections.

API Design

The new API will provide two main patterns for working with transactions:

1. Immediate transactions - For simple cross-collection operations that should be committed immediately:

const addTodo = useOptimisticMutation({ 
  mutationFn: async ({ mutations }) => {
    // Custom logic to persist all changes in a single API call
    await api.createTodo(mutations[0].modified)
  }
})

// Call as many times as you want
addTodo.mutate(() => {
  todosCollection.insert(data)
  todoStatsCollection.update(stats => {
    stats.totalCount += 1
  })
})

2. Explicit transactions - For multi-step operations where changes need to be accumulated over time:

const multiStepForm = useOptimisticMutation({ 
  mutationFn: async ({ mutations }) => {
    // Custom logic to persist all changes in a single API call
    await api.createComplexEntity({
      user: mutations.find(m => m.collection === 'users').modified,
      profile: mutations.find(m => m.collection === 'profiles').modified,
      preferences: mutations.find(m => m.collection === 'preferences').modified
    })
  }
})

// Create an explicit transaction
const transaction = multiStepForm.createTransaction({ id: 'user-signup-flow' })

// Add first step changes
transaction.mutate(() => {
  usersCollection.insert({ id: 'new-user', name: 'John Doe' })
})

// Later, add second step changes
transaction.mutate(() => {
  profilesCollection.insert({ userId: 'new-user', bio: 'Software developer' })
})

// Finally, add third step changes and commit
transaction.mutate(() => {
  preferencesCollection.insert({ userId: 'new-user', theme: 'dark' })
})

// Commit all changes when ready
await transaction.commit()

// Or discard changes if needed
// transaction.rollback()

Implementation Details

1. Transaction Registry: Create a global transaction registry to track active transactions. When a transaction's mutate method is called, it registers itself as the active transaction for the duration of the callback.

2. Collection Integration:

• Modify collection mutation methods (insert, update, delete) to check if there's an active transaction. If so, add the mutation to that transaction instead of creating a new implicit transaction.
• remove mutationFn from the Collection API.

3. Transaction Manager Refactoring: Refactor the existing TransactionManager to work with external transactions rather than being tied to a specific collection. This will involve:

   • Moving transaction creation logic out of collections
   • Adding support for cross-collection mutations
   • Implementing dependency tracking between transactions

4. Mutation Factory: Implement the useOptimisticMutation hook that creates mutation functions with optional transaction creation capabilities.

5. Transaction Object: Create a new Transaction class with methods for:

   • mutate(callback): Execute a function where collection operations are automatically linked to the transaction
   • commit(): Finalize the transaction and persist changes
   • rollback(): Discard optimistic changes and cancel the transaction
   • getState(): Get the current state of the transaction (pending, committing, completed, failed)

6. Dependency Tracking: Implement tracking of transaction dependencies to ensure that if a transaction fails, any dependent transactions are also rolled back.

Transaction Behavior

1. Building on Optimistic Changes: Each new transaction will build on top of existing optimistic changes from other active transactions, as well as synced changes from the backend.

2. Concurrent Transactions: Multiple transactions can be active simultaneously, with the understanding that conflicts will be resolved by the backend or through defensive programming. The library will not enforce strict isolation between transactions.

3. Error Handling: Transaction errors will be exposed through the promise returned by transaction.commit(). If a transaction fails, its optimistic changes will be rolled back, along with any dependent transactions.

4. Collection Awareness: Collections will be informed when they are part of a transaction, but they will not own or manage the transaction lifecycle.

5. Mutation Function: The transaction's mutationFn will receive an array of all mutations across all collections involved in the transaction, allowing for flexible backend integration.

Definition of Success

The explicit transaction feature will be considered successful if it:

1. Makes transactions first-class citizens: Developers can create, control, and interact with transactions directly, building more complex workflows on top of the transaction system.

2. Enables cross-collection operations: Developers can atomically update related data across multiple collections as part of a single transaction.

3. Provides flexible mutation persistence: The transaction's mutation function can handle persisting changes to multiple collections in the most appropriate way for the backend, whether that's a single API call or multiple coordinated calls.

4. Supports explicit lifecycle control: Developers can explicitly control when a transaction is created, accumulated, committed, or rolled back.

5. Enables multi-step operations: Developers can build up a transaction across multiple user interactions, such as multi-step forms or wizard interfaces.

6. Provides a clean, intuitive API: The API is easy to understand and use, with minimal boilerplate and clear patterns for common use cases.

7. Passes comprehensive tests: The implementation passes tests for:

   • Cross-collection mutations being correctly tied to the right transaction
   • Multiple calls to .mutate() accumulating optimistic updates correctly
   • Transaction dependencies being properly tracked and handled
   • Error handling and rollback working correctly

8. Maintains performance: The feature does not significantly impact performance compared to the existing transaction system.

9. Solves real-world use cases: The feature enables developers to implement common patterns like the user-team example without workarounds or compromises.

[KyleAMathews]

Some more thoughts — I'm thinking now that transactions have to handle persistence themselves. They can re-use other API calls to persist parts in parallel but I think it's a pretty natural expectation that an explicit transaction is written together and it'd be odd if an explicit transaction was persisted in part across several collection mutationFns.

Also I've been feeling annoyed by collections needing to have IDs. When playing around with things, I don't like having to come up with IDs and it's in practice pretty hard to reference the right collection as you might e.g. have tons of little collections. Something like this is feeling nice:

const transaction = useTransaction({ mutationFn });

transaction.mutate(() => {
  // Both inserts are automatically tied to the transaction
  usersCollection.insert({ id: 'bob-id', name: 'Bob' });
  teamMembersCollection.insert({ userId: 'bob-id', teamId: 'team-1' });
});

// Call .mutate as many times as you'd like.

// Drop optimistic changes and close the transaction
// transaction.rollback()

transaction.commit()

We can use a global registry of entering/leaving mutate functions in order to connect a collections mutation to the transaction's mutation.

[KyleAMathews]

Ok, worked with Windsurf and wrote a formal RFC

RFC: Explicit Cross-Collection Transactions for TanStack Optimistic

Summary

This RFC proposes adding explicit transaction support to TanStack Optimistic, allowing developers to group mutations across multiple collections into a single transaction with explicit lifecycle control. This addresses real-world use cases where related data needs to be updated atomically across different collections, such as adding a user to a team which requires updating both the users collection and the team_members collection.

Background

TanStack Optimistic currently supports implicit collection-level transactions, where mutations to a collection within a single tick are grouped into a transaction and processed by the collection's mutationFn. Each collection has its own TransactionManager that handles the lifecycle of these transactions.

The current transaction system has several key components:

1. TransactionManager: Manages transactions for a specific collection
2. Transaction: Represents a group of mutations with a lifecycle (pending, persisting, completed, failed, etc.)
3. MutationFn: Handles the persistence and synchronization of mutations

While this works well for single-collection operations, it doesn't address scenarios where related data needs to be updated across multiple collections as part of a single logical operation.

Transaction management is a common pattern across both server-side and client-side libraries. Server-side libraries like Knex, Prisma, and Drizzle ORM provide explicit transaction APIs that allow grouping multiple operations across tables into a single atomic unit. Client-side libraries like Dexie.js (for IndexedDB) and Replicache also provide transaction capabilities for local data stores. These libraries demonstrate the value of explicit transaction control for ensuring data consistency across related entities.

Problem

The current implicit collection-level transaction system has several limitations:

1. No cross-collection transactions: Developers cannot atomically update related data across multiple collections as part of a single transaction. This is a common requirement in real-world applications, such as adding a user to a team which requires updating both the users collection and the team_members collection.

2. No explicit lifecycle control: Developers cannot explicitly control when a transaction is committed or rolled back. The transaction lifecycle is managed automatically by the collection.

3. Limited flexibility for API interactions: Some APIs require multiple related entities to be created or updated in a single request. The current per-collection mutation functions don't accommodate these scenarios well.

4. No way to group mutations at different times: The current system only groups mutations that occur within a single tick, but some workflows require accumulating changes over time before committing them.

These limitations make it difficult to implement certain common patterns in applications, forcing developers to create workarounds or avoid using TanStack Optimistic for these scenarios.

Proposal

We propose adding explicit transaction support to TanStack Optimistic through a new useTransaction hook. This will allow developers to group mutations across multiple collections into a single transaction with explicit lifecycle control.

API Design

const transaction = useTransaction({ 
  mutationFn: async ({ transaction, collections }) => {
    // Custom logic to persist all changes in a single API call
    await api.createUserWithTeam({
      user: transaction.mutations.find(m => m.collection === 'users'),
      teamMember: transaction.mutations.find(m => m.collection === 'teamMembers')
    });
  }
});

// Execute mutations within the transaction context
transaction.mutate(() => {
  // Both inserts are automatically tied to the transaction
  usersCollection.insert({ id: 'bob-id', name: 'Bob' });
  teamMembersCollection.insert({ userId: 'bob-id', teamId: 'team-1' });
});

// Call .mutate as many times as you'd like to accumulate changes
transaction.mutate(() => {
  usersCollection.update({ id: 'bob-id', name: 'Bob Smith' });
});

// Explicitly commit the transaction when ready
transaction.commit();

// Or roll back optimistic changes and cancel the transaction
// transaction.rollback();

Implementation Details

1. Transaction Registry: Create a global registry to track active transactions. When a transaction's mutate method is called, it registers itself as the active transaction.

2. Collection Integration: Modify collection mutation methods (insert, update, delete) to check if there's an active transaction. If so, add the mutation to that transaction instead of creating a new one.

3. Transaction Coordination: The transaction will coordinate with the individual collection TransactionManagers to ensure proper locking and sequencing of operations.

4. Custom MutationFn: The transaction's mutationFn will replace the individual collection mutationFns for all operations in the transaction, allowing for custom persistence logic.

5. Lifecycle Methods:

   • mutate(): Executes a function where collection operations are automatically linked to the transaction
   • commit(): Finalizes the transaction and persists changes
   • rollback(): Discards optimistic changes and cancels the transaction

Transaction Behavior

1. Locking: When a collection is part of an active transaction, it will be locked from other transactions until the current transaction completes.

2. Ordering: If a transaction involves a collection that already has an in-progress transaction, the new transaction will wait for the existing one to complete.

3. Error Handling: Errors during transaction processing will be handled the same way as existing transactions, with the transaction marked as failed and the error exposed through the transaction object.

Future Enhancements (Not in Initial Implementation)

• Transaction state change callbacks
• Ability to pause/resume transactions
• Transaction timeout handling
• Transaction isolation levels
• Savepoints for partial rollbacks (similar to Drizzle ORM's nested transactions)

Definition of Success

The explicit transaction feature will be considered successful if it:

1. Enables cross-collection operations: Developers can atomically update related data across multiple collections as part of a single transaction.

2. Provides explicit lifecycle control: Developers can explicitly control when a transaction is committed or rolled back.

3. Maintains consistency with existing transactions: The behavior of explicit transactions is consistent with the existing implicit transactions, including error handling and transaction states.

4. Passes test coverage: The implementation passes tests for:

   • Collection mutations being correctly tied to the right transaction
   • Multiple calls to .mutate() accumulating optimistic updates correctly
   • Collections being properly locked when part of an active transaction

5. Solves real-world use cases: The feature enables developers to implement common patterns like the user-team example without workarounds.

6. Maintains performance: The feature does not significantly impact performance compared to the existing transaction system.

7. Provides a clean, intuitive API: Developers find the API intuitive and easy to use, with minimal boilerplate.

[KyleAMathews]

Thinking about this some more — useTransaction isn't what we want — we want a transaction factory.

Something like:

const addTodo = useOptimisticMutation({ mutationFn })

// Call as many times as you want.
addTodo.mutate(() => {
  todosCollection.insert(data)
})

The above transaction would immediately commit similar to the collection mutators. But is different as it allows cross-collection mutations & a custom mutationFn.

If you want to control the transaction lifespan you explicitly create a transaction:

const multiStepForm = useOptimisticMutation({ mutationFn })

const multiStepTransaction = multiStepForm.createTransaction({ id: `optional id for tooling` })

// Add first set of changes
multiStepTransaction.mutate(() => {
  // Modify a few collections
})

// Add second set of changes
multiStepTransaction.mutate(() => {
  // Modify a few collections
})

await multiStepTransaction.commit()

[KyleAMathews]

Hmm now I'm wondering if we only support useOptimisticMutation 🤔 having both this & a mutationFn inside collections is a bit confusing (having two ways of doing mutations) & perhaps ungainly as supporting inserts, updates, and deletes in the same mutationFn would be hard.

You still use the collection mutators but transactions are always explicit.

Also relevant is @samwillis' exploration of creating query collections i.e. collections which are derived from a query over other collection(s). As for these it'd definitely not make sense to have their own mutationFn but mutating them directly as part of a transaction still makes sense as you could trace from that optimistic change back to what backend change is needed.

[samwillis]

I like how this is becoming more composable, and I wander if we can go a step further and pull the sync out of the collection too?

So we end up with three things:

• Collection is just a transactional store, with all the moving parts needed for hooking these transactions up to mutations that can be applied or rolled back.
• "mutations" that can affect multiple stores, each with a composable implementation,
• "sync" implementations that update a Collection, and that a mutation can await for confirmation on.

Disconnecting sync from the collections enables alternative collections that have an alternative backend. We can eventually have those that are locally stored and not synced, or ephemeral in-memory collections for application state.

[KyleAMathews]

@samwillis it seems the question is do we want to set who can write into a collection at the collection definition time or let anyone write to it whenever (kinda like early vs. late binding).

So yeah, if a single "sync" instance could drive a bunch of collections — the late binding between sync & the collection makes sense.

Also you could imagine a collection which is initially populated by some data cached in indexeddb and then later perhaps a sync tool is connected to it.

Ok, yeah, makes sense to me!

[KyleAMathews]

Updated the issue body with the latest design.