You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
TL;DR: This proposes to split the PowerSyncBackendConnector interface into two superinterfaces: One providing credentials (for streaming downloads) and one responsible for uploading changes. We then allow connecting in upload or download-only mode.
The specifics vary slightly based on the specific SDK, but generally when wiring up the PowerSync Client SDK, you have to define a “backend connector” class with 2 methods:
fetchCredentials() which needs to return an object containing:
token: JWT for authentication with PowerSync Service. Developers can either reuse the existing JWT from their authentication provider, or mint a new JWT specific for PowerSync, by calling their own backend API
endpoint: PowerSync Service instance URL
uploadData() which gets entries from the upload queue, uploads them to the developer/customer’s backend, and marks those transactions as complete so that they are removed from the upload queue
Problem statements
There are several paper cuts involved with the tight coupling of everything in the ‘backend connector’.
If you are not fetching a JWT from your backend, but rather reusing an existing JWT from your authentication provider which you are getting from local state, then having fetchCredentials() as part of the “backend connector” does not make sense semantically.
The fact that fetchCredentials() and uploadData() are both part of a “backend connector” makes the concepts more complicated to explain in our documentation (and makes structuring the documentation more complicated)
We've been asked about an upload-only mode of the client SDK that can be used without connecting to a PowerSync Service instance. However, you are forced to call connect() on the PowerSyncDatabase in order to pass a “backend connector” with an uploadData() implementation. You cannot upload client-side mutations to your backend without connecting to a PowerSync Service instance.
The different URLs involved in fetchCredentials() (PowerSync Service instance endpoint URL and custom backend URL) can cause confusion.
When it comes to uploadData(), it would be better to standardize terminology to refer to the entries in the upload queue as “mutations” everywhere in our code/APIs and documentation. “Mutations” is the most clear description we can use.
At the moment we use a mix of “CRUD”, “CRUD operations”, “CRUD items” (which doesn’t make sense because that includes reads), “writes”, “write operations”, “mutations” and “changes” in different places, and the naming of uploadData() doesn’t align with any of those.
Goals
See whether concepts can be simplified: can we eliminate the concept of a ‘backend connector’?
Reduce the tight coupling between the different parts of the current ‘backend connector’
Allow mutations on the client-side SQLite database to be uploaded without connecting to a PowerSync Service instance
Consider separating configuring the JWT and PowerSync Service instance URL
Improving naming
Of uploadData() — standardize on “mutations”
And of fetchCredentials() — in many/most cases it won’t require fetching from your backend, if you reuse an existing JWT
Preserve convenience and type-safety, and avoid introducing new footguns or new sources of confusion
Proposal
Note: The internal document lists additional proposals, this is a summary based on discussions in comments.
While preserving backwards compatibility, eliminate the concept of a 'backend connector' in favor of these interfaces (or, where appropriate for an SDK, callback functions):
publicinterfaceAuthenticator {
// Fetch or use cached credentialspublicsuspendfunresolveCredentials(): String// Hint that the credentials were rejected by the PowerSync servicepublicfuninvalidateCredentials()
}
publicfun interfaceMutationUploader {
publicsuspendfunuploadMutations(onDatabase:PowerSyncDatabase)
}
// existing class now implements these interfacespublicabstractclassPowerSyncBackendConnector: Authenticator, MutationUploader { /* ... */ }
publicinterfaceCustomCheckpointRequestAuthenticator: Authenticator {}
Depending on the SDK, we then add overloads or optional parameters for connect():
Calling connect(backendConnector) continues to work.
Additional considerations
Local mutations that are never uploaded
What do we do with a database that has pending ps_crud items and is then connected in download-only mode? It wouldn't be able to apply checkpoints, and we only surface that as log items which are easy to overlook.
To make this error state easier to observe, maybe we should also add these errors to the sync status.
Upgrading a partial connection to a full connection
Consider a connect call only providing credentials for downloads (no mutation upload callback) followed by another call only providing a mutation uploader (and no separate credentials).
In this state, should the database be fully connected (using the initial set of credentials for downloads and the second call for uploads), or should only the last connect call have any effect (meaning that the database is upload-only?).
Making connect() calls "add up" feels nicer, but is not consistently possible when e.g. different options like retry delays are passed to the calls. So only letting the last call have an effect is more consistent.
Requesting checkpoints in download-only mode
In upload-only mode, the database has no credentials to connect to the PowerSync service and thus can't request write checkpoints / checkpoint requests. We'd upload everything from ps_crud and then keep the checkpoint gate at i64::max.
In this state, if the database is then connected in download-only mode, should it request a checkpoint? Conceptually, requesting checkpoints is part of the upload process but it probably makes sense to run that once in a download-only connection to make sure we can apply checkpoints.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
TL;DR: This proposes to split the
PowerSyncBackendConnectorinterface into two superinterfaces: One providing credentials (for streaming downloads) and one responsible for uploading changes. We then allow connecting in upload or download-only mode.Current status:
Current ‘backend connector’ architecture
The specifics vary slightly based on the specific SDK, but generally when wiring up the PowerSync Client SDK, you have to define a “backend connector” class with 2 methods:
fetchCredentials()which needs to return an object containing:token: JWT for authentication with PowerSync Service. Developers can either reuse the existing JWT from their authentication provider, or mint a new JWT specific for PowerSync, by calling their own backend APIendpoint: PowerSync Service instance URLuploadData()which gets entries from the upload queue, uploads them to the developer/customer’s backend, and marks those transactions as complete so that they are removed from the upload queueProblem statements
There are several paper cuts involved with the tight coupling of everything in the ‘backend connector’.
fetchCredentials()as part of the “backend connector” does not make sense semantically.fetchCredentials()anduploadData()are both part of a “backend connector” makes the concepts more complicated to explain in our documentation (and makes structuring the documentation more complicated)connect()on thePowerSyncDatabasein order to pass a “backend connector” with anuploadData()implementation. You cannot upload client-side mutations to your backend without connecting to a PowerSync Service instance.fetchCredentials()(PowerSync Service instance endpoint URL and custom backend URL) can cause confusion.uploadData(), it would be better to standardize terminology to refer to the entries in the upload queue as “mutations” everywhere in our code/APIs and documentation. “Mutations” is the most clear description we can use.uploadData()doesn’t align with any of those.Goals
uploadData()— standardize on “mutations”fetchCredentials()— in many/most cases it won’t require fetching from your backend, if you reuse an existing JWTProposal
Note: The internal document lists additional proposals, this is a summary based on discussions in comments.
While preserving backwards compatibility, eliminate the concept of a 'backend connector' in favor of these interfaces (or, where appropriate for an SDK, callback functions):
Depending on the SDK, we then add overloads or optional parameters for
connect():Calling
connect(backendConnector)continues to work.Additional considerations
Local mutations that are never uploaded
What do we do with a database that has pending
ps_cruditems and is then connected in download-only mode? It wouldn't be able to apply checkpoints, and we only surface that as log items which are easy to overlook.To make this error state easier to observe, maybe we should also add these errors to the sync status.
Upgrading a partial connection to a full connection
Consider a connect call only providing credentials for downloads (no mutation upload callback) followed by another call only providing a mutation uploader (and no separate credentials).
In this state, should the database be fully connected (using the initial set of credentials for downloads and the second call for uploads), or should only the last connect call have any effect (meaning that the database is upload-only?).
Making
connect()calls "add up" feels nicer, but is not consistently possible when e.g. different options like retry delays are passed to the calls. So only letting the last call have an effect is more consistent.Requesting checkpoints in download-only mode
In upload-only mode, the database has no credentials to connect to the PowerSync service and thus can't request write checkpoints / checkpoint requests. We'd upload everything from
ps_crudand then keep the checkpoint gate ati64::max.In this state, if the database is then connected in download-only mode, should it request a checkpoint? Conceptually, requesting checkpoints is part of the upload process but it probably makes sense to run that once in a download-only connection to make sure we can apply checkpoints.
All reactions