feat(@aws-amplify/datastore): add Selective Sync #7001

iartemiev · 2020-10-19T13:56:03Z

Works in concert with these CLI changes: aws-amplify/amplify-cli#5586

DataStore - Selective Sync Feature

Enables developers to filter data on the back end before it gets synced down into their users' local stores, as opposed to syncing the entire contents of the associated data source.

Additional details:

Provides code completion ("Intellisense") for returning a valid expression (just like Query predicates).
Applies the same filtering to inbound AppSync subscriptions.
Adds new DataStore.stop() method for stopping the sync & subscriptions without clearing out the contents of the local store.

Misc.:

renames asyncstorage.ts -> AsyncStorageAdapter.ts
renames indexeddb.ts -> IndexedDBAdapter.ts
make Reachability.ts more WebWorker-friendly

Addresses the following feature requests/issues:

Basic usage

Selective Sync is configured by specifying syncExpressions, a newly added optional configuration parameter.
Developers can choose which model(s) they would like to apply an expression to.

import { DataStore, syncExpression } from 'aws-amplify';
import { Post, Comment } from './models';

DataStore.configure({
  syncExpressions: [
    syncExpression(Post, () => {
      return (c) => c.rating('gt', 5);
    }),
    syncExpression(Comment, () => {
      return (c) => c.status('eq', 'active');
    }),
  ]
});

When DataStore starts syncing, only Posts with rating > 5 and Comments with status === 'active' will be synced down to the user's local store.
Developers should only specify a single syncExpression per model. Any subsequent expressions for the same model will be ignored.

Reevalute expressions at runtime

Sync expressions get evaluated in DataStore.start() so it's possible to reevaluate them at runtime.

let rating = 5;

DataStore.configure({
  syncExpressions: [
    syncExpression(Post, () => {
      return (c) => c.rating('gt', rating));
    })
  ]
});

If you want to start syncing down Posts with rating > 1, you can do:

async function changeSync() {
  rating = 1;
  await DataStore.stop(); // or DataStore.clear() if you also want to empty the contents of the local store
  await DataStore.start();
}

Upon calling start() (or executing a DataStore operation, e.g., query, save, delete, or observe), DataStore will reevaluate the syncExpressions.
In the above case, the predicate will contain the value 1, so all Posts with rating > 1 will get synced down.

Using async expressions

You can pass a Promise to syncExpression:

DataStore.configure({
  syncExpressions: [
    syncExpression(Post, async () => {
      const ratingValue = await getRatingValue();
      return (c) => c.rating('gt', ratingValue);
    })
  ]
});

DataStore will wait for the Promise to resolve before applying the expression to sync. Async expressions can also be reevaluated at runtime, just like synchronous expressions (see previous section).

Shorthand

If you don't need to add any logic to your syncExpression, you can use the following shorthand (returning the predicate, as opposed to a function or promise that in turn returns a predicate like in the above examples):

DataStore.configure({
  syncExpressions: [
    syncExpression(Post, (c) => c.rating('gt', 5))
  ]
});

Advanced use case

You can utilize Selective Sync in order to have your base sync retrieve items from DynamoDB via query against a table, LSI, or GSI. By default, the base sync will perform a scan.
In order to do that, your syncExpression should return a predicate that maps to a queryable expression.

For example, for the following schema:

type User @model
  @key(name: "byLastName", fields: ["lastName", "createdAt"]) {
  id: ID!
  firstName: String!
  lastName: String!
  createdAt: AWSDateTime!
}

This sync expression will produce a query operation:

DataStore.configure({
  syncExpressions: [
    syncExpression(User, async () => {
      const lastName = await getLastNameForSync();
      return (c) => c.lastName('eq', lastName).createdAt('gt', '2020-10-10')
    })
  ]
});

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

lgtm-com · 2020-10-19T14:03:13Z

This pull request fixes 1 alert when merging 462305b into 377acd0 - view on LGTM.com

fixed alerts:

1 for Unused variable, import, function or class

lgtm-com · 2020-10-19T14:17:39Z

This pull request fixes 1 alert when merging 4c8e9ce into 377acd0 - view on LGTM.com

fixed alerts:

1 for Unused variable, import, function or class

codecov · 2020-10-19T14:23:05Z

Codecov Report

Merging #7001 into main will decrease coverage by 0.30%.
The diff coverage is 39.43%.

@@            Coverage Diff             @@
##             main    #7001      +/-   ##
==========================================
- Coverage   73.28%   72.97%   -0.31%     
==========================================
  Files         213      213              
  Lines       13197    13295      +98     
  Branches     2585     2604      +19     
==========================================
+ Hits         9671     9702      +31     
- Misses       3332     3397      +65     
- Partials      194      196       +2

Impacted Files	Coverage Δ
packages/core/src/Util/Reachability.ts	`33.33% <0.00%> (+1.19%)`	⬆️
packages/datastore/src/sync/index.ts	`15.07% <0.00%> (-0.34%)`	⬇️
packages/datastore/src/datastore/datastore.ts	`77.43% <18.86%> (-7.82%)`	⬇️
packages/datastore/src/sync/processors/sync.ts	`16.66% <21.05%> (ø)`
...ages/datastore/src/sync/processors/subscription.ts	`39.65% <23.07%> (-0.59%)`	⬇️
...ore/src/storage/adapter/getDefaultAdapter/index.ts	`75.00% <50.00%> (ø)`
packages/datastore/src/types.ts	`83.60% <50.00%> (-1.14%)`	⬇️
...tastore/src/storage/adapter/AsyncStorageAdapter.ts	`77.21% <90.90%> (ø)`
packages/datastore/src/sync/utils.ts	`82.28% <94.44%> (+1.27%)`	⬆️
.../datastore/src/storage/adapter/IndexedDBAdapter.ts	`77.84% <100.00%> (ø)`
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 958f61e...374d241. Read the comment docs.

lgtm-com · 2020-10-19T22:58:27Z

This pull request fixes 1 alert when merging c237a19 into 7bdb09f - view on LGTM.com

fixed alerts:

1 for Unused variable, import, function or class

lgtm-com · 2020-10-19T23:46:55Z

This pull request fixes 1 alert when merging 7375ba7 into 59fdb08 - view on LGTM.com

fixed alerts:

1 for Unused variable, import, function or class

packages/datastore/src/datastore/datastore.ts

lgtm-com · 2020-10-21T02:03:48Z

This pull request fixes 1 alert when merging 9ffe730 into a85c724 - view on LGTM.com

fixed alerts:

1 for Unused variable, import, function or class

lgtm-com · 2020-10-21T12:43:08Z

This pull request fixes 1 alert when merging 2bb1d06 into 96fda2a - view on LGTM.com

fixed alerts:

1 for Unused variable, import, function or class

iartemiev · 2020-10-21T19:00:36Z

packages/datastore/src/storage/adapter/AsyncStorageAdapter.ts

+import {
+	ModelPredicateCreator,
+	ModelSortPredicateCreator,


Just FYI to reviewers, the sort code is showing up in the diff for AsyncStorageAdapter and IndexedDBAdatper due to a file rename and delete. It's not actually new code, it's just that we had duplicate files (e.g., asyncstorage.ts and AsyncStorageAdapter.ts and the latter was out of date).

I deleted AsyncStorageAdapter.ts and then renamed asyncstorage.ts -> AsyncStorageAdapter.ts. Same for IDB

amhinson

Looks good to me! 👍

packages/datastore/src/types.ts

manueliglesias

This looks great! I am very excited for this feature 😄

I added minor comments.

packages/datastore/src/types.ts

manueliglesias · 2020-10-22T03:34:10Z

packages/datastore/src/types.ts

 	};
 	conflictHandler?: ConflictHandler; // default : retry until client wins up to x times
 	errorHandler?: (error: SyncError) => void; // default : logger.warn
 	maxRecordsToSync?: number; // merge
 	syncPageSize?: number;
 	fullSyncInterval?: number;
-};
+	syncExpressions?: SyncExpression<any>[];


Same as above

manueliglesias · 2020-10-22T04:02:15Z

packages/datastore/src/types.ts

+export type SyncExpression<T extends PersistentModel> = Promise<{
+	modelConstructor: PersistentModelConstructor<T>;
+	conditionProducer:
+		| ProducerModelPredicate<T>
+		| (() => ProducerModelPredicate<T>)
+		| (() => Promise<ProducerModelPredicate<T>>);
+}>;


I think this type works better (is more flexible for the future) if you drop the Promise from here and instead add it as needed where you consume it.

manueliglesias · 2020-10-22T04:12:08Z

packages/datastore/src/sync/utils.ts

+	result[type] = predicates.map(p => {
+		if (isPredicateObj(p)) {
+			const { field, operator, operand } = p;
+
+			return { [field]: { [operator]: operand } };
+		} else {
+			result[p.type] = predicateToGraphQLCondition(p);
+		}
+	});


This map operation is confusing, it is only returning a mapped element in one circumstance and modifying an object from the outside scope in the other. This should probably be a forEach or re-written as something clearer.

packages/datastore/src/sync/utils.ts

packages/datastore/src/storage/adapter/IndexedDBAdapter.ts

packages/datastore/src/datastore/datastore.ts

lgtm-com · 2020-10-23T14:32:51Z

This pull request fixes 2 alerts when merging d60587d into abb5e36 - view on LGTM.com

fixed alerts:

2 for Unused variable, import, function or class

lgtm-com · 2020-10-23T14:49:45Z

This pull request introduces 1 alert and fixes 2 when merging 5104d0e into abb5e36 - view on LGTM.com

new alerts:

1 for Unused variable, import, function or class

fixed alerts:

2 for Unused variable, import, function or class

lgtm-com · 2020-10-23T14:59:54Z

This pull request fixes 2 alerts when merging a2146c0 into 958f61e - view on LGTM.com

fixed alerts:

2 for Unused variable, import, function or class

manueliglesias

LGTM, just wait for @elorzafe 's review

👍

Thanks @iartemiev !

elorzafe

Thanks @iartemiev this looks great!

I would double check stop function to see there is no race condition in case the app invokes start and stop repeatedly

packages/datastore/src/datastore/datastore.ts

lgtm-com · 2020-10-28T01:14:08Z

This pull request fixes 2 alerts when merging 374d241 into 958f61e - view on LGTM.com

fixed alerts:

2 for Unused variable, import, function or class

elorzafe

Thanks @iartemiev this is awesome!! 🌮 🎉 🥇

packages/datastore/src/datastore/datastore.ts

LaurentEscalier · 2020-11-25T09:06:39Z

Hey there 👋
Hey @iartemiev

I've updated aws-amplify to the latest version in my project and have since then been facing the following issue.

Datastore now adds a $filter parameter of type Model${typeName}FilterInput to the GraphQL sync operations that my app is sending:

{
   "query":"query operation($limit: Int, $nextToken: String, $lastSync: AWSTimestamp, $filter: ModelMyModelFilterInput) {\n  syncMyModels(limit: $limit, nextToken: $nextToken, lastSync: $lastSync, filter: $filter) {\n    items {\n      id\n      data\n      _version\n      _lastChangedAt\n      _deleted\n    }\n    nextToken\n    startedAt\n  }\n}\n",
   "variables":{
      "limit":1000,
      "nextToken":null,
      "lastSync":0,
      "filter":null
   }
}

Which results in the following errors in the response.

{
   "data":null,
   "errors":[
      {
         "path":null,
         "locations":[
            {
               "line":1,
               "column":84,
               "sourceName":null
            }
         ],
         "message":"Validation error of type UnknownType: Unknown type ModelMyModelFilterInput"
      },
      {
         "path":null,
         "locations":[
            {
               "line":2,
               "column":84,
               "sourceName":null
            }
         ],
         "message":"Validation error of type UnknownArgument: Unknown field argument filter @ 'syncMyModels'"
      }
   ]
}

Now, I understand why this happens: the related AppSync Backend schema does not expect any $filter parameter on this operation and also doesn't know about a type named ModelMyModelFilterInput.

The way I see it is that this Pull Request has added a change (somewhere around here) that forces me to update my AppSync Backend.

How should I proceed with that ? Where is this required Backend change documented ?

[Edit] Note that my team has previously decided to work without the Amplify CLI. So we have generated the related AppSync Backend by using a by-AWS-provided CloudFormation Stack template.

iartemiev · 2020-11-25T14:15:29Z

@LaurentEscalier there is no backend change required for those using the CLI - which is the recommended approach for DataStore - as we've always included an optional filter argument in the generated query input type. It was just unused until this change.

In your case, you can run amplify api gql-compile in the root of your project and then copy the generated schema from amplify/backend/api/your-app-name/build/schema.graphql to the schema location referenced in your CloudFormation template. Or you can copy-paste the input types piecemeal from the generated schema into the schema you're using. See this doc for more info on how to update the schema in a CF stack.

github-actions · 2022-06-02T02:14:50Z

This pull request has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels or Discussions for those types of questions.

iartemiev added the DataStore Related to DataStore category label Oct 19, 2020

iartemiev requested review from manueliglesias, amhinson and elorzafe October 19, 2020 13:56

iartemiev self-assigned this Oct 19, 2020

feat(@aws-amplify/datastore): add Selective Sync

4c8e9ce

iartemiev force-pushed the selective-sync branch from 462305b to 4c8e9ce Compare October 19, 2020 14:10

undefobj mentioned this pull request Oct 19, 2020

DataStore: support for multi-tenant apps with sharing? #5119

Closed

Merge branch 'main' into selective-sync

c237a19

Merge branch 'main' into selective-sync

7375ba7

iartemiev mentioned this pull request Oct 20, 2020

DataStore SYNC query builder does not allow/include filter by owner #6595

Closed

amhinson reviewed Oct 20, 2020

View reviewed changes

packages/datastore/src/datastore/datastore.ts Outdated Show resolved Hide resolved

run base sync on model when sync expression changes

9ffe730

Merge branch 'main' into selective-sync

2bb1d06

iartemiev commented Oct 21, 2020

View reviewed changes

amhinson approved these changes Oct 21, 2020

View reviewed changes

iartemiev marked this pull request as ready for review October 22, 2020 02:24

manueliglesias reviewed Oct 22, 2020

View reviewed changes

packages/datastore/src/types.ts Outdated Show resolved Hide resolved

manueliglesias suggested changes Oct 22, 2020

View reviewed changes

sammartinez added the needs-review label Oct 22, 2020

iartemiev added 2 commits October 23, 2020 10:22

Merge branch 'main' into selective-sync

1965209

pr review requested changes

d60587d

fix type

5104d0e

iartemiev requested a review from manueliglesias October 23, 2020 14:46

iartemiev added 2 commits October 23, 2020 10:51

remove unused import

47484c9

Merge branch 'main' into selective-sync

a2146c0

manueliglesias approved these changes Oct 27, 2020

View reviewed changes

elorzafe suggested changes Oct 28, 2020

View reviewed changes

packages/datastore/src/datastore/datastore.ts Show resolved Hide resolved

packages/datastore/src/datastore/datastore.ts Show resolved Hide resolved

pr review requested changes 2

374d241

iartemiev requested a review from elorzafe October 28, 2020 13:34

elorzafe approved these changes Oct 28, 2020

View reviewed changes

packages/datastore/src/datastore/datastore.ts Show resolved Hide resolved

iartemiev merged commit 8fa348b into aws-amplify:main Oct 28, 2020

jamesonwilliams mentioned this pull request Oct 31, 2020

[DataStore] CursorWindowAllocationException crash for large datasets aws-amplify/amplify-android#944

Closed

CryogenicPlanet pushed a commit to MLH-Fellowship/amplify-js that referenced this pull request Jan 20, 2021

feat(@aws-amplify/datastore): add Selective Sync (aws-amplify#7001)

139538b

iartemiev removed the needs-review label Jun 1, 2021

github-actions bot locked as resolved and limited conversation to collaborators Jun 2, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(@aws-amplify/datastore): add Selective Sync #7001

feat(@aws-amplify/datastore): add Selective Sync #7001

iartemiev commented Oct 19, 2020 •

edited

Loading

lgtm-com bot commented Oct 19, 2020

lgtm-com bot commented Oct 19, 2020

codecov bot commented Oct 19, 2020 •

edited

Loading

lgtm-com bot commented Oct 19, 2020

lgtm-com bot commented Oct 19, 2020

lgtm-com bot commented Oct 21, 2020

lgtm-com bot commented Oct 21, 2020

iartemiev Oct 21, 2020 •

edited

Loading

amhinson left a comment

manueliglesias left a comment

manueliglesias Oct 22, 2020

manueliglesias Oct 22, 2020

manueliglesias Oct 22, 2020

lgtm-com bot commented Oct 23, 2020

lgtm-com bot commented Oct 23, 2020

lgtm-com bot commented Oct 23, 2020

manueliglesias left a comment

elorzafe left a comment

lgtm-com bot commented Oct 28, 2020

elorzafe left a comment

LaurentEscalier commented Nov 25, 2020 •

edited

Loading

iartemiev commented Nov 25, 2020

github-actions bot commented Jun 2, 2022

feat(@aws-amplify/datastore): add Selective Sync #7001

feat(@aws-amplify/datastore): add Selective Sync #7001

Conversation

iartemiev commented Oct 19, 2020 • edited Loading

DataStore - Selective Sync Feature

Basic usage

Reevalute expressions at runtime

Using async expressions

Shorthand

Advanced use case

lgtm-com bot commented Oct 19, 2020

lgtm-com bot commented Oct 19, 2020

codecov bot commented Oct 19, 2020 • edited Loading

Codecov Report

lgtm-com bot commented Oct 19, 2020

lgtm-com bot commented Oct 19, 2020

lgtm-com bot commented Oct 21, 2020

lgtm-com bot commented Oct 21, 2020

iartemiev Oct 21, 2020 • edited Loading

Choose a reason for hiding this comment

amhinson left a comment

Choose a reason for hiding this comment

manueliglesias left a comment

Choose a reason for hiding this comment

manueliglesias Oct 22, 2020

Choose a reason for hiding this comment

manueliglesias Oct 22, 2020

Choose a reason for hiding this comment

manueliglesias Oct 22, 2020

Choose a reason for hiding this comment

lgtm-com bot commented Oct 23, 2020

lgtm-com bot commented Oct 23, 2020

lgtm-com bot commented Oct 23, 2020

manueliglesias left a comment

Choose a reason for hiding this comment

elorzafe left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Oct 28, 2020

elorzafe left a comment

Choose a reason for hiding this comment

LaurentEscalier commented Nov 25, 2020 • edited Loading

iartemiev commented Nov 25, 2020

github-actions bot commented Jun 2, 2022

iartemiev commented Oct 19, 2020 •

edited

Loading

codecov bot commented Oct 19, 2020 •

edited

Loading

iartemiev Oct 21, 2020 •

edited

Loading

LaurentEscalier commented Nov 25, 2020 •

edited

Loading