Datastore syncExpressions in a one to many relationship?

[DevTGhosh]

Which Category is your question related to?
Datastore

What AWS Services are you utilizing?
Appsync
Datastore

Provide additional details e.g. code snippets
Is it possible with Datastore syncExpressions in a one to many relationship if we sync the top model does the related items get synced too and nothing else?

Say for example

type Post @model {
  id: ID!
  comments: [Comment] @connection(keyName: "byPost", fields: ["id"])
}

type Comment @model
  @key(name: "byPost", fields: ["postID", "content"]) {
  id: ID!
  postID: ID!
  content: String!
}

Is it posible if we sync just Post model with Id then all it's related comments are synced and not all comments.

DataStore.configure({
  syncExpressions: [
    syncExpression(Post, () => {
      return (c) => c.id('eq', '123')
    }),
  ],
})

[mir1198yusuf]

This is a very much needed feature so we can only write syncExpression for top models and all related ones will be sync

[attilah]

@DevTGhosh Transferred to the JS repo as it is related more to the JS side and DataStore in general than CLI.

[iartemiev]

@DevTGhosh not at the moment. You would need to have a separate sync expression for the related items. E.g.,

DataStore.configure({
  syncExpressions: [
    syncExpression(Post, () => {
      return (c) => c.id('eq', '123')
    }),
    syncExpression(Comment, () => {
      return (c) => c.postID('eq', '123')
    }),
  ],
})

[mir1198yusuf]

@iartemiev , I can see that you passed just one postId for Comment table.
suppose I want to fetch all posts that user is part of, then all comments for only each of this post .

How can I make such expression?

The condition for Comment syncExpression depends on array returned by Post syncExpression.

[DevTGhosh]

Basically what we are asking for is say we fetch Posts by tag but wanna get Comments of the posts we get which are related by id

type Post @model {
  id: ID!
  tag: String!
  comments: [Comment] @connection(keyName: "byPost", fields: ["id"])
}

type Comment @model
  @key(name: "byPost", fields: ["postID", "content"]) {
  id: ID!
  postID: ID!
  content: String!
}

DataStore.configure({
  syncExpressions: [
    syncExpression(Post, () => {
      return (c) => c.tag('eq', 'cooking')
    }),
    syncExpression(Comment, () => {
      // what would we write here so that we can get the id of the posts above and sync comments byPoistId
    }),
  ],
})

[iyz91]

@iartemiev @renebrandel Out of curiosity at this point, what would the solution to the above be, if there is? Telling your sync engine to stop and start for many predicates is pretty odd.

If there's no current solution, it seems DataStore was not designed from the outset for many common application design patterns. Seems to be more useful for small-medium internal organization apps and demos, but not much else in the real world, which would generally be fine if the docs and AWS blog posts made that clear (which they don't). Please correct me if I'm wrong.

[renebrandel]

We’ve recently launched sync expressions and are working on improvements now that we have more feedback from customers (like everyone on this thread). Over the next few months, we’re spending some time to rethink how relationships are handled. Look out for an RFC soon.

[iyz91]

@renebrandel Great, thank your for the prompt response and update. In the meantime, is it safe to assume to that the AWS AppSync Client SDK would generally work for these more advanced scenarios with offline use, as indicated here: https://docs.amplify.aws/lib/graphqlapi/offline/q/platform/js?

I know there's never a 100% guarantee, but my concerns are it seems there's no offline support past apollo client ver 2.4.6 and I don't want to run into hidden issues down the road like I did with DataStore. Are you aware of any relatively complex implementations with subscriptions and offline use?

[renebrandel]

It'll depend on the scenarios itself. For offline-first applications, we do recommend DataStore as it provides many out-of-the-box benefits without manually configuring caching logic. AppSync Client SDK might be a better approach if you want to manually tweak those settings. You'll end up though manually managing conflict resolution logic etc. on top of it. If you can provide the use cases you're trying to tackle, then I can provide a recommendation.

[mir1198yusuf]

@renebrandel -
You can see this use case #7534 (comment)

@iartemiev is already familiar with this issue. He has also tried few approaches and came to some conclusions which he has mentioned in that issue.

This use case covers most real-world scenarios. If you can consider this use case with priority , I think Datastore could become a successful product given the promises it makes. Multi-tenancy, dynamic selective syncing (not the kind in docs), etc are many issue which can be covered in this whatsapp clone use case. But make sure to fulfill all criteria as mentioned in the use case.

Just try to fit Datastore on this use case and you will understand the shortcomings of Datastore

[iyz91]

@renebrandel Yeah I figured as much. The use case is fairly complex:

• Organizations --> Projects --> Tasks
• Users belong to N organizations, with privileges as admin or member
• Organizations contain N projects
• Projects contain N tasks
• Users can be added to projects, with privileges as editor or viewer
• Users can be assigned to tasks within projects
• "Chat rooms" exist within Organizations with varying users
• Ideally, real-time updates should be emitted for each model type, especially chats, tasks, projects
• Users cannot, in any case, get data (UI or DB level) that they are not privileged to
• Number of items can get very big, e.g. tasks, so replicating cloud store is not feasible
• Tasks, for example, would belong many different projects from multiple organizations, so selective sync won't work AFAIK

So in a broad sense it's a mix of project/task management + whatsapp. Like @mir1198yusuf mentioned, if the whatsapp use case can be implemented at scale, you guys probably have a winning product. However, I can't wait months and months for something that (might) work. I would love to use DataStore, it's premise is amazing and the last thing I want to do is manage conflict resolution, but I don't believe it can do what I need.

With my use case above, do you think the AppSync SDK route will work (assuming DataStore route already doesn't)?

[iyz91]

@renebrandel @undefobj Just providing an update. I evaluated the AppSync SDK route, primarily revolving around the old Apollo client it relies on and outstanding issues that seem to highlight the library has been abandoned, which would be risky for production even if the outstanding issues wouldn't affect me. This is unfortunate as for many users a write-through cache setup is sufficient for optimistic response and general offline use.

With DataStore not currently ready for multi-tenancy, model relationships (such as the one detailed in this issue) and subscription/sync issues, among others, and the AppSync SDK generally abandoned, customers are left with no pre-built, production ready offline capability when using AppSync for these types of apps. I could, of course, develop/adapt my own solution but part of the reason why I chose this framework was because of what was presented in the docs and blog posts for AppSync/Amplify/DataStore for offline capabilities. From what I can tell, trying to implement Apollo Client for its in memory cache with AppSync would probably be more trouble than it's worth considering the AppSync specific implementation differences and time limitations.

So I myself have opted to forgo offline use for now while using the Amplify API library for the sake of time and existing investment into the framework. The majority of my real-world uses cases will have sufficient internet access so it's an acceptable tradeoff for now. The GQL transformers and the other categories in general still work well and save me significant time. This is with the hope that, with the highest priority of the Amplify team, DataStore will be partly redesigned to generally accommodate multi-tenancy and will be a relatively easy drop-in replacement in the future for those already using the Amplify API library. Ideally, like the framework itself, "escape hatches" can be provided with write-through capability for non-trivial store/sync scenarios. If this is prioritized, and if it is successful for use cases such as a WhatsApp clone, I wouldn't be surprised if Amplify, AppSync and related services see a big jump in adoption (not that AWS is hurting of course). For now, I would strongly urge you to make the limitations clear in your documentation, especially the DataStore sections.

I know this work and DataStore in particular is terribly complex and the Amplify team is undoubtedly putting in incredible work. I'm just providing my 2 cents from the customer point of view. If anything I stated is inaccurate or you are aware of alternative solutions, I would appreciate any clarification and direction.

[Vingtoft]

Any updates on this?

[meducati]

A year later on this important issue. Any updates?

[teamparlor]

We're also in this boat. But I'm wonder the backend auth rules apply to the sync. So for example all our models relate to the organization they belong to. You must be in the organization cognito group to access the data you have access to. We also have layered a staff group which has full access. In this scenario would the sync for our customers only sync their data?

[wvidana]

Nothing? The linked RFC is not meant for syncExpressions, so we are still stuck with the same problem after a year #8901

Even if not a syncExpression for relationships, some operator to compare from a list, like this:

syncExpression(Project, () => {
  return u => u.task('in', listOfTasksForThisParticularUser)
}),

[tyarai]

Any updates on this issue? Thanks

[duckbytes]

This is my particular use case that I'm looking for a solution on.

Relevant parts of my schema:

type Task
@auth(rules: [
  {allow: private, operations: [read]},
  {allow: groups, groups: ["COORDINATOR", "RIDER", "ADMIN"], operations: [create, read, update]},
])
@model {
  id: ID!
  tenantId: ID! @index(name: "byTenantId")  
  dateCreated: AWSDate!
  assignees: [TaskAssignee] @hasMany(indexName: "byTask", fields: ["id"])
  status: TaskStatus @index(name: "byStatus", sortKeyFields: ["dateCreated"])
}

type TaskAssignee
@auth(rules: [
  {allow: private, operations: [read]},
  {allow: groups, groups: ["COORDINATOR", "RIDER", "ADMIN"], operations: [create, read, update, delete]},
])
@model {
  id: ID!
  tenantId: ID! @index(name: "byTenantId")
  taskId: ID! @index(name: "byTask", sortKeyFields: ["assigneeId"])
  assigneeId: ID! @index(name: "byAssignee", sortKeyFields: ["taskId"])
  role: Role!
  task: Task! @belongsTo(fields: ["taskId"])
  assignee: User! @belongsTo(fields: ["assigneeId"])
}

enum TaskStatus {
  NEW
  ACTIVE
  PICKED_UP
  DROPPED_OFF
  CANCELLED
  REJECTED
  ABANDONED
  COMPLETED
}

This is the syncExpression I'm using:

syncExpression(models.Task, (m) =>
    m
        .tenantId("eq", tenantId)
        .or((task) =>
            task
                .status("eq", "NEW")
                .status("eq", "ACTIVE")
                .status("eq", "PICKED_UP")
                .status("eq", "DROPPED_OFF")
                .dateCreated("gt", oneWeekAgo)
        )
),

I'd like to be able to return Task records that are either not completed (anything with new, active, picked up or dropped off status) or are completed (completed, rejected, cancelled, abandoned) but were created in the last week.

The stumbling block I'm coming upon is that my many to many table TaskAssignees is now trying to resolve non-nullable records that don't exist locally.

I get this error now when trying to make any query on TaskAssignees:

I'm not sure how to do a selective sync on Task while also only returning TaskAssignee records that can be resolved.

The only solution I can think of is to copy data (dateCreated, status) from each Task record to the TaskAssignee records associated with it, but that could be tricky and means there are two sources of truth for this data.