[QUESTION] How to use multi-model tables in dynamoose v2?

[Aposhian]

Summary:

#656 was mentioned in the dynamoose v2 rewrite #677. How do I create multi-model tables in dynamoose v2? My main problem is that the table is not being created with all the GSIs, which seemed to be fixed in Dynamoose 2.1.3 with #872, but I'm still having issues.

Code sample

Schema

const schema1 = new dynamoose.Schema({
  pk: {
    type: String,
    hashKey: true
  },
  gsi1: {
    type: String,
    rangeKey: true,
    index: {
      global: true
    }
  },
  gsi2: {
    type: String,
    index: {
      global: true
    }
  }
})

const schema2 = new dynamoose.Schema({
  pk: {
    type: String,
    hashKey: true
  },
  gsi1: {
    type: String,
    rangeKey: true,
    index: {
      global: true
    }
  }
})

Model

In dynamoose v1, I could do

const Model1 = dynamoose.model('MyTable', schema1, {
  create: true,
  waitForActive: true
})

const Model2 = dynamoose.model('MyTable', schema2)

and that worked fine, as long as the attributes of schema2 were a subset of the attributes of schema1.

Now in dynamoose v2 what is the standard way to create multiple Models on the same table WHILE actually creating the table from dynamoose? The above approach works fine if the table already exists.

However, I keep getting errors like

ValidationException: The table does not have the specified index: gsi2

Environment:

Operating System: Ubuntu
Operating System Version: 18.04
Node.js version (node -v): 12.16.3
NPM version: (npm -v): 6.14.4
Dynamoose version: 2.1.3

Other:

• I have read through the Dynamoose documentation before posting this issue
• I have searched through the GitHub issues (including closed issues) and pull requests to ensure this question has not already been raised before
• I have searched the internet and Stack Overflow to ensure this question hasn't been raised or answered before
• I have filled out all fields above
• I am running the latest version of Dynamoose

[fishcharlie]

@Aposhian 😮 I had no idea that was possible. So sorry about breaking that in v2. Single table design is something that I've only recently started to understand. The documentation and support in Dynamoose is pretty poor, which is a result of what you are seeing.

Sadly in v2.x, I'm not sure there is a way to get it as good as v1. That is just due to the fact that v2 made certain decisions that at the time I thought were the best, and I still stand by, but that make this more complicated to fix the way you likely want.

However, please look at #768 (comment). Those are 4 ideas I have to make single table design work better in Dynamoose. That is high priority for me right now and I'm working towards those items. Any input/feedback there would be great. And if you'd like to help implement or test it, shoot me a message on Slack (invite link in README badge) and I'd love to get more people involved.

The biggest thing here is that a Model now represents a DynamoDB Table in v2. As you can see from your code example, Model's in v1 represented this weird mix between a Table & Entity (ie. User, Order, Place, etc.). I believe changing the behavior of Model's to strictly represent a DynamoDB Table was the correct decision. However, it wasn't done with the foresight in mind for single table design. Which I had very little regard and foresight for.

Although I don't regret shipping v2 when I did, single table design is my biggest thing I wish I put more focus on. The items I listed in that comment above, I think will be steps in the right direction. However, even with those changes, there still is no clear concept of an Entity in Dynamoose. Which really sucks. Like you can have your document conform to one Schema or another (as mentioned in the second table item), but how do you enforce that it conforms to a SPECIFIC one. With that change there won't really be a clear way to do that. Or at least the syntax will be weird.

---

Now as for your example. I can't think of a way to enforce that all documents for Model1 conform to schema1, etc. The only way to do what you want as far as I can see, is to create a shared Schema, and have one Model. The downside to this, is shared attributes, enforcing required for one entity and not the other, become basically impossible within Dynamoose.

---

Ok the stuff below is getting into MAJOR future roadmap stuff. Not guaranteed that I'll do any of this (and it could change a lot), but it's kinda a vision I have. Would love some feedback here.

I'm thinking for v3, I want to introduce the concept of a Table. Then change the definition of Model to represent an Entity, as opposed to a Table. Granted, this would require more work for existing Dynamoose users to wrap their heads around, but I'm starting to become a supporter of this idea. Here is how that code you wrote above could look in v3.

const schema1 = new dynamoose.Schema({
  pk: {
    type: String,
    hashKey: true
  },
  gsi1: {
    type: String,
    rangeKey: true,
    index: {
      global: true
    }
  },
  gsi2: {
    type: String,
    index: {
      global: true
    }
  }
})

const schema2 = new dynamoose.Schema({
  pk: {
    type: String,
    hashKey: true
  },
  gsi1: {
    type: String,
    rangeKey: true,
    index: {
      global: true
    }
  }
})

const Model1 = dynamoose.model(schema1)

const Model2 = dynamoose.model(schema2)

const myTable = new dynamoose.Table('MyTable', [Model1, Model2], { // Not sure if this would be `new dynamoose.Table` or `dynamoose.table`
  create: true,
  waitForActive: true
});

Few notes here. myTable would be required, however, it's unclear at this point how you would use it or what the purpose would be. Potentially to get the creation request like you can do currently on the model? Still kinda pointless IMO, and seems like it should do more.

But registering that table with the Model's is required so the Model's know what table they belong to. A Model will also only be allowed to exist on 1 Table. If you try to use a Model without registering it to a Table, it will throw an error since it doesn't know how to handle that.

The biggest downside to this is that it's a breaking change. For every model you now have to create and register a table. For tables with a single entity, this adds an extra step. Potentially we could make this backwards compatible by setting it up so that if the first parameter to dynamoose.model is a String, then automatically create the table and register it. So it's all done in one step at that point. Then it'd require basically no migration for existing users of Dynamoose. But I still think it'd have to be considered a breaking change, since the whole concept of what a Model is will have changed.

There are also questions about how this would impact the plans I have for single table design (as mentioned in the chart comment above) (ie. nested Schemas (what is a Schema anymore?), having multiple Schema's per Model, etc.). I don't have answers to those questions at this time. And I semi doubt if they need answers. It would cause a slightly more bloated code base, and would almost give users too many different ways to achieve the same thing, but it would simplify the roadmap if I don't worry about any of that and just don't answer those questions and allow multiple options.

As for v3 release, as of right now I'm aiming for it to be around the time AWS-SDK v3 ships, so we can migrate to that in v3. But things like this and other breaking features are also dependent on being ready as well.

---

I'd love to chat with you more about this @Aposhian. I've been talking to a lot of people about single table design more recently. And it's a priority for me to improve Dynamoose in this regard. There are work arounds to what you want to do, but your example you provided seems extreme and like you were able to do a lot of things in terms of type safety and type checking that just doesn't exist in v2.

The goal here in the end is to create a better system. So if you are willing I'd love to chat with you more. There is a link to our Slack in a badge on the README. If you do want to help out, ping me and I'd love to chat more!!

[dolsem]

I'm thinking for v3, I want to introduce the concept of a Table. Then change the definition of Model to represent an Entity, as opposed to a Table. Granted, this would require more work for existing Dynamoose users to wrap their heads around, but I'm starting to become a supporter of this idea. Here is how that code you wrote above could look in v3.

I fully support this. I think this is the right way to go for single table design.

Few notes here. myTable would be required, however, it's unclear at this point how you would use it or what the purpose would be. Potentially to get the creation request like you can do currently on the model? Still kinda pointless IMO, and seems like it should do more.

Do note that using single table design a query might return entities of different types. So it probably doesn't make sense to do User.query() if you expect to return a User and a Pet object. You would wanna do something like Table.query() or Table.query<[User, Pet]>().

[fishcharlie]

@dolsem

Do note that using single table design a query might return entities of different types. So it probably doesn't make sense to do User.query() if you expect to return a User and a Pet object. You would wanna do something like Table.query() or Table.query<[User, Pet]>().

Oh wow. Did not even think about that or consider that. Yeah. Umm. That and transactions will also be difficult. This pattern I just suggested works great for saving documents. However retrieving them does get more complicated. Thanks for bringing that up.

Will think about this some more and come up with some more ideas. At some point I will create a new issue about this. Don't really want to hijack this thread too much since talking about this future vision is slightly off topic from the original question and it's still pretty far out.

[github-actions]

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 3 days.

[github-actions]

This issue is stale because it has been open 7 days with no activity. Remove stale label or comment or this will be closed in 3 days.