Version 3.0 Planning

[fishcharlie]

Version 3.0 Planning

The time has come to start thinking about what the next version of Dynamoose will look like.

Overview

v3.0 is planning to be pretty light in terms of breaking changes compared to v2.0. The main priority for v3.0 is to get an extremely solid foundation to be able to iterate on for a long time to come. I envision all future major versions of Dynamoose to include maybe 1 or 2 breaking changes that are relatively minor in scale. The goal is for this to be the last major version that requires any type of substantial changes to users codebases to get working with the new version. I also envision future major versions of Dynamoose to possibly take place more frequently due to the smaller amount of breaking changes that will occur.

This really includes things like solidifying the concepts of Dynamoose. What does a Model really mean? What should be the behavior of defaults? What does set mean (ex. should it occur before or after type checking, etc)? Lots of foundational questions like that I hope users will ask and discuss as part of our v3.0 process.

Goals

• Upgrade to AWS SDK for Node.js 3.0 [v3] AWS-SDK v3 #1145
• Drop support for older Node.js versions (<v10) [v3] Updating Node.js minimum version to v10 #1146
• Change Document name to Item (to better match DynamoDB naming) Renaming Document to Item #1152
• Move logger to different package
• Improve internal vs. external API
  • Move internal API behind symbol Internal properties #1172
  • Expose more APIs (ex. I use getIndexAttributes in another project, which is really bad practice and doesn't follow the Dynamoose guidelines, we should expose that), Awaiting ready status of models #1218, Public API to get HASH and RANGE key names using a Model instance #1286
• Fix website deployment error
• Changing concept of Model to represent an entity as opposed to having it represent a DynamoDB Table
  • Ensure 100% code coverage
  • Need to figure out how this works for populating
  • Need to ensure transactions work properly here
• Add dynamoose.type.Constant() helper method to easily create constant types
• Add dynamoose.type.Combine() helper method to easily create combine types
• Rename dynamoose.NULL to dynamoose.type.Null
• Rename dynamoose.THIS to dynamoose.type.This
• Change internalSettings to use internalProperties symbol
• Consider switching to Jest

Alpha

• [BUG] Nested required attribute is enforced, even when optional parent is not defined #900
• [FEATURE] Call set on search fields in query/get/delete/scan/update #1093
• The v3 documentation for Model is incorrect, it shows a lot of settings that are no longer valid

Beta

• Add not equals (ne) conditional [FEATURE] #932
• Any type (this should work for both type & schema in Schema objects)
• [BUG] TypeMismatch error when attribute is called 'model' #1023?
• Make FAQ about what order Dynamoose processes events (ie. set/get happen after type check)
• Setup automatic dark mode for documentation based on OS setting
• Support tags for table creation #1129
• Consider making global setting true on indexes by default
• [FEATURE] DynamoDB Infrequent Access #1306
• [FEATURE] Multiple Region Parallel Queries #1002
  • Foundation code
  • Add tests for adding new instances
  • Add documentation
• Consider only initializing Table when request is made (get, update, etc). We could also add a table.initialize method to do this manually.
• Fix code coverage #1320
• Rename Initalizer to be correct spelling
• Add object/item/document wide set, get, validate methods
  • Set
  • Get
  • Validate
• Make Condition.requestObject private
• Make Serializer properties private
• Promote to beta
  • Need to figure out how to merge publish changes made in v3 into v2
  • Create new v2 branch based on main
  • Update README branch strategy
  • Merge v3 into main
  • Publish new v3 release
• [FEATURE] v3 single table attribute mapping #1321
• Make more schema properties private
• Make more item properties private
• Move dynamoose-logger & dynamoose-utils back into primary repo and add system for deploying everything all at once. #1301
• [FEATURE] Create item.applyDefaults method #1155
• Remove all instances of //... in documentation
• Remove content at bottom of Table documentation page #1361
• Throw error if trying to run command on table (ie. get, scan, etc) if table has not been initialized #1360
• Add more JSDoc documentation
  • Build system to convert .d.ts files to .js files to use for documentation. Basically we can take every interface and turn it into an object and then use that to create our documentation.

---

• Plugin Support
  • Need to figure out how this is going to work with TypeScript
• Allow for schema and model prioritization to be impacted by validate methods
• Consider changing rangeKey and hashKey to not be settings of the property but instead be settings of the model (or table in v3)
• Add more functions to dynamoose.Table
• Look into more single table design improvements
• Drop native callback support and encourage users to use util.callbackify as an alternative?
  • After considering this more, I don't think this is going to happen in v3.
• [BUG] Schema set function is not called #977?
• Take TypeScript out of beta? (Will be easier if we move to Jest and write tests in TypeScript)
• Throw warnings for invalid settings (ex. timestamps on Model settings). Need to make sure this is compatible with plugins.
• Disallow Model's having multiple schemas
  • After seeing Best way to implement multiple Schemas in attribute (strategy pattern) #1277, I'm not sure we want to do this. There seem to be some valid use case for multiple schemas on a model. Will have to think about more.
• Consider removing dynamoose.UNDEFINED. This could be handled by having delete obj.a have different behavior than obj.a = undefined. We can check this by using obj.a === undefined and obj.hasOwnProperty('a').
  • This can't be done due to how update works. There is no way to pass in a single object and determine between reverting to the default, and using removing the property.

Timeline

• Discussion/RFC/Proposals - Now
• Development Starts - Now
• Beta Release - TBD
• Final Release - TBD

Blockers

• util-dynamodb TypeScript Errors: Cannot find name 'Blob' Cannot find name 'File' aws/aws-sdk-js-v3#2125

FAQ

How can I submit ideas/goals for v3.0?

Create a new issue and include v3.0 right after the ] in the issue title. Make it clear it's a proposal for v3.0, and I will add it to this issue Goals section above. Your issue you create is where discussion will occur about the proposal.

How can I support v3.0?

Right now, submit feedback! What would you like to see in v3.0? If you could have complete control over Dynamoose and could break everything to make it work better, what would you change? No ideas are off limits here. Even if I shot down an idea in the past because it was a breaking change, now is the time to bring it up again.

As this process moves forward, the ways you can support v3.0 will change. This will include submitting PRs, testing beta versions, etc.

I have an idea for v3.0, but I'm not sure if it's good.

Follow the steps above for How can I submit ideas/goals for v3.0?! Even if you are unsure about it, it will lead to discussion which will benefit Dynamoose.

What does the timeline look like for v3.0?

This is definitely a 2021 project at the earliest. The reason I'm starting discussion now is to keep breaking changes organized. I also want a very extended period where users can comment on ideas for v3.0. Right now, I don't anticipate development starting in 2020. However this could change and I might decide to start minor development in late December.

Can I submit PRs for v3.0 right now?

Yes! Feel free to submit PRs to the v3 branch. It would be a good idea to comment here and tag me so I'm aware and don't overlap with your work.

[cko-moho-khaleqi]

AWS SDK v3 is out for node

[fishcharlie]

@cko-moho-khaleqi Yep! Had a few blockers with it so kept it on there. Haven't gotten a chance to investigate resolutions that I was given yet tho.

[HuEric]

Really neat planning, great work so far from @fishcharlie and all contributors ! 👏
After watching this re:Invent Data modeling with DynamoDB video, I'm looking forward to single table design feature as it's the one of the most efficient ways to use DynamoDB.

[fishcharlie]

@HuEric What exactly are you hoping for here? It's important to know that everything in that video is possible with Dynamoose today. Big difference between single table design feature and Look into more single table design improvements.

I'd love to hear more about what you are looking for here.

[HuEric]

Sure !
What I'm looking at now, is a way to handle multiple schemas when querying one table, like inside RestaurantTable, I would like to have as results Restaurant + Order.
It seems like we are kinda locked into one schema when making queries.

[fishcharlie]

@HuEric That is totally possible right now. When defining your model, just set an array of Schemas, one schema for Restaurant and another for Order. Then in your model you'll just have both of those schemas as an array for the schema argument.

Then when making retrieving or saving documents, it will automatically determine which Schema to use depending on the data in the document.

---

The major improvement we are looking at adding for v3 is changing the idea of a Model to represent an entity or type of data (examples: Restaurant, Order, Person or Movie). Currently, a Model represents a Table in DynamoDB. In v3 we are going to consider changing a Model to represent that entity or type of data.

[HuEric]

Thanks @fishcharlie for clarifying it !
As I didn't see any mention when reading the documentation, I assumed (wrongly) that there is no way to make use of several schemas inside a same table with Dynamoose.

[fishcharlie]

@HuEric https://dynamoosejs.com/guide/Model

You can also pass in an array of Schema instances or schema objects into the schema paremeter. This is useful for cases of single table design where you want one model to have multiple options for a schema. Behind the scenes Dynamoose will automatically pick the closest schema to match to your document, and use that schema for all operations pertaining to that document. If no matching schema can be found, it will default to the first schema in the array.

[fishcharlie]

I'm looking for volunteers who want to help with Dynamoose v3. This will include the following responsibilities:

• Reviewing and testing pull requests
• Providing feedback & suggestions on API/breaking changes
• Testing v3 with your own codebase to help us catch bugs/problems early
• Being able to answer questions about how Dynamoose is being used in your applications to help guide the development process
• Responding to GitHub comments when you are tagged

My only expectation is that volunteers are active in responding to discussions or questions when tagged, and that you put in effort into testing v3 in your own codebase.

This is a great way to get involved in Dynamoose and ensure it continues to remain stable for your codebase without the responsibility that comes with being an active Dynamoose collaborator.

Please message me or post here if you are interested in helping and I'll start tagging you in PRs and discussions related to v3.

If for whatever reason you are unable to continue being active in v3 discussions, please ping me and I will remove you from the list of volunteers I tag in v3 related updates.

[simlevesque]

@fishcharlie I'd be interested in helping you with this. Feel free to message me at any time.

edit: I have an internal fork in a codebase at work so I have some idea of what's inside.

[fishcharlie]

@simlevesque All for the help! Next step is getting #1219 merged in. I just left a comment on that PR for what else needs to be done. Although you don't have write access to it, you could branch off of that branch, make the changes, and submit a different PR that merges your work into #1219.

The other tasks that someone could work on currently (without much interference) include:

• Merging master branch into v3 branch
• Fix website deployment error (run npm run site:build from terminal)
• Add dynamoose.Type.Constant() helper method to easily create constant types (likely a function that takes in a value, and returns an object like: {type: {value: "Constant", settings: {value: /* VALUE PASSED IN*/}}})
• Add dynamoose.Type.Combine() helper method to easily create combine types (same as above but for the Combine type
• Rename dynamoose.NULL to dynamoose.Type.Null
• Rename dynamoose.THIS to dynamoose.Type.This

Then of course just fixing any issues in general on this repo, helps free me up to work more on v3.

[simlevesque]

I'll see what I can do ! give me like a week

[fishcharlie]

@simlevesque If you happen to work on #1219, please do me 2 favors:

1. Create small PRs and submit often (don't want you to have a lot of pending work)
2. Watch the PR comments (ie subscribe to the PR and watch it closely).

Reasons for this is that I might have time to work on it soon (not sure yet tho). So don't want you to be doing work and have me come in and finish it up, rendering your work useless. I'll be sure within the next week if I work on it to comment on the PR tho.

Not sure if that is a concern, but the more communication and less pending work, the better.

Let me know if any of that is a problem.

[simlevesque]

No, none of this is a problem. I can let you know what I'll work on. I guess I'll start from the top of your list.

[fishcharlie]

Closing in favor of the v3 milestone.