Merge 4d25a21 into cbdaee0

CHANGELOG.md

@@ -119,4 +119,7 @@ All notable changes to this project will be documented in this file. Breaking ch
 
 ## [1.5.0] = 2021-11-07
 ### Changed
-- Queries will now fully paginate all responses. Prior to this change, ElectroDB would only return items from a single ElectroDB query result. Now ElectroDB will paginate through all query results. This will impact both uses of entity queries and service collections. If a finite number of pages is desired, use the query option [pages](./README.md#query-options) to set a max number of pagination iterations. [[read more](./README.md#query-method)]
+- Queries will now fully paginate all responses. Prior to this change, ElectroDB would only return items from a single ElectroDB query result. Now ElectroDB will paginate through all query results. This will impact both uses of entity queries and service collections. [[read more](./README.md#query-method)]
+- The query option `limit` has an extended meaning with the change to automatically paginate records on query. The option `limit` now represents a target for the number of items to return from DynamoDB. If this option is passed, Queries on entities and through collections will paginate DynamoDB until this limit is reached or all items for that query have been returned. [[read more](./README.md#query-options)]
+### Added
+- A new query option `pages` has been added to coincide with the change to automatically paginate all records when queried. The `pages` option sets a max number of pagination iterations ElectroDB will perform on a query. When this option is paired with `limit`, ElectroDB will respect the first condition reached. [[read more](./README.md#query-options)]

README.md

@@ -2605,7 +2605,7 @@ Option            | Default | Notes
 
 ElectroDB queries use DynamoDB's `query` method to find records based on your table's indexes.
 
-> _NOTE: By default, ElectroDB will paginate through all items that match your query. To limit the number of pages ElectroDB will query through, read more about the [Query Option](#query-options) `pages`, or use the ElectroDB [Pagination API](#page) for fine-grain pagination support._
+> _NOTE: By default, ElectroDB will paginate through all items that match your query. To limit the number of items ElectroDB will retrieve, read more about the [Query Options](#query-options) `pages` and `limit`, or use the ElectroDB [Pagination API](#page) for fine-grain pagination support._
 
 Forming a composite **Partition Key** and **Sort Key** is a critical step in planning **Access Patterns** in **DynamoDB**. When planning composite keys, it is crucial to consider the order in which they are *composed*.  As of the time of writing this documentation, **DynamoDB**  has the following constraints that should be taken into account when planning your **Access Patterns**:
 1. You must always supply the **Partition Key** in full for all queries to **DynamoDB**.
@@ -2817,7 +2817,7 @@ The methods: Get (`get`), Create (`put`), Update (`update`), and Delete (`delete
 
 ElectroDB queries use DynamoDB's `query` method to find records based on your table's indexes. To read more about queries checkout the section [Building Queries](#building-queries)
 
-> _NOTE: By default, ElectroDB will paginate through all items that match your query. To limit the number of pages ElectroDB will query through, read more about the [Query Option](#query-options) `pages`, or use the ElectroDB [Pagination API](#page) for fine-grain pagination support._
+> _NOTE: By default, ElectroDB will paginate through all items that match your query. To limit the number of items ElectroDB will retrieve, read more about the [Query Options](#query-options) `pages` and `limit`, or use the ElectroDB [Pagination API](#page) for fine-grain pagination support._
 
 ### Get Method
 Provide all Table Index composite attributes in an object to the `get` method. In the event no record is found, a value of `null` will be returned.
@@ -4293,7 +4293,7 @@ concurrent      | `1`                  | When performing batch operations, how m
 unprocessed     | `"item"`             | Used in batch processing to override ElectroDBs default behaviour to break apart DynamoDBs `Unprocessed` records into composite attributes. See more detail about this in the sections for [BatchGet](#batch-get), [BatchDelete](#batch-write-delete-records), and [BatchPut](#batch-write-put-records).
 response        | `"default"`          | Used as a convenience for applying the DynamoDB parameter `ReturnValues`. The options here are the same as the parameter values for the DocumentClient except lowercase. The `"none"` option will cause the method to return null and will bypass ElectroDB's response formatting -- useful if formatting performance is a concern.
 ignoreOwnership | `false`              | By default, **ElectroDB** interrogates items returned from a query for the presence of matching entity "identifiers". This helps to ensure other entities, or other versions of an entity, are filtered from your results. If you are using ElectroDB with an existing table/dataset you can turn off this feature by setting this property to `true`.
-limit           | _none_               | A convenience option for the query parameter `Limit`, used to specify the number of records to retrieve while performing a query.
+limit           | _none_               | A target for the number of items to return from DynamoDB. If this option is passed, Queries on entities and through collections will paginate DynamoDB until this limit is reached or all items for that query have been returned.
 pages           | ∞                    | How many DynamoDB pages should a query iterate through before stopping. By default ElectroDB paginate through all results for your query.
 
 # Errors:
@@ -4563,6 +4563,15 @@ When performing a query [Query](#building-queries) you can pass a [Query Options
 *What to do about it:*
 Expect this error only if you're providing a `pages` option. Double-check the value you are providing is the value you expect to be passing, and that the value passes the tests listed above.
 
+### Invalid Limit Option
+*Code: 2006*
+
+*Why this occurred:*
+When performing a query [Query](#building-queries) you can pass a [Query Options](#query-options) called `limit`, which impacts how many DynamoDB items a query should return. Your value should pass the test of both, `!isNaN(parseInt(value))` and `parseInt(value) > 0`.
+
+*What to do about it:*
+Expect this error only if you're providing a `limit` option. Double-check the value you are providing is the value you expect to be passing, and that the value passes the tests listed above.
+
 ### aws-error
 *Code: 4001*
 

src/entity.js

@@ -332,30 +332,45 @@ class Entity {
 			? {}
 			: [];
 		let ExclusiveStartKey;
-		let max = this._normalizePagesValue(config.pages);
+		let pages = this._normalizePagesValue(config.pages);
+		let max = this._normalizeLimitValue(config.limit);
+		let iterations = 0;
 		let count = 0;
 		do {
-			count++;
-			let response = await this._exec("query", {ExclusiveStartKey, ...parameters});
+			let limit = max === undefined
+				? parameters.Limit
+				: max - count;
+			let response = await this._exec("query", {ExclusiveStartKey, ...parameters, Limit: limit});
+
 			ExclusiveStartKey = response.LastEvaluatedKey;
+
 			if (validations.isFunction(config.parse)) {
 				response = config.parse(config, response);
 			} else {
 				response = this.formatResponse(response, parameters.IndexName, config);
 			}
+
 			if (config.raw || config._isPagination) {
 				return response;
 			} else if (config._isCollectionQuery) {
 				for (const entity in response) {
+					if (max) {
+						count += response[entity].length;
+					}
 					results[entity] = results[entity] || [];
 					results[entity] = [...results[entity], ...response[entity]];
 				}
 			} else if (Array.isArray(response)) {
+				if (max) {
+					count += response.length;
+				}
 				results = [...results, ...response];
 			} else {
 				return response;
 			}
-		} while(ExclusiveStartKey && count < max);
+
+			iterations++;
+		} while(ExclusiveStartKey && iterations < pages && (max === undefined || count < max));
 		return results;
 	}
 
@@ -609,6 +624,16 @@ class Entity {
 		return value;
 	}
 
+	_normalizeLimitValue(value) {
+		if (value !== undefined) {
+			value = parseInt(value);
+			if (isNaN(value) || value < 1) {
+				throw new e.ElectroError(e.ErrorCodes.InvalidLimitOption, "Query option 'limit' must be of type 'number' and greater than zero.");
+			}
+		}
+		return value;
+	}
+
 	_deconstructKeys(index, keyType, key, backupFacets = {}) {
 		if (typeof key !== "string" || key.length === 0) {
 			return null;
@@ -780,7 +805,8 @@ class Entity {
 				config.unprocessed = UnprocessedTypes.raw;
 			}
 
-			if (!isNaN(option.limit)) {
+			if (option.limit !== undefined) {
+				config.limit = option.limit;
 				config.params.Limit = option.limit;
 			}
 

src/errors.js

@@ -157,6 +157,12 @@ const ErrorCodes = {
     name: "InvalidPagesOption",
     sym: ErrorCode,
   },
+  InvalidLimitOption: {
+    code: 2006,
+    section: "invalid-limit-option",
+    name: "InvalidLimitOption",
+    sym: ErrorCode,
+  },
   InvalidAttribute: {
     code: 3001,
     section: "invalid-attribute",