Implement async iterables

README.md

@@ -385,7 +385,7 @@ stringifier           DOMString operation(); // `toString()` is not needed.
 
 IDL indexed and named properties require multiple things on the implementation class to work properly.
 
-The first is the getters, (optional) setters, and (optional) deleters operations. Much like stringifiers, getters, setters, and deleters can either be standalone or aliased to a named operation (though not an attribute). If an operation is standalone, then the implementation class must implement following symbol-named methods. The `utils` object below refers to the default export from the generated utilities file `utils.js`.
+The first is the getters, (optional) setters, and (optional) deleters operations. Much like stringifiers, getters, setters, and deleters can either be standalone or aliased to a named operation (though not an attribute). If an operation is standalone, then the implementation class must implement the following symbol-named methods. The `utils` object below refers to the default export from the generated utilities file `utils.js`.
 
 - Getters: `utils.indexedGet`, `utils.namedGet`
 - Setters: `utils.indexedSetNew`, `utils.indexedSetExisting`, `utils.namedSetNew`, `utils.namedSetExisting`
@@ -395,6 +395,20 @@ The second is the interface's supported property indices/names. By default, the
 
 If the getter function always returns a constant value for unsupported properties, webidl2js also offers a non-standard extended attribute `[WebIDL2JSValueAsUnsupported]` (documented below) that would simply call the getter function to check if a property index/name is supported, so that `supportsPropertyIndex`/`supportsPropertyName` would not need to be implemented separately. However, when using the extended attribute, be very sure that the value specified in the attribute is returned *if and only if* the property is unsupported.
 
+### Iterables
+
+For synchronous value iterable declarations, there is no need to add implementation-class code: they will be automatically generated based on the indexed property getter and `length` property.
+
+For synchronous pair iterable declarations, the implementation class needs to implement the `[Symbol.iterator]()` property, returning an iterable of `[key, value]` pairs. These can be impls; the generated code will convert them into wrappers as necessary.
+
+### Async iterables
+
+[Asynchronous iterable declarations](https://heycam.github.io/webidl/#idl-async-iterable) require the implementation class to implement the following symbol-named methods, corresponding to algorithms from the Web IDL specification. The `utils` object below refers to the default export from the generated utilities file `utils.js`.
+
+- `utils.asyncIteratorNext`: corresponds to the [get the next iteration result](https://heycam.github.io/webidl/#dfn-get-the-next-iteration-result) algorithm, and receives a single argument containing an instance of the generated async iterator. For pair asynchronous iterables, the return value must be a `[key, value]` pair array, or `utils.asyncIteratorEOI` to signal the end of the iteration. For value asynchronous iterables, the return value must be the value, or `utils.asyncIteratorEOI` to signal the end of the iteration.
+- `utils.asyncIteratorInit`: corresponds to the [asynchronous iterator initialization steps](https://heycam.github.io/webidl/#asynchronous-iterator-initialization-steps), and receives two arguments: the instance of the generated async iterator, and an array containing the post-conversion arguments. This method is optional.
+- `utils.asyncIteratorReturn`: corresponds to the [asynchronous iterator return](https://heycam.github.io/webidl/#asynchronous-iterator-return) algorithm, and receives two arguments: the instance of the generated async iterator, and the argument passed to the `return()` method. This method is optional.
+
 ### Other, non-exposed data and functionality
 
 Your implementation class can contain other properties and methods in support of the wrapped properties and methods that the wrapper class calls into. These can be used to factor out common algorithms, or store private state, or keep caches, or anything of the sort.
@@ -449,6 +463,7 @@ webidl2js is implementing an ever-growing subset of the Web IDL specification. S
   - Stringifiers
   - Named and indexed `getter`/`setter`/`deleter` declarations
   - `iterable<>` declarations
+  - `async iterable<>` declarations
   - Class strings (with the semantics of [heycam/webidl#357](https://github.com/heycam/webidl/pull/357))
 - Dictionary types
 - Enumeration types
@@ -508,6 +523,12 @@ A couple of non-standard extended attributes are baked in to webidl2js:
 
 When the `[WebIDL2JSCallWithGlobal]` extended attribute is specified on static IDL operations, the generated interface code passes the [current global object](https://html.spec.whatwg.org/multipage/webappapis.html#current-global-object) as the first parameter to the implementation code. All other parameters follow `globalObject` and are unchanged. This could be used to implement factory functions that create objects in the current realm.
 
+### `[WebIDL2JSHasReturnSteps]`
+
+This extended attribute can be applied to async iterable declarations. It declares that the implementation class will implement the `[idlUtils.asyncIteratorReturn]()` method.
+
+This is necessary because we need to figure out at code-generation time whether to generate a `return()` method on the async iterator prototype. At that point, only the Web IDL is available, not the implementation class properties. So, we need a signal in the Web IDL itself.
+
 ### `[WebIDL2JSValueAsUnsupported=value]`
 
 This extended attribute can be applied to named or indexed getters or setters. It says that whether the interface supports a given property name/index can be automatically derived by looking at the return value of its indexed getter/setter: whenever `value` is returned, the name/index is unsupported. Typically, `value` is either `undefined` or `_null`.

lib/constructs/async-iterable.js

@@ -0,0 +1,70 @@
+"use strict";
+
+const utils = require("../utils");
+const { generateAsyncIteratorArgConversions } = require("../parameters");
+
+class Iterable {
+  constructor(ctx, I, idl) {
+    this.ctx = ctx;
+    this.interface = I;
+    this.idl = idl;
+    this.name = idl.type;
+  }
+
+  get isValue() {
+    return this.idl.idlType.length === 1;
+  }
+
+  get isPair() {
+    return this.idl.idlType.length === 2;
+  }
+
+  get isAsync() {
+    return true;
+  }
+
+  get hasReturnSteps() {
+    return Boolean(utils.getExtAttr(this.idl.extAttrs, "WebIDL2JSHasReturnSteps"));
+  }
+
+  generateFunction(key, kind, requires) {
+    const conv = generateAsyncIteratorArgConversions(
+      this.ctx, this.idl, this.interface, `Failed to execute '${key}' on '${this.interface.name}': `);
+    requires.merge(conv.requires);
+
+    this.interface.addMethod(this.interface.defaultWhence, key, [], `
+      if (!exports.is(this)) {
+        throw new TypeError("'${key}' called on an object that is not a valid instance of ${this.interface.name}.");
+      }
+
+      ${conv.body}
+
+      const asyncIterator = exports.createDefaultAsyncIterator(this, "${kind}");
+      if (this[implSymbol][utils.asyncIteratorInit]) {
+        this[implSymbol][utils.asyncIteratorInit](asyncIterator, args);
+      }
+      return asyncIterator;
+    `);
+  }
+
+  generate() {
+    const whence = this.interface.defaultWhence;
+    const requires = new utils.RequiresMap(this.ctx);
+
+    // https://heycam.github.io/webidl/#define-the-asynchronous-iteration-methods
+
+    if (this.isPair) {
+      this.generateFunction("keys", "key", requires);
+      this.generateFunction("values", "value", requires);
+      this.generateFunction("entries", "key+value", requires);
+      this.interface.addProperty(whence, Symbol.asyncIterator, `${this.interface.name}.prototype.entries`);
+    } else {
+      this.generateFunction("values", "value", requires);
+      this.interface.addProperty(whence, Symbol.asyncIterator, `${this.interface.name}.prototype.values`);
+    }
+
+    return { requires };
+  }
+}
+
+module.exports = Iterable;

lib/constructs/interface.js

@@ -4,6 +4,7 @@ const utils = require("../utils");
 const Attribute = require("./attribute");
 const Constant = require("./constant");
 const Iterable = require("./iterable");
+const AsyncIterable = require("./async-iterable");
 const Operation = require("./operation");
 const Types = require("../types");
 const Overloads = require("../overloads");
@@ -306,7 +307,9 @@ class Interface {
           if (this.iterable) {
             throw new Error(`Interface ${this.name} has more than one iterable declaration`);
           }
-          this.iterable = new Iterable(this.ctx, this, member);
+          this.iterable = member.async ?
+            new AsyncIterable(this.ctx, this, member) :
+            new Iterable(this.ctx, this, member);
           break;
         default:
           if (!this.ctx.options.suppressErrors) {
@@ -368,13 +371,15 @@ class Interface {
     // https://heycam.github.io/webidl/#dfn-reserved-identifier
     const forbiddenMembers = new Set(["constructor", "toString"]);
     if (this.iterable) {
-      if (this.iterable.isValue) {
-        if (!this.supportsIndexedProperties) {
-          throw new Error(`A value iterator cannot be declared on ${this.name} which does not support indexed ` +
-                          "properties");
+      if (!this.iterable.isAsync) {
+        if (this.iterable.isValue) {
+          if (!this.supportsIndexedProperties) {
+            throw new Error(`A value iterator cannot be declared on ${this.name} which does not support indexed ` +
+                            "properties");
+          }
+        } else if (this.iterable.isPair && this.supportsIndexedProperties) {
+          throw new Error(`A pair iterator cannot be declared on ${this.name} which supports indexed properties`);
         }
-      } else if (this.iterable.isPair && this.supportsIndexedProperties) {
-        throw new Error(`A pair iterator cannot be declared on ${this.name} which supports indexed properties`);
       }
       for (const n of ["entries", "forEach", "keys", "values"]) {
         forbiddenMembers.add(n);
@@ -416,7 +421,83 @@ class Interface {
   }
 
   generateIterator() {
-    if (this.iterable && this.iterable.isPair) {
+    if (!this.iterable) {
+      return;
+    }
+
+    if (this.iterable.isAsync) {
+      this.str += `
+        const AsyncIteratorPrototype = Object.create(utils.AsyncIteratorPrototype, {
+          [Symbol.toStringTag]: {
+            value: "${this.name} AsyncIterator",
+            configurable: true
+          }
+        });
+
+        Object.assign(AsyncIteratorPrototype, {
+          next() {
+            const internal = this[utils.iterInternalSymbol];
+            if (!internal) {
+              return Promise.reject(new TypeError("next() called on an object that is not an async iterator prototype object"));
+            }
+
+            const nextSteps = () => {
+              if (internal.isFinished) {
+                return Promise.resolve({ value: undefined, done: true });
+              }
+
+              const nextPromise = internal.target[implSymbol][utils.asyncIteratorNext](this);
+              return nextPromise.then(next => {
+                internal.ongoingPromise = null;
+                if (next === utils.asyncIteratorEOI) {
+                  internal.isFinished = true;
+                  return { value: undefined, done: true };
+                }`;
+      if (this.iterable.isPair) {
+        this.str += `
+                return utils.iteratorResult(next.map(utils.tryWrapperForImpl), kind);
+        `;
+      } else {
+        this.str += `
+                return { value: utils.tryWrapperForImpl(next), done: false };
+        `;
+      }
+      this.str += `
+              });
+            };
+
+            internal.ongoingPromise = internal.ongoingPromise ? internal.ongoingPromise.then(nextSteps) : nextSteps();
+            return internal.ongoingPromise;
+          }
+      `;
+
+      if (this.iterable.hasReturnSteps) {
+        this.str += `
+          ,
+          return(value) {
+            const internal = this[utils.iterInternalSymbol];
+            if (!internal) {
+              return Promise.reject(new TypeError("return() called on an object that is not an async iterator prototype object"));
+            }
+
+            if (internal.ongoingPromise) {
+              return Promise.reject(new TypeError("return() cannot be called while an ongoing call to next() has not settled"));
+            }
+
+            if (internal.isFinished) {
+              return Promise.reject(new TypeError("return() cannot be called after async iteration is already finished"));
+            }
+            internal.isFinished = true;
+
+            const returnPromise = internal.target[implSymbol][utils.asyncIteratorReturn](this, value);
+            return returnPromise.then(() => ({ value, done: true }));
+          }
+        `;
+      }
+      this.str += `
+        });
+      `;
+    } else if (this.iterable.isPair) {
       this.str += `
         const IteratorPrototype = Object.create(utils.IteratorPrototype, {
           next: {
@@ -431,21 +512,7 @@ class Interface {
 
               const pair = values[index];
               internal.index = index + 1;
-              const [key, value] = pair.map(utils.tryWrapperForImpl);
-
-              let result;
-              switch (kind) {
-                case "key":
-                  result = key;
-                  break;
-                case "value":
-                  result = value;
-                  break;
-                case "key+value":
-                  result = [key, value];
-                  break;
-              }
-              return { value: result, done: false };
+              return utils.iteratorResult(pair.map(utils.tryWrapperForImpl), kind);
             },
             writable: true,
             enumerable: true,
@@ -532,17 +599,30 @@ class Interface {
       };
     `;
 
-    if (this.iterable && this.iterable.isPair) {
-      this.str += `
-        exports.createDefaultIterator = (target, kind) => {
-          const iterator = Object.create(IteratorPrototype);
-          Object.defineProperty(iterator, utils.iterInternalSymbol, {
-            value: { target, kind, index: 0 },
-            configurable: true
-          });
-          return iterator;
-        };
-      `;
+    if (this.iterable) {
+      if (this.iterable.isAsync) {
+        this.str += `
+          exports.createDefaultAsyncIterator = (target, kind) => {
+            const iterator = Object.create(AsyncIteratorPrototype);
+            Object.defineProperty(iterator, utils.iterInternalSymbol, {
+              value: { target, kind, ongoingPromise: null, isFinished: false },
+              configurable: true
+            });
+            return iterator;
+          };
+        `;
+      } else if (this.iterable.isPair) {
+        this.str += `
+          exports.createDefaultIterator = (target, kind) => {
+            const iterator = Object.create(IteratorPrototype);
+            Object.defineProperty(iterator, utils.iterInternalSymbol, {
+              value: { target, kind, index: 0 },
+              configurable: true
+            });
+            return iterator;
+          };
+        `;
+      }
     }
   }
 

lib/constructs/iterable.js

@@ -18,6 +18,10 @@ class Iterable {
     return this.idl.idlType.length === 2;
   }
 
+  get isAsync() {
+    return false;
+  }
+
   generateFunction(key, kind) {
     this.interface.addMethod(this.interface.defaultWhence, key, [], `
       if (!exports.is(this)) {

lib/output/utils.js

@@ -45,6 +45,7 @@ function tryImplForWrapper(wrapper) {
 
 const iterInternalSymbol = Symbol("internal");
 const IteratorPrototype = Object.getPrototypeOf(Object.getPrototypeOf([][Symbol.iterator]()));
+const AsyncIteratorPrototype = Object.getPrototypeOf(Object.getPrototypeOf(async function* () {}).prototype);
 
 function isArrayIndexPropName(P) {
   if (typeof P !== "string") {
@@ -72,6 +73,22 @@ function isArrayBuffer(value) {
   }
 }
 
+function iteratorResult([key, value], kind) {
+  let result;
+  switch (kind) {
+    case "key":
+      result = key;
+      break;
+    case "value":
+      result = value;
+      break;
+    case "key+value":
+      result = [key, value];
+      break;
+  }
+  return { value: result, done: false };
+}
+
 const supportsPropertyIndex = Symbol("supports property index");
 const supportedPropertyIndices = Symbol("supported property indices");
 const supportsPropertyName = Symbol("supports property name");
@@ -84,6 +101,11 @@ const namedSetNew = Symbol("named property set new");
 const namedSetExisting = Symbol("named property set existing");
 const namedDelete = Symbol("named property delete");
 
+const asyncIteratorNext = Symbol("async iterator get the next iteration result");
+const asyncIteratorReturn = Symbol("async iterator return steps");
+const asyncIteratorInit = Symbol("async iterator initialization steps");
+const asyncIteratorEOI = Symbol("async iterator end of iteration");
+
 module.exports = exports = {
   isObject,
   hasOwn,
@@ -97,6 +119,7 @@ module.exports = exports = {
   tryImplForWrapper,
   iterInternalSymbol,
   IteratorPrototype,
+  AsyncIteratorPrototype,
   isArrayBuffer,
   isArrayIndexPropName,
   supportsPropertyIndex,
@@ -109,5 +132,10 @@ module.exports = exports = {
   namedGet,
   namedSetNew,
   namedSetExisting,
-  namedDelete
+  namedDelete,
+  asyncIteratorNext,
+  asyncIteratorReturn,
+  asyncIteratorInit,
+  asyncIteratorEOI,
+  iteratorResult
 };