Adopt typed throws in AsyncIteratorProtocol and AsyncSequence #70635

DougGregor · 2023-12-27T07:55:42Z

Introduce a new associated type Failure into the two protocols involved
in async sequences, which represents the type thrown when the sequence
fails. Introduce a defaulted _next(_:) operations that is isolated to the
given actor (or nil) and either throws Failure or produces the next
element of the sequence. Provide a default implementation of next(_:)
in terms of next() that force-casts the thrown error to the Failure type,
and a default implementation in terms of next() in terms of next(_:) that
lets new async iterators only define the new entrypoint.

Drop @rethrows from AsyncSequence and AsyncIteratorProtocol;
this unofficial feature is on its way out.

Introduce special associated type inference logic for the Failure
type of an AsyncIteratorProtocol conformance when there is no
specific next(_:) witness. This inference logic looks at the
witness for next():

If next() throws nothing, Failure is inferred to Never.
If next() throws, Failure is inferred to any Error.
If next() rethrows, Failure is inferred to T.Failure, where
T is the first type parameter with a conformance to either
AsyncSequence or AsyncIteratorProtocol.

The default implementation and the inference rule, together, allow
existing async sequences to continue working as before. Switch the
contract between the compiler and library for the async for loop to
use next(_:) rather than next() when targeting a new-enough
deployment target.

Give AsyncSequence and AsyncIteratorProtocol primary associated
types for the element and failure types, which will allow them to be
used more generally with existential and opaque types, i.e.,
any AsyncSequence<Image, NetworkError>.

DougGregor · 2023-12-27T07:55:49Z

@swift-ci please smoke test

DougGregor · 2023-12-27T07:56:38Z

lib/Sema/TypeCheckProtocolInference.cpp

+  // If there is a generic parameter named Failure, don't try to use next()
+  // to infer Failure.
+  if (auto genericSig = dc->getGenericSignatureOfContext()) {
+    for (auto gp : genericSig.getGenericParams()) {
+      // Packs cannot witness associated type requirements.
+      if (gp->isParameterPack())
+        continue;
+
+      if (gp->getName() == assocType->getName())
+        return llvm::None;
+    }
+  }


This is very sketchy and I want to try eliminating it.

DougGregor · 2024-01-01T15:25:33Z

@swift-ci please smoke test

DougGregor · 2024-01-02T03:23:08Z

@swift-ci please smoke test

DougGregor · 2024-01-02T15:00:55Z

@swift-ci please smoke test

DougGregor · 2024-01-02T15:01:01Z

@swift-ci please build toolchain

DougGregor · 2024-01-02T16:45:12Z

@swift-ci please smoke test

DougGregor · 2024-01-02T16:45:17Z

@swift-ci please build toolchain

DougGregor · 2024-01-05T01:22:38Z

@swift-ci please smoke test

DougGregor · 2024-01-05T06:03:59Z

@swift-ci please smoke test

DougGregor · 2024-01-05T16:44:38Z

<unknown>:0: error: 'nextElement()' is only available in macOS 9999 or newer
<unknown>:0: error: 'nextElement()' is only available in macOS 9999 or newer

DougGregor · 2024-01-06T08:19:00Z

@swift-ci please smoke test

DougGregor · 2024-01-08T06:17:41Z

@swift-ci please smoke test

DougGregor · 2024-01-08T16:14:14Z

@swift-ci please smoke test

…fects Our decision procedure for what type of error is thrown during iteration of an async sequence was effectively duplicated between the checking for the next/nextElement call (which is synthesized) and separate logic for the async for..in loop. The latter is strictly redundant, so switch to relying only on the former. This is staging for customizing the next/nextElement call further.

… on availability This allows us to not break backward deployment

…cking Some values (such as `undef`) might not be associated with a function.

This isn't done very well, and is expected to be replaced, but it allows us to avoid bogus Sendable diagnostics for iteration over async sequences.

Use an optional isolated parameter to this new `next(_:)` overload to keep it on the same actor as the caller, and pass `#isolation` when desugaring the async for..in loop. This keeps async iteration loops on the same actor, allowing non-Sendable values to be used with many async sequences.

…latMap` Allow `AsyncSequence.flatMap` to be defined with "incorrect" availability, meaning that the function can refer to the `Failure` associated type in its where clause even though the function is back-deployed to before the `Failure` associated type was introduced. We believe this is safe, and that this notion can be generalized to any use of an associated type in a same-type constraint of a function (yes, it sounds weird), but for now introduce a narrower hack to see how things work in practice and whether it addresses all of the source-compatibility concerns we've uncovered.

DougGregor · 2024-01-26T00:05:26Z

@swift-ci please smoke test

DougGregor · 2024-01-26T00:05:32Z

@swift-ci please test source compatibility

DougGregor · 2024-01-26T00:05:37Z

@swift-ci please build toolchain

DougGregor · 2024-01-26T00:23:58Z

@swift-ci please build toolchain Windows

DougGregor · 2024-01-26T00:32:55Z

@swift-ci please smoke test Linux

DougGregor · 2024-01-26T00:42:02Z

@swift-ci please smoke test Linux

DougGregor · 2024-01-26T02:57:46Z

@swift-ci please smoke test Linux

DougGregor · 2024-01-26T03:06:57Z

@swift-ci please test windows

ktoso

Read through the whole thing so far, not much to add but looking good, heroic effort Doug :)

ktoso · 2024-01-26T11:54:11Z

lib/Sema/CSGen.cpp

+    ArgumentList *nextArgs;
+    if (nextFn && nextFn->getParameters()->size() == 1) {
+      auto isolationArg =
+        new (ctx) CurrentContextIsolationExpr(stmt->getForLoc(), Type());


Oh I like this name for it 👍

ktoso · 2024-01-26T12:18:36Z

stdlib/public/Concurrency/AsyncIteratorProtocol.swift

@@ -86,13 +86,40 @@ import Swift
 /// checking for cancellation as described above, or in `deinit` if it's
 /// a reference type.
 @available(SwiftStdlib 5.1, *)
-@rethrows


goodbye 👋 😊

ktoso · 2024-01-26T12:21:11Z

stdlib/public/Concurrency/AsyncIteratorProtocol.swift

+  /// required to maintain backward compatibility with existing async iterators.
+  @available(SwiftStdlib 5.11, *)
+  @inlinable
+  public mutating func next(_ actor: isolated (any Actor)?) async throws(Failure) -> Element? {


are we going to do the "conform to just one of them" dance the same as with executors, or different trickery?

We're relying on the deprecation of the old one to help here, at least from Swift 6 on

Okey, so not the the "implement just one of them" -- might be for the best, that was causing a lot of pain in executor rollout, probably not worth it.

DougGregor · 2024-01-26T20:27:39Z

@swift-ci please test source compatibility

DougGregor · 2024-01-26T20:27:56Z

@swift-ci please build toolchain macOS

DougGregor · 2024-01-26T20:28:49Z

@swift-ci please build toolchain Windows

DougGregor · 2024-01-26T20:31:50Z

Lots of spurious failures, so rerunning a number of jobs. Yesterday was a rough day to be a CI system.

DougGregor · 2024-01-29T18:21:51Z

This is under review now: https://forums.swift.org/t/se-0421-generalize-effect-polymorphism-for-asyncsequence-and-asynciteratorprotocol/69662

DougGregor · 2024-01-29T19:51:22Z

I'm going to speculatively merge this to unblock more testing. We can back out the merge or change things depending on how the review goes.

DougGregor requested review from ktoso, kavon, AnthonyLatsis, hborla, slavapestov and xedin as code owners December 27, 2023 07:55

DougGregor commented Dec 27, 2023

View reviewed changes

DougGregor force-pushed the async-sequence-typed-throws branch from df5248d to c12c895 Compare January 1, 2024 15:24

DougGregor force-pushed the async-sequence-typed-throws branch from e8b574b to 635326e Compare January 6, 2024 07:02

DougGregor requested review from ahoppen, bnbarham, CodaFi, hamishknight and rintaro as code owners January 6, 2024 07:02

DougGregor force-pushed the async-sequence-typed-throws branch from 635326e to 3324fcd Compare January 6, 2024 07:18

DougGregor force-pushed the async-sequence-typed-throws branch from e3afbe6 to d7a8cba Compare January 8, 2024 06:11

DougGregor requested a review from tshortli as a code owner January 8, 2024 06:11

DougGregor added 12 commits January 25, 2024 16:04

Add availability for the Failure associated type of async sequences

3482ca0

Only substitute into the error type when there are substitutions

6bc3087

Fix an assertion to also consider typed throws

31c0ab7

Implement AsyncThrowingStream.nextElement() with typed throws

ec945bf

Choose between AsyncIteratorProtocol's next() and nextElement() based…

dc85ae3

… on availability This allows us to not break backward deployment

Make sure we always have a SILFunction for region-based isolation che…

e96dcac

…cking Some values (such as `undef`) might not be associated with a function.

Partial actor isolation checking for #isolation

3558237

This isn't done very well, and is expected to be replaced, but it allows us to avoid bogus Sendable diagnostics for iteration over async sequences.

Remove workaround for do..catch not handling concrete error types

9bacd3e

Update SILGen test to account for addition of @isolated

a3ae340

DougGregor force-pushed the async-sequence-typed-throws branch from 557409d to 4c990dc Compare January 26, 2024 00:05

ktoso approved these changes Jan 26, 2024

View reviewed changes

DougGregor merged commit 0cc5297 into apple:main Jan 29, 2024
8 checks passed

DougGregor deleted the async-sequence-typed-throws branch January 29, 2024 19:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adopt typed throws in AsyncIteratorProtocol and AsyncSequence #70635

Adopt typed throws in AsyncIteratorProtocol and AsyncSequence #70635

DougGregor commented Dec 27, 2023 •

edited

DougGregor commented Dec 27, 2023

DougGregor Dec 27, 2023

DougGregor commented Jan 1, 2024

DougGregor commented Jan 2, 2024

DougGregor commented Jan 2, 2024

DougGregor commented Jan 2, 2024

DougGregor commented Jan 2, 2024

DougGregor commented Jan 2, 2024

DougGregor commented Jan 5, 2024

DougGregor commented Jan 5, 2024

DougGregor commented Jan 5, 2024

DougGregor commented Jan 6, 2024

DougGregor commented Jan 8, 2024

DougGregor commented Jan 8, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

ktoso left a comment

ktoso Jan 26, 2024

ktoso Jan 26, 2024

ktoso Jan 26, 2024

DougGregor Jan 27, 2024

ktoso Jan 27, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 29, 2024

DougGregor commented Jan 29, 2024

Adopt typed throws in AsyncIteratorProtocol and AsyncSequence #70635

Adopt typed throws in AsyncIteratorProtocol and AsyncSequence #70635

Conversation

DougGregor commented Dec 27, 2023 • edited

DougGregor commented Dec 27, 2023

DougGregor Dec 27, 2023

Choose a reason for hiding this comment

DougGregor commented Jan 1, 2024

DougGregor commented Jan 2, 2024

DougGregor commented Jan 2, 2024

DougGregor commented Jan 2, 2024

DougGregor commented Jan 2, 2024

DougGregor commented Jan 2, 2024

DougGregor commented Jan 5, 2024

DougGregor commented Jan 5, 2024

DougGregor commented Jan 5, 2024

DougGregor commented Jan 6, 2024

DougGregor commented Jan 8, 2024

DougGregor commented Jan 8, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

ktoso left a comment

Choose a reason for hiding this comment

ktoso Jan 26, 2024

Choose a reason for hiding this comment

ktoso Jan 26, 2024

Choose a reason for hiding this comment

ktoso Jan 26, 2024

Choose a reason for hiding this comment

DougGregor Jan 27, 2024

Choose a reason for hiding this comment

ktoso Jan 27, 2024

Choose a reason for hiding this comment

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 26, 2024

DougGregor commented Jan 29, 2024

DougGregor commented Jan 29, 2024

DougGregor commented Dec 27, 2023 •

edited