[SR-15108] Various crashes (EXC_BAD_ACCESS) and double free using async tasks.

[swift-ci]

Previous ID	SR-15108
Radar	rdar://problem/82306884
Original Reporter	beaumont (JIRA User)
Type	Bug

Attachment: Download

Environment

Here is a repro branch of Swift NIO: https://github.com/simonjbeaumont/swift-nio/tree/async-double-free-repro

The branch has commented out all other tests from the Package.swift and has added a single test:

`_NIOConcurrencyTests.AsyncAwaitHelpersTests.testPromiseCompletedWithSuccessfulTaskInClassAsyncWithAwaitWithYield`

The structure of types in the test is a little arbitrary since it represents an analog of an issue I was hitting in grpc-swift.

I am able to reproduce the failure reliably using the latest Swift toolchain for Xcode, `swift-5.5-DEVELOPMENT-SNAPSHOT-2021-08-23-a`, and also on Linux using, Docker image `swiftlang/swift nightly-5.5 63768f6ffe90`

Additional Detail from JIRA

Votes	0
Component/s	Compiler
Labels	Bug, Concurrency
Assignee	@kavon
Priority	Medium

md5: d0ed450d90301f18856997ddba42e5e5

Issue Description:

I have created a branch of Swift NIO with a test case that runs a block of code one million times. It crashes after O(10k) iterations.

Some times it segfaults, sometimes I get output about double-free but the crash itself isn't in a deterministic place, nor is the output.

From the logging in the test it appears that, when it does crash, some code that should have only been executed once is executed twice.

Here is a repro branch of Swift NIO: https://github.com/simonjbeaumont/swift-nio/tree/async-double-free-repro

The branch has commented out all other tests from the Package.swift and has added a single test:

`_NIOConcurrencyTests.AsyncAwaitHelpersTests.testPromiseCompletedWithSuccessfulTaskInClassAsyncWithAwaitWithYield`

The structure of types in the test is a little arbitrary since it represents an analog of an issue I was hitting in grpc-swift.

I am able to reproduce the failure reliably using the latest Swift toolchain for Xcode, `swift-5.5-DEVELOPMENT-SNAPSHOT-2021-08-23-a`, and also on Linux using, Docker image `swiftlang/swift nightly-5.5 63768f6ffe90`

Here is an example output:

--- Starting test iteration 6916
on thread 3 in completionHandler(_:): was success with value: OK
on thread 5 in mockError(): calling handleError(_:) with error: HandlerError()
on thread 5 in handleError(_:): failing promise with error: HandlerError() 
on thread 5 in handleError(_:): canceling task 
on thread 5 in deinit: .
--- Starting test iteration 6917 
on thread 5 in mockError(): calling handleError(_:) with error: HandlerError() 
on thread 5 in handleError(_:): failing promise with error: HandlerError() 
on thread 5 in completionHandler(_:): was failure with error: HandlerError() 
on thread 4 in completionHandler(_:): was failure with error: HandlerError() 
on thread 5 in handleError(_:): failing promise with error: HandlerError() 
on thread 4 in handleError(_:): failing promise with error: HandlerError() 
on thread 5 in handleError(_:): canceling task 
on thread 4 in handleError(_:): canceling task 
on thread 5 in handleError(_:): canceling task 
xctest(83107,0x70000bc9a000) malloc: Double free of object 0x101427570 xctest(83107,0x70000bc9a000) malloc: *** set a breakpoint in malloc_error_break to debug

It shouldn't be possible to see `canceling task` printed more than twice.

[typesanitizer]

@swift-ci create

[kavon]

On Xcode 13.2 beta and macOS 12, I'm not seeing the double-free anymore (both on apple silicon and intel). We recently fixed some async stack corruption issues, so that might explain why that behavior changed. But, the test still doesn't pass successfully. I wanted to check in with you to see if this is still problematic.

When I run the tests on that branch, I first encounter an issue with the way you're obtaining the thread ID in the logging function:

After commenting that out in two places (using just "unknown" instead), I'm now seeing print-outs indicating that a task was cancelled twice:

Starting test iteration 95323
{{ on thread unknown in mockError(): calling handleError(_🙂 with error: HandlerError()}}
{{ on thread unknown in handleError(_🙂: canceling task}}
{{ on thread unknown in handleError(_🙂: failing promise with error: HandlerError()}}
{{ on thread unknown in completionHandler(_🙂: was failure with error: CancellationError()}}
{{ on thread unknown in handleError(_🙂: canceling task}}
{{ on thread unknown in handleError(_🙂: failing promise with error: CancellationError()}}
{{ on thread unknown in deinit: .}}

and then on that same loop iteration with the double cancellation, the test crashes when trying to deallocate an `EventLoopFuture` at the end of the loop's body. In the `EventLoopFuture.deinit`, the crash happens here:

I tried to follow the control-flow / events here, and from my reading, and I can imagine how the task could be cancelled twice: `handleError` is called in `AsyncAwaitHelpersTests.mockError`, which leads to the completion handler `AsyncAwaitHelpersTests.completionHandler` to run. But, that handler calls `handleError` in its failure case, which would lead to that print-out. After calling `self.promise.fail` again, the reentrance into `handleError` stops, since `EventLoopFuture._setValue` sets `_value` to prevent it from invoking the callbacks again.

But, this doesn't explain why I'm seeing a crash in `EventLoopFuture.deinit`. Do you have any ideas? I also don't understand why the cancellation is not printed out twice on other loop iterations. Does somebody overwrite the `task` to prevent that?

(sorry for the bad formatting above... I lost a fight with Jira's mark-up system and can't correct it!)