starting vapor calls .wait on an EventLoop

[weissi]

In SwiftNIO we improved the wait-on-EventLoop detection to now catch many more of those issues. In fact it hits one in Vapor itself

Steps to reproduce

git clone https://github.com/vapor/vapor.git
git clone https://github.com/apple/swift-nio.git
cd vapor && swift package edit --path ../swift-nio swift-nio
swift run BoilerplateRun

Expected behavior

Running default command: .build/x86_64-apple-macosx10.10/debug/BoilerplateRun serve
Server starting on http://127.0.0.1:8080

Actual behavior

Running default command: .build/x86_64-apple-macosx10.10/debug/BoilerplateRun serve
Server starting on http://127.0.0.1:8080
Precondition failed: BUG DETECTED: wait() must not be called when on an EventLoop.
Calling wait() on any EventLoop can lead to
- deadlocks
- stalling processing of other connections (Channels) that are handled on the EventLoop that wait was called on

Further information:
- current eventLoop: Optional(SelectableEventLoop { selector = Selector { descriptor = 5 }, scheduledTasks = PriorityQueue(count: 0): [ScheduledTask(readyTime: 140055643366487), ScheduledTask(readyTime: 140055643395267), ScheduledTask(readyTime: 140055643406321), ScheduledTask(readyTime: 140055643421808)] })
- event loop associated to future: SelectableEventLoop { selector = Selector { descriptor = 8 }, scheduledTasks = PriorityQueue(count: 0): [] }: file /Users/johannes/extsrc/vapor/Sources/Vapor/Server/NIOServer.swift, line 61
Illegal instruction: 4

Environment

• today's Vapor master (4e53675)
• today's SwiftNIO master (0ce3cf5b79215a23cd47318cd00f62926a8a0aaa)

Analysis

Essentially the problem is that NIOServer.start is called from an EventLoop and itself calls wait on a future. This loop (and the following code) needs to be rewritten to be asynchronous

            for _ in 0..<config.workerCount {
                // initialize each event loop
                let eventLoop = group.next()
                let subContainer = container.subContainer(on: eventLoop)
                let responder = try subContainer.make(Responder.self)
                // perform cache set on the event loop
                try eventLoop.submit {
                    containerCache.currentValue = ThreadContainer(container: subContainer)
                    responderCache.currentValue = ThreadResponder(responder: responder)
                }.wait()  /// <<< problem here

Crash trace

Thread 4:
0   libswiftCore.dylib               0x000000010d603c60 _swift_runtime_on_report
1   libswiftCore.dylib               0x000000010d6552bb _swift_stdlib_reportFatalErrorInFile
2   libswiftCore.dylib               0x000000010d35db71 closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never
3   libswiftCore.dylib               0x000000010d5e3982 partial apply forwarder for closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never
4   libswiftCore.dylib               0x000000010d5e5c49 closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Neverpartial apply forwarder with unmangled suffix ".832"
5   libswiftCore.dylib               0x000000010d35d2ca function signature specialization <Arg[1] = [Closure Propagated : reabstraction thunk helper from @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> () to @escaping @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> (@out ()), Argument Types : [@callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> ()]> of generic specialization <()> of Swift.StaticString.withUTF8Buffer<A>((Swift.UnsafeBufferPointer<Swift.UInt8>) -> A) -> A
6   libswiftCore.dylib               0x000000010d5e3893 partial apply forwarder for closure #1 (Swift.UnsafeBufferPointer<Swift.UInt8>) -> () in Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never
7   libswiftCore.dylib               0x000000010d35d2ca function signature specialization <Arg[1] = [Closure Propagated : reabstraction thunk helper from @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> () to @escaping @callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> (@out ()), Argument Types : [@callee_guaranteed (@unowned Swift.UnsafeBufferPointer<Swift.UInt8>) -> ()]> of generic specialization <()> of Swift.StaticString.withUTF8Buffer<A>((Swift.UnsafeBufferPointer<Swift.UInt8>) -> A) -> A
8   libswiftCore.dylib               0x000000010d5273f8 function signature specialization <Arg[1] = Exploded> of Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never
9   libswiftCore.dylib               0x000000010d35cb59 Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never
10  BoilerplateRun                   0x000000010c76ad81 NIO.EventLoopFuture.wait(file: Swift.StaticString, line: Swift.UInt) throws -> A EventLoopFuture.swift:812:13
11  BoilerplateRun                   0x000000010c9975ac Vapor.NIOServer.start(hostname: Swift.Optional<Swift.String>, port: Swift.Optional<Swift.Int>) -> NIO.EventLoopFuture<()> NIOServer.swift:61:19
12  BoilerplateRun                   0x000000010c9994d7 protocol witness for Vapor.Server.start(hostname: Swift.Optional<Swift.String>, port: Swift.Optional<Swift.Int>) -> NIO.EventLoopFuture<()> in conformance Vapor.NIOServer : Vapor.Server in Vapor <compiler-generated>:0
13  BoilerplateRun                   0x000000010c956143 Vapor.ServeCommand.run(using: Command.CommandContext) throws -> NIO.EventLoopFuture<()> ServeCommand.swift:39:23
14  BoilerplateRun                   0x000000010c956604 protocol witness for Command.CommandRunnable.run(using: Command.CommandContext) throws -> NIO.EventLoopFuture<()> in conformance Vapor.ServeCommand : Command.CommandRunnable in Vapor <compiler-generated>:0
15  BoilerplateRun                   0x000000010c5bcb17 (extension in Command):Console.Console.(_run in _241066C70BE048BADDF40F0EFCBC6045)(_: Command.CommandRunnable, input: inout Command.CommandInput, on: Service.Container) throws -> NIO.EventLoopFuture<()> Console+Run.swift:84:33
16  BoilerplateRun                   0x000000010c5ba61b (extension in Command):Console.Console.run(_: Command.CommandRunnable, input: inout Command.CommandInput, on: Service.Container) -> NIO.EventLoopFuture<()> Console+Run.swift:17:24
17  BoilerplateRun                   0x000000010c5bc63b (extension in Command):Console.Console.(_run in _241066C70BE048BADDF40F0EFCBC6045)(_: Command.CommandRunnable, input: inout Command.CommandInput, on: Service.Container) throws -> NIO.EventLoopFuture<()> Console+Run.swift:78:28
18  BoilerplateRun                   0x000000010c5ba61b (extension in Command):Console.Console.run(_: Command.CommandRunnable, input: inout Command.CommandInput, on: Service.Container) -> NIO.EventLoopFuture<()> Console+Run.swift:17:24
19  BoilerplateRun                   0x000000010c94b3ec closure #2 () throws -> NIO.EventLoopFuture<()> in Vapor.Application.asyncRun() -> NIO.EventLoopFuture<()> Application.swift:99:28
20  BoilerplateRun                   0x000000010c94cbb6 partial apply forwarder for closure #2 () throws -> NIO.EventLoopFuture<()> in Vapor.Application.asyncRun() -> NIO.EventLoopFuture<()> <compiler-generated>:0
21  BoilerplateRun                   0x000000010c591306 reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>, @error @owned Swift.Error) to @escaping @callee_guaranteed (@in_guaranteed ()) -> (@owned NIO.EventLoopFuture<()>, @error @owned Swift.Error) <compiler-generated>:0
22  BoilerplateRun                   0x000000010c94cc1b reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.EventLoopFuture<()>, @error @owned Swift.Error) to @escaping @callee_guaranteed (@in_guaranteed ()) -> (@owned NIO.EventLoopFuture<()>, @error @owned Swift.Error)partial apply forwarder with unmangled suffix ".7" <compiler-generated>:0
23  BoilerplateRun                   0x000000010c588f4e closure #1 (A) -> () in (extension in Async):NIO.EventLoopFuture.flatMap<A>(to: A1.Type, _: (A) throws -> NIO.EventLoopFuture<A1>) -> NIO.EventLoopFuture<A1> Future+Map.swift:50:34
24  BoilerplateRun                   0x000000010c5890a9 partial apply forwarder for closure #1 (A) -> () in (extension in Async):NIO.EventLoopFuture.flatMap<A>(to: A1.Type, _: (A) throws -> NIO.EventLoopFuture<A1>) -> NIO.EventLoopFuture<A1> <compiler-generated>:0
25  BoilerplateRun                   0x000000010c76811e closure #1 () -> NIO.(CallbackList in _1CAEE0056AB83634CE1A29DE57BC300C) in NIO.EventLoopFuture.whenSuccess((A) -> ()) -> () EventLoopFuture.swift:630:17
26  BoilerplateRun                   0x000000010c76a525 partial apply forwarder for closure #1 () -> NIO.(CallbackList in _1CAEE0056AB83634CE1A29DE57BC300C) in NIO.EventLoopFuture.whenSuccess((A) -> ()) -> () <compiler-generated>:0
27  BoilerplateRun                   0x000000010c765e05 NIO.EventLoopFuture.(_addCallback in _1CAEE0056AB83634CE1A29DE57BC300C)(() -> NIO.(CallbackList in _1CAEE0056AB83634CE1A29DE57BC300C)) -> NIO.(CallbackList in _1CAEE0056AB83634CE1A29DE57BC300C) EventLoopFuture.swift:596:16
28  BoilerplateRun                   0x000000010c767b5c closure #1 () -> () in NIO.EventLoopFuture.(_whenComplete in _1CAEE0056AB83634CE1A29DE57BC300C)(() -> NIO.(CallbackList in _1CAEE0056AB83634CE1A29DE57BC300C)) -> () EventLoopFuture.swift:605:22
29  BoilerplateRun                   0x000000010c76fe75 partial apply forwarder for closure #1 () -> () in NIO.EventLoopFuture.(_whenComplete in _1CAEE0056AB83634CE1A29DE57BC300C)(() -> NIO.(CallbackList in _1CAEE0056AB83634CE1A29DE57BC300C)) -> () <compiler-generated>:0
30  BoilerplateRun                   0x000000010c6fbf5c reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_guaranteed (@in_guaranteed ()) -> (@out ()) <compiler-generated>:0
31  BoilerplateRun                   0x000000010c760891 reabstraction thunk helper from @escaping @callee_guaranteed () -> () to @escaping @callee_guaranteed (@in_guaranteed ()) -> (@out ())partial apply forwarder with unmangled suffix ".175" <compiler-generated>:0
32  BoilerplateRun                   0x000000010c7581f7 reabstraction thunk helper from @escaping @callee_guaranteed (@in_guaranteed ()) -> (@out ()) to @escaping @callee_guaranteed () -> () <compiler-generated>:0
33  BoilerplateRun                   0x000000010c758321 partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed (@in_guaranteed ()) -> (@out ()) to @escaping @callee_guaranteed () -> () <compiler-generated>:0
34  BoilerplateRun                   0x000000010c758217 closure #3 () -> () in NIO.SelectableEventLoop.run() throws -> () EventLoop.swift:702:25
35  BoilerplateRun                   0x000000010c758271 partial apply forwarder for closure #3 () -> () in NIO.SelectableEventLoop.run() throws -> () <compiler-generated>:0
36  BoilerplateRun                   0x000000010c6d9b46 reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @escaping @callee_guaranteed () -> (@out (), @error @owned Swift.Error) <compiler-generated>:0
37  BoilerplateRun                   0x000000010c7582bb reabstraction thunk helper from @callee_guaranteed () -> (@error @owned Swift.Error) to @escaping @callee_guaranteed () -> (@out (), @error @owned Swift.Error)partial apply forwarder with unmangled suffix ".50" <compiler-generated>:0
38  BoilerplateRun                   0x000000010c752be2 closure #1 () throws -> A in NIO.(withAutoReleasePool in _D5D78C61B22284700B9BD1ACFBC25157)<A>(() throws -> A) throws -> A EventLoop.swift:427:13
39  BoilerplateRun                   0x000000010c7608f6 partial apply forwarder for closure #1 () throws -> A in NIO.(withAutoReleasePool in _D5D78C61B22284700B9BD1ACFBC25157)<A>(() throws -> A) throws -> A <compiler-generated>:0
40  libswiftObjectiveC.dylib         0x000000010dd31f2f ObjectiveC.autoreleasepool<A>(invoking: () throws -> A) throws -> A
41  BoilerplateRun                   0x000000010c752b49 NIO.(withAutoReleasePool in _D5D78C61B22284700B9BD1ACFBC25157)<A>(() throws -> A) throws -> A EventLoop.swift:426:16
42  BoilerplateRun                   0x000000010c756f65 NIO.SelectableEventLoop.run() throws -> () EventLoop.swift:701:21
43  BoilerplateRun                   0x000000010c75a9be closure #1 (NIO.Thread) -> () in static NIO.MultiThreadedEventLoopGroup.(setupThreadAndEventLoop in _D5D78C61B22284700B9BD1ACFBC25157)(name: Swift.String, initializer: (NIO.Thread) -> ()) -> NIO.SelectableEventLoop EventLoop.swift:856:23
44  BoilerplateRun                   0x000000010c7603ba partial apply forwarder for closure #1 (NIO.Thread) -> () in static NIO.MultiThreadedEventLoopGroup.(setupThreadAndEventLoop in _D5D78C61B22284700B9BD1ACFBC25157)(name: Swift.String, initializer: (NIO.Thread) -> ()) -> NIO.SelectableEventLoop <compiler-generated>:0
45  BoilerplateRun                   0x000000010c75af5f reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed NIO.Thread) -> () to @escaping @callee_guaranteed (@in_guaranteed NIO.Thread) -> (@out ()) <compiler-generated>:0
46  BoilerplateRun                   0x000000010c7ec431 partial apply forwarder for reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed NIO.Thread) -> () to @escaping @callee_guaranteed (@in_guaranteed NIO.Thread) -> (@out ()) <compiler-generated>:0
47  BoilerplateRun                   0x000000010c7ecaa0 closure #1 (Swift.UnsafeMutableRawPointer) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static NIO.Thread.spawnAndRun(name: Swift.Optional<Swift.String>, body: (NIO.Thread) -> ()) -> () Thread.swift:105:13
48  BoilerplateRun                   0x000000010c7ecb09 @objc closure #1 (Swift.UnsafeMutableRawPointer) -> Swift.Optional<Swift.UnsafeMutableRawPointer> in static NIO.Thread.spawnAndRun(name: Swift.Optional<Swift.String>, body: (NIO.Thread) -> ()) -> () <compiler-generated>:0
49  libsystem_pthread.dylib          0x00007fff69553881 _pthread_body
50  libsystem_pthread.dylib          0x00007fff695537f3 _pthread_start
51  libsystem_pthread.dylib          0x00007fff695533bd thread_start

[weissi]

can confirm it's fixed. Thanks for the fast turnaround @tanner0101