"Received signal 11" in app_analyze

[finestructure]

I finally caught a glimpse as to why analysis sometimes hangs:

2023-01-21T18:57:54.937913401Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] Analyzing (limit: 25) ... [component: analyze]
2023-01-21T18:57:54.983740764Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] Checkout directory: /checkouts [component: analyze]
2023-01-21T18:57:54.987633060Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] pulling https://github.com/yonaskolb/Stringly.git in /checkouts/github.com-yonaskolb-stringly [component: analyze]
2023-01-21T18:57:55.021965033Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] pulling https://github.com/dagronf/DSFActionBar.git in /checkouts/github.com-dagronf-dsfactionbar [component: analyze]
2023-01-21T18:57:55.029915226Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] pulling https://github.com/ordo-one/package-frostflake.git in /checkouts/github.com-ordo-one-package-frostflake [component: analyze]
2023-01-21T18:57:55.271462231Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] pulling https://github.com/chojnac/Dumpling.git in /checkouts/github.com-chojnac-dumpling [component: analyze]
2023-01-21T18:57:55.273393230Z app_analyze.1.t1dix7fv4zph@p3    | [ INFO ] pulling https://github.com/raymccrae/swift-jsonpatch.git in /checkouts/github.com-raymccrae-swift-jsonpatch [component: analyze]
2023-01-21T18:57:55.277729726Z app_analyze.1.t1dix7fv4zph@p3    | Received signal 11. Backtrace:
2023-01-21T18:58:04.984287787Z app_analyze.1.t1dix7fv4zph@p3    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection timeout. [component: server, database-id: psql]
2023-01-21T18:58:04.984746687Z app_analyze.1.t1dix7fv4zph@p3    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection timeout. [component: server, database-id: psql]

The backtrace itself isn't in the logs but hopefully the crash is reproducible with one of the packages in question.

[finestructure]

None of these packages individually are causing a signal 11 when running locally (macOS/arm64). It's going to be difficult to run this on Linux/x86 but maybe Linux/arm64 can reproduce it.

[finestructure]

Doesn't crash on Linux/arm64 either.

I have no way to test this on Linux/x86. The next best thing we can try is to make sure the backtrace shows up in the logs.

[finestructure]

It's not clear to me why the backtrace isn't captured by our logging. We're logging both stderr and stdout and yet there's no trace (😅) of the output beyond Received signal 11. Backtrace:.

[finestructure]

I'll close this for now, will reopen and add more details if it happens again.

[finestructure]

Another lock-up, different error and we got a partially symbolicated stack trace:

2023-01-27T15:37:57.557322522Z app_analyze.1.9jb9jx9vkshk@p2    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your con[130/1048]
meout. [component: server, database-id: psql]
2023-01-27T15:37:57.557341622Z app_analyze.1.9jb9jx9vkshk@p2    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection ti
meout. [component: server, database-id: psql]
2023-01-27T15:37:57.557377022Z app_analyze.1.9jb9jx9vkshk@p2    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection ti
meout. [component: server, database-id: psql]
2023-01-27T15:37:57.557428923Z app_analyze.1.9jb9jx9vkshk@p2    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection ti
meout. [component: server, database-id: psql]
2023-01-27T15:37:57.557445323Z app_analyze.1.9jb9jx9vkshk@p2    | [ ERROR ] Connection request timed out. This might indicate a connection deadlock in your application. If you're running long running requests, consider increasing your connection ti
meout. [component: server, database-id: psql]
2023-01-26T13:50:08.693743753Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78b36187, closure vapor/console-kit#1 (Swift.Int32) -> () in static Backtrace.Backtrace.install(signals: Swift.Array<Swift.Int32>) -> () at /build/.build/checkouts/swift-backtrace/Sources/Backt
race/Backtrace.swift:80
2023-01-26T13:50:08.693750253Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78b36187, @objc closure vapor/console-kit#1 (Swift.Int32) -> () in static Backtrace.Backtrace.install(signals: Swift.Array<Swift.Int32>) -> () at /build/<compiler-generated>:79
2023-01-26T13:50:08.693754653Z app_analyze.1.luku8u31ozjl@p2    | 0x7f4d69f8841f
2023-01-26T13:50:08.693758753Z app_analyze.1.luku8u31ozjl@p2    | 0x561f799f0571
2023-01-26T13:50:08.694152755Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79533bd0, closure vapor/console-kit#4 () throws -> Swift.String in (extension in ShellOut):Foundation.Process.(launchBash in _839723A297212BDF262C1834C3E29C1F)(with: Swift.String, outputHandle:
 Swift.Optional<Foundation.FileHandle>, errorHandle: Swift.Optional<Foundation.FileHandle>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:442
2023-01-26T13:50:08.694178555Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79533bd0, partial apply forwarder for closure vapor/console-kit#4 () throws -> Swift.String in (extension in ShellOut):Foundation.Process.(launchBash in _839723A297212BDF262C1834C3E29C1F)(with:
 Swift.String, outputHandle: Swift.Optional<Foundation.FileHandle>, errorHandle: Swift.Optional<Foundation.FileHandle>) throws -> Swift.String at /build/<compiler-generated>:0
2023-01-26T13:50:08.694185355Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7982a3eb
2023-01-26T13:50:08.694189055Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79829285
2023-01-26T13:50:08.694192655Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7982d0fb
2023-01-26T13:50:08.694196255Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79824068
2023-01-26T13:50:08.694202855Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79840cd9
2023-01-26T13:50:08.694206355Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7982d071
2023-01-26T13:50:08.694210455Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7982914b
2023-01-26T13:50:08.694214055Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79829594
2023-01-26T13:50:08.694217755Z app_analyze.1.luku8u31ozjl@p2    | 0x561f795318e0, (extension in ShellOut):Foundation.Process.(launchBash in _839723A297212BDF262C1834C3E29C1F)(with: Swift.String, outputHandle: Swift.Optional<Foundation.FileHandle>,
errorHandle: Swift.Optional<Foundation.FileHandle>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:441
2023-01-26T13:50:08.694237255Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79531d27, ShellOut.shellOut(to: Swift.String, arguments: Swift.Array<Swift.String>, at: Swift.String, process: Foundation.Process, outputHandle: Swift.Optional<Foundation.FileH
andle>, errorHandle: Swift.Optional<Foundation.FileHandle>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:40
2023-01-26T13:50:08.694242255Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79531d27, ShellOut.shellOut(to: ShellOut.ShellOutCommand, at: Swift.String, process: Foundation.Process, outputHandle: Swift.Optional<Foundation.FileHandle>, errorHandle: Swift
.Optional<Foundation.FileHandle>) throws -> Swift.String at /build/.build/checkouts/ShellOut/Sources/ShellOut.swift:106
2023-01-26T13:50:08.694566856Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7880f4d8, closure vapor/console-kit#1 (ShellOut.ShellOutCommand, Swift.String) throws -> Swift.String in variable initialization expression of static App.Shell.live : App.Shell at /build/Source
s/App/Core/AppEnvironment.swift:291
2023-01-26T13:50:08.694694157Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78830c1d, App.Shell.run(command: ShellOut.ShellOutCommand, at: Swift.String) throws -> Swift.String at Sources/App/Core/AppEnvironment.swift:283
2023-01-26T13:50:08.694709557Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78830c1d, function signature specialization <Arg[2] = Dead> of static App.Git.revisionInfo(_: App.Reference, at: Swift.String) throws -> App.Git.RevisionInfo at /build/Sources/
App/Core/Git.swift:70
2023-01-26T13:50:08.694851657Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78812395, static App.Git.revisionInfo(_: App.Reference, at: Swift.String) throws -> App.Git.RevisionInfo at /build/<compiler-generated>:0
2023-01-26T13:50:08.694873258Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78812395, implicit closure vapor/console-kit#12 (App.Reference, Swift.String) throws -> App.Git.RevisionInfo in implicit closure vapor/console-kit#11 (App.Git.Type) -> (App.Reference, Swift.String) throws -> Ap
p.Git.RevisionInfo in variable initialization expression of static App.Git.live : App.Git at /build/<compiler-generated>:271
2023-01-26T13:50:08.694901558Z app_analyze.1.luku8u31ozjl@p2    | 0x561f78812395, partial apply forwarder for implicit closure vapor/console-kit#12 (App.Reference, Swift.String) throws -> App.Git.RevisionInfo in implicit closure vapor/console-kit#11 (App.Git.Type) -> (App.Reference
, Swift.String) throws -> App.Git.RevisionInfo in variable initialization expression of static App.Git.live : App.Git at /build/<compiler-generated>:0
2023-01-26T13:50:08.696367663Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7878eefe, closure vapor/console-kit#3 (App.Reference) throws -> App.Version in static App.Analyze.getIncomingVersions(client: Vapor.Client, logger: Logging.Logger, package: App.Joined<App.Packa
ge, App.Repository>) async throws -> Swift.Array<App.Version> at /build/Sources/App/Commands/Analyze.swift:438
2023-01-26T13:50:08.696417564Z app_analyze.1.luku8u31ozjl@p2    | 0x561f7878eefe, function signature specialization <Arg[1] = Owned To Guaranteed, Arg[2] = Owned To Guaranteed> of function signature specialization <Arg[0] = [Closure Propagated : cl
osure vapor/console-kit#3 (App.Reference) throws -> App.Version in static App.Analyze.getIncomingVersions(client: Vapor.Client, logger: Logging.Logger, package: App.Joined<App.Package, App.Repository>) async throws -> Swift.Array<App.Version>, Argument Types : [Swi
ft.StringApp.Joined<App.Package, App.Repository>]> of generic specialization <Swift.Array<App.Reference>, App.Version> of (extension in Swift):Swift.Collection.map<A>((A.Element) throws -> A1) throws -> Swift.Array<A1> at /build/<compiler-generated
>:0
2023-01-26T13:50:08.696437964Z app_analyze.1.luku8u31ozjl@p2    | 0x561f787920c7, function signature specialization <Arg[0] = [Closure Propagated : closure vapor/console-kit#3 (App.Reference) throws -> App.Version in static App.Analyze.getIncomingVersions(client: V
apor.Client, logger: Logging.Logger, package: App.Joined<App.Package, App.Repository>) async throws -> Swift.Array<App.Version>, Argument Types : [Swift.StringApp.Joined<App.Package, App.Repository>]> of generic specialization <Swift.Array<App.Refe
rence>, App.Version> of (extension in Swift):Swift.Collection.map<A>((A.Element) throws -> A1) throws -> Swift.Array<A1> at /build/<compiler-generated>:0
2023-01-26T13:50:08.696444064Z app_analyze.1.luku8u31ozjl@p2    | 0x561f787920c7, (1) suspend resume partial function for function signature specialization <Arg[3] = Dead> of static App.Analyze.getIncomingVersions(client: Vapor.Client, logger: Logg
ing.Logger, package: App.Joined<App.Package, App.Repository>) async throws -> Swift.Array<App.Version> at /build/Sources/App/Commands/Analyze.swift:437
2023-01-26T13:50:08.696486964Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79e6520d
2023-01-26T13:50:08.696503764Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79e658fb
2023-01-26T13:50:08.696508064Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79837424
2023-01-26T13:50:08.696511764Z app_analyze.1.luku8u31ozjl@p2    | 0x561f798371d2
2023-01-26T13:50:08.696515264Z app_analyze.1.luku8u31ozjl@p2    | 0x561f79842ac1
2023-01-26T13:50:08.696518764Z app_analyze.1.luku8u31ozjl@p2    | 0x7f4d69f7c608
2023-01-26T13:50:08.696538164Z app_analyze.1.luku8u31ozjl@p2    | 0x7f4d69ac0132
2023-01-26T13:50:08.696542264Z app_analyze.1.luku8u31ozjl@p2    | 0xffffffffffffffff
2023-01-26T13:50:08.846038854Z app_analyze.1.luku8u31ozjl@p2    | /bin/bash: line 1: 20617 Illegal instruction     (core dumped) ./Run analyze --env prod --limit 25
2023-01-26T13:50:28.883543917Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] Analyzing (limit: 25) ... [component: analyze]
2023-01-26T13:50:28.953630593Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] Checkout directory: /checkouts [component: analyze]
2023-01-26T13:50:28.958182711Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/genius/future.git in /checkouts/github.com-genius-future [component: analyze]
2023-01-26T13:50:28.987394827Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/xiiagency/SwiftConcurrency.git in /checkouts/github.com-xiiagency-swiftconcurrency [component: analyze]
2023-01-26T13:50:29.001886784Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/JUSTINMKAUFMAN/SwiftUIPhone.git in /checkouts/github.com-justinmkaufman-swiftuiphone [component: analyze]
2023-01-26T13:50:29.226366270Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/patchthecode/JTAppleCalendar.git in /checkouts/github.com-patchthecode-jtapplecalendar [component: analyze]
2023-01-26T13:50:29.231274989Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/RougeWare/Swift-Optional-Tools.git in /checkouts/github.com-rougeware-swift-optional-tools [component: analyze]
2023-01-26T13:50:29.254494181Z app_analyze.1.luku8u31ozjl@p2    | [ INFO ] pulling https://github.com/apple/swift-corelibs-xctest.git in /checkouts/github.com-apple-swift-corelibs-xctest [component: analyze]

[finestructure]

Common theme:

Connection request timed out. This might indicate a connection deadlock in your application.

Not sure if that's a symptom or the cause, but worth investigating.

Worth noting that these hangs started appearing when I converted more ELFs to a/a recently.

[finestructure]

It's been a while since we've seen this, so I'm going to close this for now.