Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

MetadataAllocator::Allocate fatalError - Fixed in iOS 15.4.1 #790

Closed
sergiymomot opened this issue Mar 24, 2022 · 13 comments
Closed

MetadataAllocator::Allocate fatalError - Fixed in iOS 15.4.1 #790

sergiymomot opened this issue Mar 24, 2022 · 13 comments
Assignees
@maxep
Labels
crash SDK crashes

Comments

@sergiymomot
Copy link

sergiymomot commented Mar 24, 2022 •
edited
Loading

The crash

Started noticing the following crash reports in Firebase.

Error headline in Firebase Crashlytics:

<compiler-generated> line 2147483647
FilesOrchestrator.getReadableFile(excludingFilesNamed:)

Stack trace:

Crashed: com.datadoghq.ios-sdk-tracing-read-write
0  libsystem_kernel.dylib         0x6bbc __pthread_kill + 8
1  libsystem_pthread.dylib        0xd854 pthread_kill + 208
2  libsystem_c.dylib              0x1f6a4 abort + 124
3  libswiftCore.dylib             0x2ed01c swift::warningv(unsigned int, char const*, char*) + 138
4  libswiftCore.dylib             0x2f6698 swift::MetadataAllocator::Allocate(unsigned long, unsigned long) + 516
5  libswiftCore.dylib             0x300fd8 swift::allocateMetadata(unsigned long, unsigned long) + 36
6  libswiftCore.dylib             0x2eb504 swift_initEnumMetadataSinglePayload + 172
7  libswiftCore.dylib             0x2c3f5c type metadata completion function for ClosedRange<>.Index + 56
8  libswiftCore.dylib             0x302f68 swift::MetadataCacheEntryBase<(anonymous namespace)::GenericCacheEntry, void const*>::doInitialization(swift::MetadataWaitQueue::Worker&, swift::MetadataRequest) + 280
9  libswiftCore.dylib             0x2f74cc _swift_getGenericMetadata(swift::MetadataRequest, void const* const*, swift::TargetTypeContextDescriptor<swift::InProcess> const*) + 1732
10 libswiftCore.dylib             0x2dad68 __swift_instantiateCanonicalPrespecializedGenericMetadata + 32
11 libswiftCore.dylib             0x318d10 (anonymous namespace)::DecodedMetadataBuilder::createBoundGenericType(swift::TargetContextDescriptor<swift::InProcess> const*, __swift::__runtime::llvm::ArrayRef<swift::TargetMetadata<swift::InProcess> const*>, swift::TargetMetadata<swift::InProcess> const*) const + 332
12 libswiftCore.dylib             0x3175a0 swift::Demangle::__runtime::TypeDecoder<(anonymous namespace)::DecodedMetadataBuilder>::decodeMangledType(swift::Demangle::__runtime::Node*, unsigned int, bool) + 19196
13 libswiftCore.dylib             0x312024 swift_getTypeByMangledNodeImpl(swift::MetadataRequest, swift::Demangle::__runtime::Demangler&, swift::Demangle::__runtime::Node*, void const* const*, std::__1::function<swift::TargetMetadata<swift::InProcess> const* (unsigned int, unsigned int)>, std::__1::function<swift::TargetWitnessTable<swift::InProcess> const* (swift::TargetMetadata<swift::InProcess> const*, unsigned int)>) + 552
14 libswiftCore.dylib             0x311d9c swift_getTypeByMangledNode + 540
15 libswiftCore.dylib             0x312510 swift_getTypeByMangledNameImpl(swift::MetadataRequest, __swift::__runtime::llvm::StringRef, void const* const*, std::__1::function<swift::TargetMetadata<swift::InProcess> const* (unsigned int, unsigned int)>, std::__1::function<swift::TargetWitnessTable<swift::InProcess> const* (swift::TargetMetadata<swift::InProcess> const*, unsigned int)>) + 968
16 libswiftCore.dylib             0x30fd14 swift_getTypeByMangledName + 540
17 libswiftCore.dylib             0x30ff28 swift_getTypeByMangledNameInContext + 132
18 App                            0x95d87c __swift_instantiateConcreteTypeFromMangledName + 4312684668 (<compiler-generated>:4312684668)
19 App                            0x973828 FilesOrchestrator.getReadableFile(excludingFilesNamed:) + 4312774696 (<compiler-generated>:4312774696)
20 App                            0x978f7c FileReader.readNextBatch() + 33 (FileReader.swift:33)
21 App                            0x979368 protocol witness for Reader.readNextBatch() in conformance FileReader + 4312798056 (<compiler-generated>:4312798056)
22 App                            0x978d44 partial apply for closure #1 in DataReader.readNextBatch() + 4312796484 (<compiler-generated>:4312796484)
23 libswiftDispatch.dylib         0x183c partial apply for thunk for @callee_guaranteed () -> (@out A, @error @owned Error) + 20
24 libswiftDispatch.dylib         0x1854 thunk for @callee_guaranteed () -> (@out A, @error @owned Error)partial apply + 12
25 libswiftDispatch.dylib         0x29ac closure #1 in closure #1 in OS_dispatch_queue._syncHelper<A>(fn:execute:rescue:) + 120
26 libswiftDispatch.dylib         0x2a74 partial apply for thunk for @callee_guaranteed () -> () + 20
27 libswiftDispatch.dylib         0x2478 thunk for @escaping @callee_guaranteed () -> () + 20
28 libdispatch.dylib              0x64094 _dispatch_client_callout + 16
29 libdispatch.dylib              0x46464 _dispatch_lane_barrier_sync_invoke_and_complete + 52
30 libswiftDispatch.dylib         0x1e18 implicit closure #2 in implicit closure #1 in OS_dispatch_queue.sync<A>(execute:) + 152
31 libswiftDispatch.dylib         0x10ac partial apply for implicit closure #2 in implicit closure #1 in OS_dispatch_queue.sync<A>(execute:) + 40
32 libswiftDispatch.dylib         0x1bb0 OS_dispatch_queue._syncHelper<A>(fn:execute:rescue:) + 256
33 libswiftDispatch.dylib         0x1140 OS_dispatch_queue.sync<A>(execute:) + 140
34 App                            0x978bcc protocol witness for Reader.readNextBatch() in conformance DataReader + 4312796108 (<compiler-generated>:4312796108)
35 App                            0x98a5fc closure #1 in DataUploadWorker.init(queue:fileReader:dataUploader:uploadConditions:delay:featureName:internalMonitor:) + 62 (DataUploadWorker.swift:62)
36 App                            0x9b7f34 thunk for @escaping @callee_guaranteed () -> () + 4313055028 (<compiler-generated>:4313055028)
37 libdispatch.dylib              0x46048 _dispatch_block_async_invoke2 + 104
38 libdispatch.dylib              0x64094 _dispatch_client_callout + 16
39 libdispatch.dylib              0x3bab8 _dispatch_continuation_pop$VARIANT$armv81 + 432
40 libdispatch.dylib              0x4d16c _dispatch_source_invoke$VARIANT$armv81 + 1560
41 libdispatch.dylib              0x3f380 _dispatch_lane_serial_drain$VARIANT$armv81 + 308
42 libdispatch.dylib              0x3ff44 _dispatch_lane_invoke$VARIANT$armv81 + 388
43 libdispatch.dylib              0x498e0 _dispatch_workloop_worker_thread + 608
44 libsystem_pthread.dylib        0x1e10 _pthread_wqthread + 284
45 libsystem_pthread.dylib        0x193c start_wqthread + 8

Datadog SDK versions:

1.8.0

Last stable Datadog SDK version:

We currently have version 1.8.0 integrated, but it doesn't seem like a version issue, since 1.8.0 was integrated some time before the crash started to happen.

Volume:

About 10-15 crashes a day, but it is our top crash report.

OS version:

From Firebase:

  • 95% iOS 15
  • 5% iOS 12

But looking at individual crash reports, I was only able to observe iOS 15.4.0 version specifically (and iOS 12.1.*).

Deployment Target:

The minimum deployment target is iOS 12, iPhone only

Device version:

All sorts of devices, but older iPhones (6*, 7, 8 models) are in dominance.

Environment:

Definitely not a disk space issue since most reports in Crashlytics show over 50Gb of free disk space.
However, most reports showed relatively low free RAM space - <100Mb
@sergiymomot sergiymomot added the crash SDK crashes label Mar 24, 2022
@sergiymomot
Copy link
Author

Potentially related crash, also happens only on iOS 15.4.0

Directory.swift line 49
Directory.files()

Crashed: com.datadoghq.ios-sdk-tracing-read-write
SIGABRT ABORT 0x00000001bce72bbc

Stack trace:

Crashed: com.datadoghq.ios-sdk-tracing-read-write
0  libsystem_kernel.dylib         0x6bbc __pthread_kill + 8
1  libsystem_pthread.dylib        0xd854 pthread_kill + 208
2  libsystem_c.dylib              0x1f6a4 abort + 124
3  libswiftCore.dylib             0x2ed01c swift::warningv(unsigned int, char const*, char*) + 138
4  libswiftCore.dylib             0x2f6698 swift::MetadataAllocator::Allocate(unsigned long, unsigned long) + 516
5  libswiftCore.dylib             0x2ff674 swift::_getWitnessTable(swift::TargetProtocolConformanceDescriptor<swift::InProcess> const*, swift::TargetMetadata<swift::InProcess> const*, void const* const*) + 2272
6  libswiftCore.dylib             0x31d574 swift::TargetProtocolConformanceDescriptor<swift::InProcess>::getWitnessTable(swift::TargetMetadata<swift::InProcess> const*) const + 404
7  libswiftCore.dylib             0x31edb4 swift_conformsToProtocolMaybeInstantiateSuperclasses(swift::TargetMetadata<swift::InProcess> const*, swift::TargetProtocolDescriptor<swift::InProcess> const*, bool) + 1652
8  libswiftCore.dylib             0x31e4ec swift_conformsToProtocol + 100
9  libswiftCore.dylib             0x2e5ddc findBridgeWitness(swift::TargetMetadata<swift::InProcess> const*) + 72
10 libswiftCore.dylib             0x2e5be8 bridgeAnythingNonVerbatimToObjectiveC(swift::OpaqueValue*, swift::TargetMetadata<swift::InProcess> const*, bool) + 300
11 libswiftCore.dylib             0x7d598 _ContiguousArrayStorage._getNonVerbatimBridgingBuffer() + 344
12 libswiftCore.dylib             0x1beff4 __SwiftDeferredNSArray.withUnsafeBufferOfObjects<A>(_:) + 212
13 libswiftCore.dylib             0x1bddd8 @objc __SwiftNativeNSArrayWithContiguousStorage.getObjects(_:range:) + 92
14 CoreFoundation                 0x2473c -[NSArray initWithArray:range:copyItems:] + 248
15 CoreFoundation                 0x29838 -[__NSPlaceholderArray initWithArray:copyItems:] + 104
16 CoreServicesInternal           0x2cb4 _URLEnumeratorCreateForDirectoryURL + 80
17 Foundation                     0x87744 -[NSFileManager contentsOfDirectoryAtURL:includingPropertiesForKeys:options:error:] + 192
18 App                            0x970644 Directory.files() + 49 (Directory.swift:49)
19 App                            0x9739d4 FilesOrchestrator.getReadableFile(excludingFilesNamed:) + 93 (FilesOrchestrator.swift:93)
20 App                            0x978f7c FileReader.readNextBatch() + 33 (FileReader.swift:33)
21 App                            0x979368 protocol witness for Reader.readNextBatch() in conformance FileReader + 4305425256 (<compiler-generated>:4305425256)
22 App                            0x978d44 partial apply for closure #1 in DataReader.readNextBatch() + 4305423684 (<compiler-generated>:4305423684)
23 libswiftDispatch.dylib         0x183c partial apply for thunk for @callee_guaranteed () -> (@out A, @error @owned Error) + 20
24 libswiftDispatch.dylib         0x1854 thunk for @callee_guaranteed () -> (@out A, @error @owned Error)partial apply + 12
25 libswiftDispatch.dylib         0x29ac closure #1 in closure #1 in OS_dispatch_queue._syncHelper<A>(fn:execute:rescue:) + 120
26 libswiftDispatch.dylib         0x2a74 partial apply for thunk for @callee_guaranteed () -> () + 20
27 libswiftDispatch.dylib         0x2478 thunk for @escaping @callee_guaranteed () -> () + 20
28 libdispatch.dylib              0x64094 _dispatch_client_callout + 16
29 libdispatch.dylib              0x11960 _dispatch_lane_barrier_sync_invoke_and_complete + 52
30 libswiftDispatch.dylib         0x1e18 implicit closure #2 in implicit closure #1 in OS_dispatch_queue.sync<A>(execute:) + 152
31 libswiftDispatch.dylib         0x10ac partial apply for implicit closure #2 in implicit closure #1 in OS_dispatch_queue.sync<A>(execute:) + 40
32 libswiftDispatch.dylib         0x1bb0 OS_dispatch_queue._syncHelper<A>(fn:execute:rescue:) + 256
33 libswiftDispatch.dylib         0x1140 OS_dispatch_queue.sync<A>(execute:) + 140
34 App                            0x978bcc protocol witness for Reader.readNextBatch() in conformance DataReader + 4305423308 (<compiler-generated>:4305423308)
35 App                            0x98a5fc closure #1 in DataUploadWorker.init(queue:fileReader:dataUploader:uploadConditions:delay:featureName:internalMonitor:) + 62 (DataUploadWorker.swift:62)
36 App                            0x9b7f34 thunk for @escaping @callee_guaranteed () -> () + 4305682228 (<compiler-generated>:4305682228)
37 libdispatch.dylib              0x11540 _dispatch_block_async_invoke2 + 104
38 libdispatch.dylib              0x64094 _dispatch_client_callout + 16
39 libdispatch.dylib              0x6bb4 _dispatch_continuation_pop$VARIANT$mp + 440
40 libdispatch.dylib              0x188d8 _dispatch_source_invoke$VARIANT$mp + 1668
41 libdispatch.dylib              0xa60c _dispatch_lane_serial_drain$VARIANT$mp + 344
42 libdispatch.dylib              0xb1f0 _dispatch_lane_invoke$VARIANT$mp + 408
43 libdispatch.dylib              0x14ec4 _dispatch_workloop_worker_thread + 632
44 libsystem_pthread.dylib        0x1e10 _pthread_wqthread + 284
45 libsystem_pthread.dylib        0x193c start_wqthread + 8

@LeffelMania
Copy link

We're seeing the same thing manifest in the DatadogSDK and other places in our app. We're pretty confident it was caused by a change in iOS 15.4.0 released on March 14th, based on the timing and impact we're seeing in our crash reports. I've filed a bug with the Swift community here and I've filed the same in Feedback Assistant, but it would be great to get more development teams to make noise about it.

@MightyFine
Copy link

We're seeing the same thing 🥲

Crashed: com.datadoghq.ios-sdk-tracing-read-write
0  libsystem_kernel.dylib         0x6bbc __pthread_kill + 8
1  libsystem_pthread.dylib        0xd854 pthread_kill + 208
2  libsystem_c.dylib              0x1f6a4 abort + 124
3  libswiftCore.dylib             0x2ed01c swift::warningv(unsigned int, char const*, char*) + 138
4  libswiftCore.dylib             0x2f6698 swift::MetadataAllocator::Allocate(unsigned long, unsigned long) + 516
5  libswiftCore.dylib             0x2f71f4 _swift_getGenericMetadata(swift::MetadataRequest, void const* const*, swift::TargetTypeContextDescriptor<swift::InProcess> const*) + 1004
6  libswiftCore.dylib             0x2dad68 __swift_instantiateCanonicalPrespecializedGenericMetadata + 32
7  libswiftCore.dylib             0x318d10 (anonymous namespace)::DecodedMetadataBuilder::createBoundGenericType(swift::TargetContextDescriptor<swift::InProcess> const*, __swift::__runtime::llvm::ArrayRef<swift::TargetMetadata<swift::InProcess> const*>, swift::TargetMetadata<swift::InProcess> const*) const + 332
8  libswiftCore.dylib             0x3175a0 swift::Demangle::__runtime::TypeDecoder<(anonymous namespace)::DecodedMetadataBuilder>::decodeMangledType(swift::Demangle::__runtime::Node*, unsigned int, bool) + 19196
9  libswiftCore.dylib             0x312024 swift_getTypeByMangledNodeImpl(swift::MetadataRequest, swift::Demangle::__runtime::Demangler&, swift::Demangle::__runtime::Node*, void const* const*, std::__1::function<swift::TargetMetadata<swift::InProcess> const* (unsigned int, unsigned int)>, std::__1::function<swift::TargetWitnessTable<swift::InProcess> const* (swift::TargetMetadata<swift::InProcess> const*, unsigned int)>) + 552
10 libswiftCore.dylib             0x311d9c swift_getTypeByMangledNode + 540
11 libswiftCore.dylib             0x312510 swift_getTypeByMangledNameImpl(swift::MetadataRequest, __swift::__runtime::llvm::StringRef, void const* const*, std::__1::function<swift::TargetMetadata<swift::InProcess> const* (unsigned int, unsigned int)>, std::__1::function<swift::TargetWitnessTable<swift::InProcess> const* (swift::TargetMetadata<swift::InProcess> const*, unsigned int)>) + 968
12 libswiftCore.dylib             0x30fd14 swift_getTypeByMangledName + 540
13 libswiftCore.dylib             0x30ff28 swift_getTypeByMangledNameInContext + 132
14 theathletic-ios                0xed1754 __swift_instantiateConcreteTypeFromMangledName + 4325463892 (<compiler-generated>:4325463892)
15 theathletic-ios                0xee5658 FilesOrchestrator.getReadableFile(excludingFilesNamed:) + 4325545560 (<compiler-generated>:4325545560)
16 theathletic-ios                0xeead5c FileReader.readNextBatch() + 33 (FileReader.swift:33)
17 theathletic-ios                0xeeb148 protocol witness for Reader.readNextBatch() in conformance FileReader + 4325568840 (<compiler-generated>:4325568840)
18 theathletic-ios                0xeeab24 partial apply for closure #1 in DataReader.readNextBatch() + 4325567268 (<compiler-generated>:4325567268)
19 libswiftDispatch.dylib         0x183c partial apply for thunk for @callee_guaranteed () -> (@out A, @error @owned Error) + 20
20 libswiftDispatch.dylib         0x1854 thunk for @callee_guaranteed () -> (@out A, @error @owned Error)partial apply + 12
21 libswiftDispatch.dylib         0x29ac closure #1 in closure #1 in OS_dispatch_queue._syncHelper<A>(fn:execute:rescue:) + 120
22 libswiftDispatch.dylib         0x2a74 partial apply for thunk for @callee_guaranteed () -> () + 20
23 libswiftDispatch.dylib         0x2478 thunk for @escaping @callee_guaranteed () -> () + 20
24 libdispatch.dylib              0x64094 _dispatch_client_callout + 16
25 libdispatch.dylib              0x46464 _dispatch_lane_barrier_sync_invoke_and_complete + 52
26 libswiftDispatch.dylib         0x1e18 implicit closure #2 in implicit closure #1 in OS_dispatch_queue.sync<A>(execute:) + 152
27 libswiftDispatch.dylib         0x10ac partial apply for implicit closure #2 in implicit closure #1 in OS_dispatch_queue.sync<A>(execute:) + 40
28 libswiftDispatch.dylib         0x1bb0 OS_dispatch_queue._syncHelper<A>(fn:execute:rescue:) + 256
29 libswiftDispatch.dylib         0x1140 OS_dispatch_queue.sync<A>(execute:) + 140
30 theathletic-ios                0xeea9ac protocol witness for Reader.readNextBatch() in conformance DataReader + 4325566892 (<compiler-generated>:4325566892)
31 theathletic-ios                0xef77cc closure #1 in DataUploadWorker.init(queue:fileReader:dataUploader:uploadConditions:delay:featureName:) + 63 (DataUploadWorker.swift:63)
32 theathletic-ios                0xf21644 thunk for @escaping @callee_guaranteed () -> () + 4325791300 (<compiler-generated>:4325791300)
33 libdispatch.dylib              0x46048 _dispatch_block_async_invoke2 + 104
34 libdispatch.dylib              0x64094 _dispatch_client_callout + 16
35 libdispatch.dylib              0x3bab8 _dispatch_continuation_pop$VARIANT$armv81 + 432
36 libdispatch.dylib              0x4d16c _dispatch_source_invoke$VARIANT$armv81 + 1560
37 libdispatch.dylib              0x3f380 _dispatch_lane_serial_drain$VARIANT$armv81 + 308
38 libdispatch.dylib              0x3ff44 _dispatch_lane_invoke$VARIANT$armv81 + 388
39 libdispatch.dylib              0x498e0 _dispatch_workloop_worker_thread + 608
40 libsystem_pthread.dylib        0x1e10 _pthread_wqthread + 284
41 libsystem_pthread.dylib        0x193c start_wqthread + 8

I've voted on that issue you linked to @LeffelMania, let me know if there's a better way to make noise around this. Hope we can get some information on how to avoid it.

@maxep
Copy link
Member

maxep commented Mar 28, 2022

Hey 👋 Thanks for your reports, this is super helpful. We are currently investigating this issue, we will keep you posted with our findings!

@maxep maxep self-assigned this Mar 28, 2022
@maxep
Copy link
Member

maxep commented Mar 28, 2022

Thanks, @LeffelMania, for filling SR-16047! We also suspect that this bug was introduced in Swift 5.6, and more specifically by this assertion.

Could you please answer the following to help us move forward?

  • Did your impacted application binaries were built with Xcode 13.3?
  • Have you identified steps to reproduce?

@LeffelMania
Copy link

  • Our application binaries were built in CircleCI with our configuration set to Xcode 13.2.1 (which is a good reminder for us to update that value to 13.3).
# config.yml
  machine:
    macos:
      xcode: "13.2.1"
  • We have not identified steps to reproduce. It's manifesting in this library and a JSON serialization library we use, and it's a fairly low-frequency crash relative to the size of our user base (though it's been enough to drop our 7-day crash-free rate from 99.99% to 99.94%), so it seems like it's just something that sporadically happens and we don't have the internal volume to reproduce it.

@sergiymomot
Copy link
Author

Same, our latest app version was built using Xcode 13.2.1

@mohamedTarek95
Copy link

We're facing the same issue as well. App built using Xcode 13.2.x

@maxep
Copy link
Member

maxep commented Mar 29, 2022

From SR-16047:
It appears that this crash results from a bug in the compiler, bug which has been fixed in swiftlang/llvm-project#3917

The Swift 5.6 Runtime ends up being affected by this issue, and since the Swift Runtime is part of the OS, there is not much we can do to workaround this. We are still looking for steps to reproduce to be able to find mitigation and reduce crash rate.

@maxep maxep changed the title FilesOrchestrator.getReadableFile(excludingFilesNamed:) crash Swift 5.6 MetadataAllocator::Allocate fatalError Mar 29, 2022
@sergiymomot
Copy link
Author

Great news on SR-16047:

Today's 15.4.1 update includes this fix.

Basically, an Apple engineer said that the MetadataAllocator issue has been fixed in iOS 15.4.1 release.

@maxep you can close the issue or keep it for reference, if you wish

@MightyFine
Copy link

Happy days!! Thanks for chasing that up and making it happen 🙂

@maxep maxep changed the title Swift 5.6 MetadataAllocator::Allocate fatalError Swift 5.6 MetadataAllocator::Allocate fatalError - Fixed in iOS 15.4.1 Apr 1, 2022
@maxep
Copy link
Member

maxep commented Apr 1, 2022

Great news indeed 👍 I will keep this issue open for now.
Thanks everyone!

@maxep maxep mentioned this issue Jun 7, 2022
Closed
@maxep maxep changed the title Swift 5.6 MetadataAllocator::Allocate fatalError - Fixed in iOS 15.4.1 MetadataAllocator::Allocate fatalError - Fixed in iOS 15.4.1 Sep 14, 2022
@maxep
Copy link
Member

maxep commented May 3, 2023

Closing as iOS 15.4 is far behind. Thanks everyone for your help on this 🙏

@maxep maxep closed this as completed May 3, 2023
@maxep maxep mentioned this issue Sep 19, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maxep maxep

Labels
crash SDK crashes
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@LeffelMania @maxep @MightyFine @sergiymomot @mohamedTarek95