How to have performant fetches from a data stack without locking the main thread, but still allow reads concurrently with writes

[jimbengtsson92]

In our app all of our writes basically look like this:

dataStack.perform(asynchronous: { (transaction) -> Void in
    // Create, update or delete on transaction
    try transaction.create(Into<MyEntity>()) / deleteAll(From<MyEntity>(.where(...)))
}, completion: { result in
    completion()
})

and our reads like this:

DispatchQueue.main.async { [self] in
    do {
        // Fetch one or many entities as snapshots from dataStack
        // We use the snapshots in order to work with objects that might be deleted while we are reading them
        let snapshots = dataStack.fetchAll(From<MyEntity>(.where(...))).compactMap { $0.asSnapshot(in: dataStack) }
        completion(snapshots.transformToMyModels())
    } catch {
        // Handle error
    }
}

With this setup we can achieve concurrent read and writes, but when performing a read this is locking the main thread. If I do not perform the fetch on the main thread I get a fatalError saying "Attempted to fetch from a 'CoreStore.DataStack' outside the main thread." error from CoreStore.

One way I have found we can do it to perform reads without locking the main thread is to use a transaction for these as well, like this:

dataStack.perform(asynchronous: { (transaction) -> [MyModel] in
    // Fetch one or many entities as snapshots from transaction
    let snapshots = transaction.fetchAll(From<MyEntity>(.where(...))).compactMap { $0.asSnapshot(in: transaction) }
    return snapshots.transformToMyModels()
}, completion: { result in
    completion(result)
})

With this approach I cannot however perform reads concurrently with writes, since they are both run on the same background serial queue that is created from the transaction. This is making the performance way worse in the cases when we are trying to do writes at around the same time as we do reads.

Do you, or anyone else, have any advice on how to do create fetches that do not lock the main thread, but can still run concurrently with writes?

[JohnEstropia]

The fetches are fine, but the error message indicates you are accessing the fetched objects' properties outside the main thread

[jimbengtsson92]

@JohnEstropia

The fetches are fine, but the error message indicates you are accessing the fetched objects' properties outside the main thread

The error comes from this method in DataStack+Querying.swift:

public func fetchAll<B: FetchChainableBuilderType>(_ clauseChain: B) throws -> [B.ObjectType] {

        Internals.assert(
            Thread.isMainThread,
            "Attempted to fetch from a \(Internals.typeName(self)) outside the main thread."
        )
        return try self.mainContext.fetchAll(clauseChain)
    }

So it looks like it is the actual fetch that is the problem.

An example of a complete fetch where this error happens is:

let scheduledEventSnapshots = try self.dataStack.fetchAll(
                        From<ScheduledEventEntity>()
                            .where(combineByOr: (\.scheduledUntil >= from && \.scheduledFrom <= to), (\.scheduledFrom < from && \.scheduledUntil > to))
                    ).compactMap { $0.asSnapshot(in: self.dataStack) }

[JohnEstropia]

@jimbengtsson92 I see, then it's exactly as the method says: fetches from DataStack instances should come from the main thread. If you want to fetch things asynchronously, you can write something like:

DispatchQueue.global(qos: .utility).async {
    withExtendedLifetime(dataStack.beginUnsafe()) { transaction in
        let snapshots = try? transaction.fetchAll(...).compactMap({ $0.asSnapshot()  })
        completion(snapshots)
    }
}
// code written by hand, so there might be compiler errors

Just be careful not to access the relationship objects directly from the snapshots, as those need to be fetched again from either a DataStack or a transaction.

Then when you need to edit the objects for these snapshots inside transactions, convert them back into the transaction instance by doing snapshot.asEditable(in: transaction)