Multiple SortDescriptors sorting got slow with newer Realm versions

[Fab1n]

Goals

Sorted Results using SortDescriptors like this should be as fast as in the past (like in v3.19.0):

let sortedModels = realm.objects(Model.self)
    .sorted(by: [
        SortDescriptor(keyPath: "b", ascending: false), //Property `b` of type `Bool` (`false` on all objects)
        SortDescriptor(keyPath: "id", ascending: false)
    ])

Expected Results

Switching from Realm 3.19.0 to 10.1.4 sorted results should be equally performant as with the version switched from.

Actual Results

The performance of these sorted results dropped by a factor of 2 minimum.

Steps for others to Reproduce

Please follow the steps provided in the code sample repository.

But in a nutshell I will give you the TLDR; version here:

Having

• an encrypted Realm,
• an entity with 2 specific fields (String (UUID) and Bool (all false!) in this case - but could be others),
• having a lot of objects with this one entity (1_000_000 in this case),
• then sorting using SortDescriptors, first by the Bool property, then by the String property

is giving you a 2x performance drop on >= 10.1.4 compared to 3.19.0.

(Sidenote: Removing the first sort descriptor makes the sorting faster from 3.19.0 to 10.1.4)

Here is most of the code from the sample project. It is meant for discussion here.

Please use the sample project together with Instruments (and signposts) to see the time spent executing the sorted query by yourself.

If have included a detailed test description as well as a detailed problem description in the README.md file there.

The model should look like this:

open class Model: Object {
    @objc open dynamic var id = UUID().uuidString
    @objc open dynamic var b: Bool = false
    
    open override class func indexedProperties() -> [String] {
        return ["id"]
    }
}

The requirements for the model properties are quite strechy, you basically could use any property types there, as long as it is given that one property's value largely is the same on all objects of the to-be-sorted Results.

Connection to the database could look like this:

func appMigration(_ migration: Migration, oldSchemaVersion: UInt64) {

}

let dbName = "database.realm"
let documents = try! FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor: nil, create: false)
let dbURL = (try! FileManager.default.url(for: .itemReplacementDirectory, in: .userDomainMask, appropriateFor: documents, create: true)).appendingPathComponent("database.realm")
let encryptionKey: Data = {
    return Data(repeating: 7, count: 64)
}()

var config: Realm.Configuration {
    print("dbURL \(dbURL)")
    return .init(fileURL: dbURL, encryptionKey: encryptionKey, schemaVersion: 1, migrationBlock: appMigration, shouldCompactOnLaunch: { (Double($1) / Double($0)) < 0.5 })
}

let realm: Realm = {
    return try! Realm(configuration: config)
}()

Generating data should look like this:

func generateData() {
    let r = realm
    
    try! r.write {
        r.deleteAll()
        
        for _ in 0..<1_000_000 {
            let m = Model()
            r.add(m)
        }
    }
}

Executing the query should look like this:

func fetchResults() {
    let r = realm //Realm getter for the realm on disk - needs to be encrypted
    let txs = r.objects(Model.self)
        .sorted(by: [
            SortDescriptor(keyPath: "b", ascending: false), //This line 
            SortDescriptor(keyPath: "id", ascending: false) //along with this line in that combination 
                                                            //is causing performance issues 
                                                            //when most `b` values are equal
        ])
    for _ in 0...10 {
        _ = txs.randomElement()
    }
}

Code Sample

https://github.com/Fab1n/realm-encrypted-sort-performance

Referenced Issues:

These issues are interesting for the topic of sorting, but may not directly be connected to this issue.
#6786
#4089

Version of Realm and Tooling

ProductName:	macOS
ProductVersion:	11.1
BuildVersion:	20C69

/Applications/Xcode.app/Contents/Developer
Xcode 12.3
Build version 12C33

/usr/local/lib/ruby/gems/3.0.0/bin/pod
1.10.0

Realm versions used here 3.19.0, 10.1.4 and 10.5.1

/bin/bash
GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin20)

carthage not found
(not in use here)

/usr/local/bin/git
git version 2.30.0

iOS/OSX version: iOS Simulator 14.3 iPhone 12 Pro / macOS 11.1

[leemaguire]

@Fab1n Thanks for providing us with the above information and sample app. I've ran the two versions through time profiler and sent the findings onto the team.

[finnschiermer]

I can confirm there is some unexpected kind of interaction between the sorting and the encryption layer, not seen before Core6.

[Fab1n]

Thanks @finnschiermer and @leemaguire for your help and confirmation.
Please keep me on track with any fixes in Core or Realm regarding this issue as well as any more findings.

[jedelbo]

I will have a look at this.

[jedelbo]

Was not able to spot the main cause for this degradation. But I can see that the new design takes a hit when encryption is on - this does not happen so much in the old design.

[jedelbo]

I assume the performance improvement we have done provides remedy for the issue. Closing.