Issues with thousands of objects

[jpmcosta]

Issue Basics

• ObjectBox version: 1.2.1
• Reproducibility:
  • Issue 1: always
  • Issue 2 most times, it some persistence to reproduce

Reproducing the bug

Description

I have created a project to test ObjectBox's performance with thousands of objects:
https://github.com/jpmcosta/ObjectBoxTestProject/tree/ab97ac23ce6fa408f1313992fc72091fd52b3a9d

I have found 2 issues that I'm going to describe here as they seem to be related and I think it would be spammy to create multiple issues.

Code summary

The project has an App class that recreates data on each run. You can specify ITEM_COUNT and NOTE_COUNT_PER_ITEM to change the number of objects created.

A ProjectAdapter (ItemListAdapter) is used to display items of a project in a RecyclerView, using the following query:

fun BoxStore.getProjectItemsQuery(id: Long): Query<Item> =
        boxFor(Item::class.java).query()
                .equal(Item_.parentProjectId, id)
                .build()

When you click on an item in the list, Item.isActive is toggled.

Entities

@Entity
class Project {
    
    @Id
    var id: Long = NEW_ID

    @Backlink
    lateinit var items: ToMany<Item>
}

@Entity
class Item {

    @Id
    var id: Long = NEW_ID

    var isActive: Boolean = true

    @Backlink
    lateinit var notes: ToMany<Note>

    lateinit var parentProject: ToOne<Project>
}

@Entity
class Note {

    @Id
    var id: Long = NEW_ID

    var text: String? = null

    lateinit var parentItem: ToOne<Item>
}

Issue 1

If you set ITEM_COUNT to 20K and NOTE_COUNT_PER_ITEM to 4 (total: 80K objects), the objects are never created (I've waited a few minutes).

Issue 2

If you set ITEM_COUNT to 20K/80K and NOTE_COUNT_PER_ITEM to 0 (total: 20K/80K objects), the objects are created. However, there are 2 issues:

• there's lag when toggling Item.isActive (I guess that's just a limitation)
• if you click repeatedly on items in the list, eventually the DataObserver defined in ProjectAdapter will stop getting notified of changes. The toggle operation will still work (you can test that by scrolling the list to check if the item has changed).
  • to reproduce this issue it helps to spam click multiple items at the same time

Misc

I couldn't reproduce those issues when I removed @Backlink annotation from Item.notes AND removed parentItem property from Note.

[greenrobot]

I think you're the first one using relations to do bulk-puts. Didn't expect that use case. Will look in detail soon.

[jpmcosta]

Then, may be important to say that I'm only doing those bulk-puts on that test project, not in my application.
Still, since I detected some issues, I had to report them.

[greenrobot]

Sure, and why not. We'll definitely add something like this to our test suite.

[dgandhi17]

@greenrobot any updates on this ? as in my new app I have to deal with it so before I move with this library i will have idea. how to deal with it

[greenrobot]

@rspl17 Please give some details about what you are doing.

[dgandhi17]

@greenrobot : I have exchange address book with 100k of data. having many to many relationship between person and groups. so that I am planning to use objectbox for faster performance, as I may have complex queries.

[greenrobot]

@rspl17 OK, so you should put your entities first. Then init the relations. Both can be done in a single transaction.

[dgandhi17]

@greenrobot do the greenrobot team has any chat channel giiter or slack for quick support ?

[motornyimaksym]

@greenrobot I wondering if there are any problems updating relations when existing entity has 10k related items or 100k and even 1000k. E.g. I have 120k entities of Work and few entities of WorkType. They has Many-To-Many relation. When I want to update some WorkType I'm trying to get it from db like this:
boxStore.boxFor(WorkType::class.java)[workTypeId]
And during this process I can get OOM exception if there are already 100k+ related works attached to this WorkType. Can you confirm that such case possible?
It's easy to use One-To-Many relations by Backlink annotation cause I'm just updating one small field ToOne in one small entity (e.g. Work).
Even though OOM maybe is not achievable the time taken for update Many-To-Many is growing up like crazy when I got more and more related Works to WorkType.

[greenrobot]

@rspl17 Support is done via stackoverflow.com
@Yazon2006 We're currently looking into the original issue. If you are getting an OOM, please open a new ticket with a stack trace and some more info - it's not entirely clear to me what you are trying to do. Thanks!

[greenrobot-team]

OK, starting by trying to reproduce with the given project:

Issue 1

• can not reproduce on emulator (Android 8.0, x86), though put takes a few seconds and uses up to 150 MB of memory
• can not reproduce on Moto X Play (Android 7.1.1), though put takes about a minute and up to 75 MB of memory

Note: it takes considerably longer if the debugger is attached.

Issue 2
Can reproduce.

• The lag between tapping and the UI update is due to the large dataset. I recommend using a paging approach instead of showing the whole data set. Not sure how easy this is to do with ObjectBox.
• Though the observer not getting notified after a while is caused by the subscription getting garbage collected. You can verify this by forcing garbage collection using the memory profiler in Android Studio. This can be easily fixed by keeping a reference to the subscription, using your code:

# ProjectAdapter.kt:22
- boxStore.getProjectItemsQuery(projectId)
+ subscription = boxStore.getProjectItemsQuery(projectId)
        .subscribe()
        .on(AndroidScheduler.mainThread())
        .observer(this)

As there is already a subscription property I guess you just forgot to assign it.

-ut

[greenrobot-team]

Update: finally could do a little profiling and it turns out the majority of time is spent constructing and adding. Only a fraction with the actual put() call.

For 2000 items with 4 notes each:

Creating objects took 10167 ms
Putting took 2447 ms

A reasonable speed increase can be achieved by changing the backing list of ToMany from a CopyOnWriteArrayList to a simple ArrayList. Code snippet:

val listFactory = ListFactory.ArrayListFactory()
project.items.listFactory = listFactory
...
    item.notes.listFactory = listFactory

Creating objects took 8625 ms
Putting took 2563 ms

Modified example project: ObjectBoxThousandsProject.zip

Test notes: with Android Profiler enabled, x86 emulator, Android 8.0.
-ut

[greenrobot-team]

Closing this issue due to inactivity. 💤 Feel free to re-open with more details or submit a new issue.