Found key was not removed

[PierceZ]

I'm seeing this logged a bunch when I call remove() on a Query:
Found key was not removed: 123

Query looks like this:
dataBox.query().equal(Data_.parentId, parentId).build().remove();

It's being run inside boxStore.runInTx().

The thing is, Data has 0 rows at this point. And the IDs it's logging are not part of Data.

The IDs are actually part of another table, let's call it OtherData. They aren't related in any way. Why is it logging out that some keys weren't removed for a different table? Even if it was the correct table, I'm not sure why it's logging this at all.

EDIT: On further inspection, it seems like it's logging this for every similar query, regardless of the table. I have a 3rd table (let's call it ThirdTable) where I'm performing a similar query, it does have some data that is being removed, but it also logs out "Found key was not removed" for every row in the OtherData table.

Also, when the transaction finishes when I have 2 of these queries, it logs out: "Illegal state: Tx destroyed/inactive, writeable cursor still available".

If I comment one out, it doesn't log the illegal state message.

Normally I'm seeing a bunch of "Found key was not removed" for each key, and then a 2nd time when it runs the 2nd query, and then at the end when the transaction is complete, I get the illegal state message. Aside from the logs, it seems to be all working properly though. All the data looks correct.

Also, I forgot to mention. ThirdTable has a 1 to many relation with Data, as in, ThirdTable has a list of Data, and Data has one ThirdTable. I'm using @Relation and @Backlink.

[greenrobot]

Thanks for your observations in your app. To go the next step, we would need to isolate the issues. Only then we would be able to reproduce and fix issues. Your help in breaking down the issue would greatly help us. First, some general questions: to what degree does your app run multi-threaded? Do you use indexes, especially on the queried properties? Do you use explicit transactions? Anything else special about your app?

[PierceZ]

The app is multi-threaded in the sense that I'm running ObjectBox transactions concurrently. However, in this case it's actually only 1 transaction that's running, so I would say it's not relevant to this issue most likely.

I do use indexes, and the property that is being queried is indexed. They are all named the same too ("parentId"), if that matters.

I'm not sure what you mean by explicit transactions. I use the method:

boxStore.runInTx(() -> {
// queries happen here
});

I can't think of anything else particularly special at the moment. Aside from uninstalling the app and doing a clean build, is there any kind of reset I can do? Is it possible the default.json file is in a bad state or something, and could be fixed by deleting it? I assume doing a clean build would've re-built it anyway so I doubt that would help, but let me know if there's anything I should try, or anything I should log out.

I was narrowing down the issue and I can say for sure that having only 1 line of code in the transaction (the query I mentioned, calling .remove()) causes the issue. To me it seems clear that something strange is happening in .remove(). Is there any way I can log out more details about what's happening here?

[PierceZ]

Well, I have a temporary workaround. Or at least, the errors aren't being logged with this change.

All I did was add an extra condition and it "fixes" it. I would prefer to fix it properly, but it's something at least.

Here's what I did, I changed this:

                            dataBox.query()
                                    .equal(Data_.parentId, parentId)
                                    .build()
                                    .remove();

Into this:

                            dataBox.query()
                                    .equal(Data_.parentId, parentId)
                                    .notEqual(Data_.id, 0)
                                    .build()
                                    .remove();

You can see I just added what I would consider a redundant condition (the id is never 0 so this is pointless). It fixes it though, for whatever reason. It seems like there's some weird issue with just doing an equal query on its own.

This also "fixed" the "Illegal state: Tx destroyed/inactive, writeable cursor still available" that was being logged.

[greenrobot]

Really looks weird. Thanks for tracking it down. Can you show how you defined the Data entity, especially id and parentId (, parent) properties including the annotations?

[PierceZ]

Sure:

@Entity
public class Data {

    @Id(assignable = true)
    private long id;

    @Index
    private long parentId;

...

There's a lot more fields, but they're just normal String, float, Date, etc. Most of them use @SerializedName("name_here") from Gson if that matters.

[PierceZ]

Here's also the OtherData class, which is the one that the IDs were referencing:

@Entity
public class OtherData {

    @Id(assignable = true)
    @SerializedName("my_id")
    long id;

    @SerializedName("my_name")
    String name;

    @Index
    long parentId;

    @Index
    long thirdDataId;

    @Relation
    ThirdData thirdData;

    /** Used to resolve relations */
    @Internal
    @Generated(98765)
    transient BoxStore __boxStore;

    @Internal
    @Generated(12345)
    transient ToOne<ThirdData> thirdDataToOne = new ToOne<>(this, OtherData_.thirdData);
...

[greenrobot]

Quick note: this probably won't make it into upcoming 0.9.13 release. We'll need to investigate how to reproduce this on our end with unit tests. Will begin this shortly after 0.9.13 release is done.

[PierceZ]

Ok. Let me know if you need more information.

[Martinsos]

Hi @PierceZ,
I am working on reproducing this bug and was hoping to ask you a few more things!

You mentioned ThirdTable has 1 to many relations with Data, and that OtherData has no relations with any of those two. In the code snippets that you provided, OtherData is in relation with ThirdData (which is ThirdTable I guess?), so I assume you by accident swapped OtherData and Data?

To make sure I got this right, here is what I did in order to replicate it and you tell me how close is that to your situation:

1. I have entity Data, which has id and indexed parent id.
2. I have entity OtherData, which has id, indexed parentId, indexed thirdDataId, and @relation to ThirdData.
3. I have entity ThirdData, which has id, indexed parentId, and @Backlink to the OtherData (it is a list).

Then I do this (starting with an empty database):

Data data = new Data();
data.setId(123);
data.setParentId(42);
dataBox.put(data);

data = new Data();
data.setId(1234);
data.setParentId(314);
dataBox.put(data);

ThirdData thirdData = new ThirdData();
thirdData.setId(1);
thirdData.setParentId(42);
thirdDataBox.put(thirdData);

boxStore.runInTx(new Runnable() {
    @Override
    public void run() {
        otherDataBox.query().equal(OtherData_.parentId, 42).build().remove();
    }
});

This should be the same as what you described, since it queries the entity which has no entries in objectbox and it has no relations to other entities involved.
However, I do not get the logs you were describing, so I assume there is some difference between this and your situation -> it would be great if you could help me figure out what is the difference!

By the way, what is the version of objectbox you were trying this on? Do you still get the problem in the new version?

[PierceZ]

I think you misread what I wrote. I said OtherData has no relation to Data, I never said it wasn't related to ThirdData (which it is, as you mentioned). I also made a mistake in trying to disguise my code, sorry ThirdData is the same as ThirdTable.

To clarify the relationships:

• ParentData owns all of these objects. I don't have any explicit relations set for that though, just the indexed long parentId field.
• Data is a child of ParentData (there's only 1 Data for each ParentData).
• OtherData I wrote in a confusing way. The parentId there refers to ParentData, and the thirdDataId refers to ThirdData. In reality, ThirdData has a list of OtherDatas (1 to many relationship). So ThirdData is really the parent of OtherData, and then ParentData is the parent of ThirdData.
• ThirdData has a list of OtherData (1 to many). ParentData is the parent of ThirdData.

Like I said, none of these objects have ObjectBox relations to ParentData, just indexed long parentId fields. The only actual ObjectBox relationship that exists is the 1 to many between ThirdData and OtherData (ThirdData has many OtherDatas). So that's probably not relevant, I just brought it up to clarify the data model. Hopefully that makes more sense now.

In regards to your replication:

1. This is correct.
2. This is correct.
3. This is correct.

The example you posted isn't quite what is happening. Here's what it looks like:

// parsed from GSON
ThirdData[] thirdDatas = ...;
OtherData[][] otherDatas = ...;
Parent parent = ...;
Data data = ...;
boxStore.runInTx(() -> {


                            thirdDataBox.query()
                                    .equal(ThirdData_.parentId, parent.getId())
                                    .build()
                                    .remove();
                            thirdDataBox.put(thirdDatas);

                            
                            otherDataBox.query().equal(OtherData_.parentId, parent.getId()).build().remove();
                            if (otherDatas != null) {
                                List<OtherData> otherDataList = new ArrayList<>();
                                for (int i = 0; i < otherDatas.length; i++) {
                                    OtherData[] otherDataArray = otherDatas[i];
                                    if (otherDataArray != null) {
                                        for (OtherData otherData : otherDataArray) {
                                            otherData.setThirdDataId(thirdDatas[i].getId());
                                            otherData.setParentId(parent.getId());
                                            otherDataList.add(otherData);
                                        }
                                    }
                                }
                                otherDataBox.put(otherDataList);
                            }

                            dataBox.query()
                                    .equal(Data_.parentId, parent.getId())
                                    .build()
                                    .remove();
                            if (data != null) {
                                data.setParentId(parent.getId());
                                dataBox.put(data);
                            }
                        });

I'm on version 0.9.12, I haven't changed since I posted this issue and haven't had a chance to try it on the new version. I just left it with my temporary fix I posted and it seemed ok. I figured since this issue wasn't resolved, the new version probably didn't fix it, but it's possible it did. I can try it later if you think it would have fixed it.

[PierceZ]

I just tested it on 0.9.13 and it's still the same issue.

[Martinsos]

@PierceZ, thank you for the quick answer!

In your first post, you said:

Also, I forgot to mention. ThirdTable has a 1 to many relation with Data, as in, ThirdTable has a list of Data, and Data has one ThirdTable. I'm using @relation and @Backlink.

That is what got me confused, since later you say that ThirdTable is in relation with OtherData.

Anyway, this helps a lot, you provided a lot of details and I managed to replicate it based on that!
Now we can get to fixing it -> we will keep you updated.

[PierceZ]

@Martinsos no problem.

I actually confused myself, because this issue is from about a month ago. I had forgotten what I was referencing in my actual code. After figuring it out again, I just copied the actual code and changed the references.

Anyway, I'm glad you could reproduce it.

[greenrobot]

@PierceZ We think all issues are resolved with 0.9.16-RC. Please let us know if we missed something!