Record Type comparison not included as able to satisfy a sort

[alecgrieser]

If one has a record type with a primary key like:

concat(recordType(), field("id"))

The the query:

RecordQuery.newBuilder()
    .setRecordType(recordType)
    .setSort(concat(recordType(), field("id")))
    .build();

Can be satisfied by scanning over records with a primary key beginning with that record type's record type key. This plan would look like:

[IS recordType]

In the RecordQueryPlan's toString representation. However, the actual plan is:

Scan(<,>) | [IS recordType]

That is, it does a full scan over everything and then filters out the other types. Practically speaking, the result you could back would be identical as if the sort were not specified (in this case), but (a) the planner isn't guaranteeing it and (b) there is no way to specify reverse-ness.

---

Adding this test to RecordTypeKeyTest demonstrates this:

    @ParameterizedTest(name = "testSort [reverse = 0]")
    @BooleanSource
    public void testSort(boolean reverse) throws Exception {
        List<FDBStoredRecord<Message>> recs = saveSomeRecords(BASIC_HOOK);

        try (FDBRecordContext context = openContext()) {
            openSimpleRecordStore(context, BASIC_HOOK);

            RecordQuery query = RecordQuery.newBuilder()
                    .setRecordType("MySimpleRecord")
                    .setSort(recordStore.getRecordMetaData().getRecordType("MySimpleRecord").getPrimaryKey(), reverse)
                    // .setSort(recordStore.getRecordMetaData().getRecordType("MySimpleRecord").getPrimaryKey().getSubKey(1, 2), reverse)
                    .build();
            RecordQueryPlan plan = planner.plan(query);

            List<FDBStoredRecord<Message>> expectedResults = recs.subList(0, 2);
            if (reverse) {
                expectedResults = Lists.reverse(expectedResults);
            }

            assertEquals(expectedResults, recordStore.executeQuery(query)
                    .map(FDBQueriedRecord::getStoredRecord).asList().join());
            assertThat(plan, scan(bounds(hasTupleString("[IS MySimpleRecord"))));
        }
    }

Note that if one includes the record type key in the sort predicate, then it will do a full scan. If one omits the record type key (which it should be able to), then it throws an error.

[alecgrieser]

This behavior is seen with the current planner (i.e., RecordQueryPlanner). I'm not sure if the new planner has this behavior, though I actually think it doesn't support record type key expressions (yet), so it's somewhat of a moot point.

[alecgrieser]

Oh, interesting. I apparently put "fix" too close to "#744" in the PR title of #750, which accidentally closed this issue. Re-opening.

[rahul-nitkkr]

@alecgrieser is there any timeline or target release as to when this might be fixed ?

[alecgrieser]

There aren't really any updates for this, no. I don't believe anyone has yet started on it yet, with people currently all assigned to other issues.

[alecgrieser]

For future reference, this issue was also referenced in this forum post: https://forums.foundationdb.org/t/full-range-scan-performed-in-sort-when-not-required/1995