Full Text Seach: where clause ignored or unclear behavior

[pchalasani]

Assuming I've set up an fts_index... when doing this:

table.search(query).where(filter).limit(10).to_list()

I expect the FTS to be limited to records where filter is True. But it doesn't seem to be the case,
as in the first query below -- the filter seems to be ignored, or I don't understand the behavior, or I'm missing some setting.

I expected the first query below to return NO results. The second one seems to work as expected.

In [1]: import lancedb
   ...:
   ...: uri = "data/sample-lancedb"
   ...: db = lancedb.connect(uri)
   ...:
   ...: table = db.create_table("my_table",
   ...:             data=[{"vector": [3.1, 4.1], "text": "Frodo was a happy puppy"},
   ...:                   {"vector": [5.9, 26.5], "text": "There are several kittens playing"}])

In [2]: table.create_fts_index("text")

In [3]: table.search("Frodo").where("text like '%kitten%'").limit(10).to_list()
Out[3]:
[{'vector': [3.0999999046325684, 4.099999904632568],
  'text': 'Frodo was a happy puppy',
  'score': 0.6931471824645996}]

In [4]: table.search(None).where("text like '%kitten%'").limit(10).to_list()
Out[4]:
[{'vector': [5.900000095367432, 26.5],
  'text': 'There are several kittens playing'}]

[changhiskhan]

I expected the first query below to return NO results. The second one seems to work as expected.

If we get the FTS results then apply the filter on top, I think that should be good right?

[pchalasani]

If we get the FTS results then apply the filter on top, I think that should be good right?

that would be fine, yes. I think you mean something like this --

results = table.search("Frodo")
filtered = results.where("text like '%kitten%'")

It doesn't work yet, I assume you mean you're planning to implement it.

[changhiskhan]

It doesn't work yet, I assume you mean you're planning to implement it.
yeah just wanted to confirm this would solve your need before I go implement it

[changhiskhan]

@pchalasani you can take a look at the unit test in #739 to see usage. Lmk if this is what you were expecting.

[pchalasani]

Lmk if this is what you were expecting.

@changhiskhan yes it looks like the behavior I was expecting, for all results, the where condition should hold. In other words, the FTS should be restricted to records that match the where condition.
[edited]