bug(node): filtering doesn't work, no errors thrown

[flaviut]

LanceDB version

0.4.12

What happened?

import * as lancedb from 'vectordb'
import { Field, Schema as ArrowSchema } from 'apache-arrow'
import * as types from 'apache-arrow/type'
import { WriteMode } from 'vectordb'

const db = await lancedb.connect({
  // work around https://github.com/lancedb/lancedb/issues/1078
  awsCredentials: undefined,
  awsRegion: undefined,
  uri: './scratch',
})

const table = await db.createTable({
  name: 'table',
  schema: new ArrowSchema([
    new Field(
      'vector',
      new types.FixedSizeList(4, new Field('item', new types.Float32()))
    ),
    new Field('allowedGroups', new types.Utf8()),
  ]),
  writeOptions: {
    writeMode: WriteMode.Overwrite,
  }
})

await table.add([
  {
    vector: [1, 2, 3, 4],
    allowedGroups: '[[admin]]\t[[user]]'
  },
  {
    vector: [5, 6, 7, 8],
    allowedGroups: '[[user]]',
  },
])

console.log(await table.filter("allowedGroups = 'none'").execute())
console.log(await table.filter("allowedGroups LIKE '%[[admin]]%'").execute())
console.log(await table.filter("'123t3").execute())

[
  { vector: [ 1, 2, 3, 4 ], allowedGroups: '[[admin]]\t[[user]]' },
  { vector: [ 5, 6, 7, 8 ], allowedGroups: '[[user]]' }
]
[
  { vector: [ 1, 2, 3, 4 ], allowedGroups: '[[admin]]\t[[user]]' },
  { vector: [ 5, 6, 7, 8 ], allowedGroups: '[[user]]' }
]
[
  { vector: [ 1, 2, 3, 4 ], allowedGroups: '[[admin]]\t[[user]]' },
  { vector: [ 5, 6, 7, 8 ], allowedGroups: '[[user]]' }
]

I would expect this to have instead returned:

[ ]
[
  { vector: [ 1, 2, 3, 4 ], allowedGroups: '[[admin]]\t[[user]]' }
]
Error: Invalid filter expression
    at file:///...

this issue is particularly concerning, because this filtering may be used for access control. Applications that depend on that behavior may leak data to unauthorized users.

Are there known steps to reproduce?

No response

[wjones127]

Okay this was a weird one. The bug turns out to be we are silently ignoring filter expressions that have wrong capitalization that have any syntax errors. In your case, the syntax error was about the case sensitivity of your column names.

In SQL, the column references are not case sensitive. To make them case sensitive, you need to wrap in a backtick ("`"):

console.log(await table.filter("`allowedGroups` LIKE '%[[admin]]%'").execute())

Or you can use lower-case column names.

I'll see if I can make this return a proper error, as it is indeed not good that it would let data through.