Fix filter operators broken across spec and memory driver by Copilot · Pull Request #867 · objectstack-ai/objectstack

Copilot · 2026-03-03T09:17:40Z

Comprehensive fix for filter operators (contains, notContains, startsWith, endsWith, between, null) that were broken or inconsistent across the spec definitions and memory driver.

The critical root cause: parseFilterAST() converts AST operators to $-prefixed FilterCondition format ($contains, $startsWith, etc.), but the memory driver's convertToMongoQuery() passthrough branch sent these directly to Mingo — which doesn't understand them. All string/range/null filters returned empty results.

Changes

Spec — filter.zod.ts

StringOperatorSchema / FieldOperatorsSchema — add $notContains: z.string().optional()
VALID_AST_OPERATORS — add notcontains, not_contains
AST_OPERATOR_MAP — map both aliases → $notContains
FILTER_OPERATORS — add $notContains to the const tuple
Filter<T> — add $notContains to the generic type

Memory Driver — memory-driver.ts (Bug 1 root cause fix)

Replace convertToMongoQuery() passthrough (return filters) with normalizeFilterCondition() that recursively translates non-standard operators to Mingo-compatible format:
- $contains → $regex (case-insensitive)
- $notContains → $not: { $regex } (case-insensitive)
- $startsWith → $regex with ^ anchor
- $endsWith → $regex with $ anchor
- $between → $gte / $lte
- $null: true/false → $eq: null / $ne: null
Handle multiple regex-producing operators on the same field (e.g. { $startsWith: 'A', $endsWith: 'n' }) by promoting to $and so no regex silently overwrites another
Add notcontains / not_contains to convertConditionToMongo() for legacy AST format

Memory Matcher — memory-matcher.ts

Add $notContains operator to checkCondition()
Add $null operator to checkCondition()
Fix undefined value guard to exclude $null so $null: true correctly matches records where the field is missing

// Before: Mingo doesn't understand $contains → empty results
parseFilterAST(['name', 'contains', 'Evan'])
// → { name: { $contains: 'Evan' } }  ← passed to Mingo as-is, returns []

// After: normalized to Mingo-compatible format
// → { name: { $regex: /Evan/i } }  ← Mingo matches correctly

Tests

2 new spec tests for $notContains schema validation and AST parsing
16 new memory driver tests covering $contains, $notContains, $startsWith, $endsWith, $between, $null via FilterCondition object format, including $and/$or logical combinations, combined $startsWith + $endsWith on the same field, and $null: true on records with missing fields

Cross-repo note

This addresses bugs 1–3 from the audit (driver passthrough, missing notContains, startsWith/endsWith normalization) plus $between and $null handling. Bugs 4–7 (isEmpty rendering, between dual-input UI, in/notIn expansion, DataScopeManager case sensitivity) are frontend concerns that require coordinated changes in objectui.

Original prompt

This section details on the original issue you should resolve

<issue_title>[Bug] Filter operators (contains, notContains, isEmpty, between) broken or inconsistent across frontend and server</issue_title>
<issue_description>## Summary

The contains text search filter returns no results while equals works correctly (see screenshot). After a thorough code audit, 7 related filter issues were identified across the objectui and spec repos.

Root Cause Analysis

Bug 1 (Critical): contains filter returns empty results — Server-side double-conversion

Frontend sends AST: ['name', 'contains', 'Evan']
Server parseFilterAST() converts to { name: { $contains: 'Evan' } }
Driver receives { name: { $contains: 'Evan' } } but does not translate $contains → database-native (SQL/Mongo regex)
Results are always empty.

Other issues:

notContains operator missing from mapping tables
isEmpty/isNotEmpty only matches null, not empty strings
between filter renders only one input, needs min/max
Case sensitivity inconsistency (contains)
in/notIn operator expanded in frontend; drivers support natively
Same double-conversion bug for startsWith/endsWith

Proposed Fixes

Remove or refactor parseFilterAST() call and let drivers process raw AST arrays
Ensure drivers (esp. Memory & SQL) correctly convert $contains, $startsWith, $endsWith, $notContains to native DB queries
Update frontend FilterBuilder for correct input rendering & operator mapping

Reproduction

ListView/Contact, filter Name with Contains, value = 'Evan'
No items found; Equals with exact value works

Affected Files

objectui and spec repos, details in full analysis

Cross-Repo Impact

Requires coordinated changes in both objectui and spec repositories (especially data engine drivers).</issue_description>

Comments on the Issue (you are @copilot in this section)

@hotlong

Let me do a thorough audit of all filter-related code paths to identify any similar issues before creating a comprehensive issue.

好的，我已经完成了对两个仓库中所有过滤相关代码的全面排查。下面我来总结所有发现的问题，然后创建一个合并的任务。

🔍 全面排查结果

问题 1（原始报告）：`contains` 过滤器在服务端不工作

数据流完整追踪：

前端 FilterBuilder → 用户选择 Contains，生成 operator: 'contains' ✅
ListView mapOperator() → 'contains' → 'contains' ✅
ObjectStackAdapter → 序列化 ['name', 'contains', 'Evan'] 为 JSON 发送 ✅
服务端 parseFilterAST() → ['name', 'contains', 'Evan'] → { name: { $contains: 'Evan' } } ✅
🔴 toQueryAST() → 直接透传 options.filter 到 ast.where，此时 filter 已经被 parseFilterAST 转换成了 { name: { $contains: 'Evan' } } 这种 MongoDB 对象格式
🔴 Driver convertToMongoQuery() → 接收到的是 已转换的对象格式，走 MongoDB format passthrough 分支直接返回 { name: { $contains: 'Evan' } }
🔴 Mingo 查询引擎 → Mingo 不认识 $contains 操作符！Mingo 使用 $regex 做模糊匹配，所以查询返回空��果。

根因：parseFilterAST 将 AST contains 转换为 $contains，但 Mingo 不支持 $contains。Memory Driver 的 convertConditionToMongo 方法能正确处理 contains → $regex，但 passthrough 分支跳过了这个转换。

问题 2：`notContains` 操作符缺失 AST 映射

function mapSpecOperator(op: string): string {
  switch (op) {
    case 'contains': return 'contains';
    case 'notContains': return 'notcontains';
    // ...
  }
}

notcontains 在 parseFilterAST 的 AST_OPERATOR_MAP 中 没有映射：

const AST_OPERATOR_MAP: Record<string, string> = {
  'contains': '$contains',
  'like': '$contains',
  // ❌ 缺少 'notcontains' 映射！
};

notcontains 会走 fallback 变成 $notcontains，而 Driver 同样不认识。

问题 3：`startswith` 操作符大小写不一致

前端 mapOperator 中没有 startsWith/endsWith 映射（FilterBuilder 未暴露这些操作符给用户），但 VALID_AST_OPERATORS 中同时存在 startswith 和 starts_with：

export const VALID_AST_OPERATORS = new Set([
  'contains', 'like',
  'startswith', 'starts_with',
  'endswith', 'ends_with',
]);

而 Memory Driver convertConditionToMongo 处理了 startswith/starts_with，但 AST_OPERATOR_MAP 映射到的是 $startsWith（驼峰），如果 filter 对象格式被 passthrough，Mingo 同样不认识。

问题 4：`isEmpty`/`isNotEmpty` 的 null 转换可能导致意外行为

function convertFilterGroupToAST(group: ...

</details>



<!-- START COPILOT CODING AGENT SUFFIX -->

- Fixes objectstack-ai/spec#866

<!-- START COPILOT CODING AGENT TIPS -->
---

🔒 GitHub Advanced Security automatically protects Copilot coding agent pull requests. You can protect all pull requests by enabling Advanced Security for your repositories. [Learn more about Advanced Security.](https://gh.io/cca-advanced-security)

vercel · 2026-03-03T09:17:44Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
objectstack-play	Ready	Preview, Comment	Mar 3, 2026 10:25am
spec	Ready	Preview, Comment	Mar 3, 2026 10:25am

Add $notContains to StringOperatorSchema, FieldOperatorsSchema, Filter<T>, FILTER_OPERATORS, VALID_AST_OPERATORS, and AST_OPERATOR_MAP with both 'notcontains' and 'not_contains' AST aliases. Fixes #34 Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

hotlong · 2026-03-03T09:31:02Z

@copilot 仔细评估是否已经完成任务中要求的所有开发要排查所有的过滤问题

…$notContains/$startsWith/$endsWith/$between/$null to Mingo-compatible format The root cause of Bug 1 (contains filter returns empty results): parseFilterAST() converts AST to FilterCondition with $-prefixed operators ($contains, $startsWith, etc.) but the memory driver passes them through to Mingo which doesn't understand these non-standard operators. Changes: - memory-driver.ts: Add normalizeFilterCondition/normalizeFieldOperators to convert $contains→$regex, $notContains→$not/$regex, $startsWith→$regex, $endsWith→$regex, $between→$gte/$lte, $null→$eq/$ne null - memory-driver.ts: Add notcontains/not_contains to convertConditionToMongo - memory-matcher.ts: Add $notContains and $null operator handling - memory-driver.test.ts: Add 12 tests for FilterCondition object format Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

… clarity Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

Copilot · 2026-03-03T09:45:38Z

@copilot 仔细评估是否已经完成任务中要求的所有开发要排查所有的过滤问题

After a thorough audit of all filter-related code paths across the monorepo, I've expanded the fix to cover all spec-repo-side issues from the 7-bug analysis:

Commit fdb9155 addresses the critical root cause (Bug 1) plus related gaps:

Bug 1 (Critical) — $contains/$startsWith/$endsWith passthrough: convertToMongoQuery() passed FilterCondition objects directly to Mingo, which doesn't recognize these operators. Added normalizeFilterCondition() to translate them to $regex.
Bug 2 — $notContains missing: Added to spec schemas, AST maps, memory driver, and memory matcher.
Bug 3 — startswith case: Already handled by existing AST_OPERATOR_MAP. Memory driver now also normalizes $startsWith→$regex in the passthrough path.
$null operator — Added to memory-matcher and to the normalization pipeline ($null: true → $eq: null).
$between operator — Added normalization ($between: [min, max] → $gte/$lte).
notcontains/not_contains — Added as separate case in convertConditionToMongo() for legacy AST format.

12 new driver-level tests confirm all operator formats work end-to-end.

Bugs 4-7 (isEmpty rendering, between UI, in/notIn expansion, DataScopeManager case sensitivity) are frontend concerns in the objectui repo and require a cross-repo issue.

Copilot

Pull request overview

This PR fixes critical filter operator bugs in the ObjectStack spec and memory driver. The root cause was that parseFilterAST() converts AST operators to $-prefixed FilterCondition format (e.g., $contains), but the memory driver's convertToMongoQuery() previously had a passthrough branch that sent these directly to Mingo, which doesn't understand non-standard operators like $contains, resulting in empty results for all string/range/null filters.

Changes:

filter.zod.ts in packages/spec: Adds $notContains to StringOperatorSchema, FieldOperatorsSchema, Filter<T> type, VALID_AST_OPERATORS, AST_OPERATOR_MAP, and FILTER_OPERATORS
memory-driver.ts: Replaces the Mingo passthrough with normalizeFilterCondition() and normalizeFieldOperators() that translate $contains, $notContains, $startsWith, $endsWith, $between, and $null to Mingo-native operators; also adds notcontains/not_contains to convertConditionToMongo()
memory-matcher.ts: Adds $notContains and $null cases to checkCondition(); adds corresponding tests across filter.test.ts and memory-driver.test.ts

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`packages/spec/src/data/filter.zod.ts`	Adds `$notContains` operator to all relevant schemas, types, maps and constants
`packages/spec/src/data/filter.test.ts`	Adds tests for `$notContains` schema validation, AST parsing, and `isFilterAST` detection
`packages/plugins/driver-memory/src/memory-driver.ts`	Replaces Mingo passthrough with `normalizeFilterCondition` + `normalizeFieldOperators`; adds `notcontains`/`not_contains` to `convertConditionToMongo`
`packages/plugins/driver-memory/src/memory-matcher.ts`	Adds `$notContains` and `$null` cases to `checkCondition`
`packages/plugins/driver-memory/src/memory-driver.test.ts`	Adds 12 integration tests covering all new operators in FilterCondition object format

Copilot · 2026-03-03T09:55:41Z

+        case '$contains':
+          result.$regex = new RegExp(this.escapeRegex(val), 'i');
+          break;
+        case '$notContains':
+          result.$not = { $regex: new RegExp(this.escapeRegex(val), 'i') };
+          break;
+        case '$startsWith':
+          result.$regex = new RegExp(`^${this.escapeRegex(val)}`, 'i');
+          break;
+        case '$endsWith':
+          result.$regex = new RegExp(`${this.escapeRegex(val)}$`, 'i');
+          break;


In normalizeFieldOperators, the cases for $contains, $startsWith, and $endsWith all write to result.$regex. If a filter has more than one of these operators on the same field (e.g., { name: { $startsWith: 'A', $endsWith: 'n' } }), the second operator silently overwrites the $regex set by the first. Only the last processed operator's regex pattern is applied, and the earlier constraints are lost. To fix this, each of these operators should be combined into a single regex (e.g., concatenating both ^ and $ anchors into one expression), or this limitation should be documented.

Copilot · 2026-03-03T09:55:41Z

+            case '$null':
+                // $null: true → value must be null/undefined; $null: false → value must not be null/undefined
+                if (target === true && value != null) return false;
+                if (target === false && value == null) return false;
+                break;


In checkCondition, there is an early-return guard at line 106: if (value === undefined && op !== '$exists' && op !== '$ne') { return false; }. Because $null is not excluded from this guard, when a field is missing (value === undefined) and the operator is $null: true, the guard fires and returns false — meaning $null: true fails to match fields that don't exist on a record. Since $null: true is semantically "field is null or missing", the guard should also exclude $null (i.e., the condition should be op !== '$exists' && op !== '$ne' && op !== '$null') so that the $null case in the switch can evaluate correctly for undefined values.

Copilot · 2026-03-03T09:55:42Z

+    it('should handle $contains inside $or', async () => {
+      const results = await driver.find(testTable, {
+        object: testTable,
+        where: { $or: [{ name: { $startsWith: 'Al' } }, { name: { $startsWith: 'Ev' } }] },
+      });
+      expect(results).toHaveLength(2);
+      expect(results.map((r: any) => r.name).sort()).toEqual(['Alice Johnson', 'Evan Davis']);
+    });


The test at line 680 is named 'should handle $contains inside $or', but the test body actually uses $startsWith inside the $or clause — not $contains. The description should be updated to 'should handle $startsWith inside $or' to accurately reflect what is being tested.

hotlong · 2026-03-03T10:00:36Z

@copilot fix all

…st name 1. normalizeFieldOperators: collect regex-producing operators ($contains, $startsWith, $endsWith) into an array; when multiple exist on the same field, promote to $and so no regex silently overwrites another. 2. memory-matcher checkCondition: exclude $null from the undefined early- return guard so $null: true correctly matches missing fields. 3. Rename test from "$contains inside $or" to "$startsWith inside $or" to match actual test body. 4. Add tests: combined $startsWith + $endsWith on same field, $null: true on records with missing fields. Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

Copilot · 2026-03-03T10:16:28Z

@copilot fix all

Fixed all three review issues in 41b6831:

Multi-regex overwrite — normalizeFieldOperators now collects regex-producing ops ($contains, $startsWith, $endsWith) into an array. Single regex is inlined; multiple are promoted to $and via _multiRegex sentinel so no regex silently overwrites another. Added test: $startsWith: 'A' + $endsWith: 'son' correctly returns only "Alice Johnson".
$null undefined guard — Excluded $null from the early-return at line 106 of memory-matcher.ts, so $null: true now correctly matches records where the field is missing entirely. Added test confirming this.
Test name — Renamed 'should handle $contains inside $or' → 'should handle $startsWith inside $or' to match actual test body.

Initial plan

ec6a21c

Copilot AI assigned Copilot and hotlong Mar 3, 2026

Copilot started work on behalf of hotlong March 3, 2026 09:17 View session

vercel Bot deployed to Preview – objectstack-play March 3, 2026 09:25 View deployment

Copilot AI changed the title ~~[WIP] Fix broken filter operators in frontend and server~~ Add missing $notContains operator to filter DSL Mar 3, 2026

Copilot finished work on behalf of hotlong March 3, 2026 09:27

vercel Bot deployed to Preview – spec March 3, 2026 09:28 View deployment

Copilot started work on behalf of hotlong March 3, 2026 09:31 View session

Copilot AI and others added 2 commits March 3, 2026 09:41

refactor: separate contains/notcontains into distinct case blocks for…

07b0307

… clarity Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

Copilot AI changed the title ~~Add missing $notContains operator to filter DSL~~ Fix filter operators broken across spec and memory driver Mar 3, 2026

Copilot finished work on behalf of hotlong March 3, 2026 09:46

hotlong marked this pull request as ready for review March 3, 2026 09:48

Copilot AI review requested due to automatic review settings March 3, 2026 09:48

Copilot started reviewing on behalf of hotlong March 3, 2026 09:48 View session

Copilot AI reviewed Mar 3, 2026

View reviewed changes

vercel Bot deployed to Preview – spec March 3, 2026 10:00 View deployment

Copilot started work on behalf of hotlong March 3, 2026 10:00 View session

vercel Bot deployed to Preview – objectstack-play March 3, 2026 10:01 View deployment

vercel Bot deployed to Preview – objectstack-play March 3, 2026 10:13 View deployment

refactor: rename __$andRegex sentinel to _multiRegex for clarity

f661b46

Co-authored-by: hotlong <50353452+hotlong@users.noreply.github.com>

Copilot finished work on behalf of hotlong March 3, 2026 10:17

hotlong merged commit 75d08cd into main Mar 3, 2026
1 of 3 checks passed

vercel Bot deployed to Preview – objectstack-play March 3, 2026 10:18 View deployment

vercel Bot deployed to Preview – spec March 3, 2026 10:25 View deployment

Copilot AI mentioned this pull request Mar 3, 2026

Add changeset for filter operators patch #868

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix filter operators broken across spec and memory driver#867

Fix filter operators broken across spec and memory driver#867
hotlong merged 6 commits into
mainfrom
copilot/fix-filter-operators-issue

Copilot AI commented Mar 3, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Mar 3, 2026 •

edited

Loading

Uh oh!

hotlong commented Mar 3, 2026

Uh oh!

Copilot AI commented Mar 3, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 3, 2026

Uh oh!

Copilot AI Mar 3, 2026

Uh oh!

Copilot AI Mar 3, 2026

Uh oh!

hotlong commented Mar 3, 2026

Uh oh!

Copilot AI commented Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

Copilot AI commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Tests

Cross-repo note

Root Cause Analysis

Proposed Fixes

Reproduction

Affected Files

Cross-Repo Impact

Comments on the Issue (you are @copilot in this section)

🔍 全面排查结果

问题 1（原始报告）：contains 过滤器在服务端不工作

问题 2：notContains 操作符缺失 AST 映射

问题 3：startswith 操作符大小写不一致

问题 4：isEmpty/isNotEmpty 的 null 转换可能导致意外行为

Uh oh!

vercel Bot commented Mar 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hotlong commented Mar 3, 2026

Uh oh!

Copilot AI commented Mar 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

hotlong commented Mar 3, 2026

Uh oh!

Copilot AI commented Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Mar 3, 2026 •

edited

Loading

问题 1（原始报告）：`contains` 过滤器在服务端不工作

问题 2：`notContains` 操作符缺失 AST 映射

问题 3：`startswith` 操作符大小写不一致

问题 4：`isEmpty`/`isNotEmpty` 的 null 转换可能导致意外行为

vercel Bot commented Mar 3, 2026 •

edited

Loading