Skip to content

Releases: pardnchiu/ToriiDB

v0.5.1

19 Apr 04:17

Choose a tag to compare

v0.5.0 -> v0.5.1

Summary

Add OS keychain as a secondary source for OPENAI_API_KEY. Lookup order is env (including .env) first, keychain fallback second — darwin uses security, linux uses secret-tool, other platforms read $HOME/.secrets.

翻譯 新增 OS keychain 作為 `OPENAI_API_KEY` 的備援來源。查找順序為 env(含 `.env`)優先、keychain 次之 — darwin 走 `security`、linux 走 `secret-tool`、其他平台讀 `$HOME/.secrets`。

Changes

FEAT

  • Add keychain fallback for OPENAI_API_KEY in openai.New() via github.com/pardnchiu/go-utils/filesystem/keychain; service name ToriiDB, fallback path os.UserHomeDir() (falls back to . on lookup error). Env takes precedence — keychain is only consulted when env is empty.
翻譯
  • openai.New()github.com/pardnchiu/go-utils/filesystem/keychainOPENAI_API_KEY 加入 keychain 後備;service 名稱為 ToriiDB,fallback 路徑為 os.UserHomeDir()(查詢失敗時退回 .)。env 優先,僅在 env 為空時才讀取 keychain。

Files Changed

File Status Tag
core/openai/client.go Modified FEAT

Generated by SKILL

v0.5.0

18 Apr 15:51

Choose a tag to compare

v0.4.5 -> v0.5.0

Summary

Add semantic vector search backed by OpenAI embeddings. Vectors are inlined on each Entry, persisted through AOF, and queried via new VSEARCH / VSIM / VGET commands; a content-addressed embedding cache under __torii:embed:* de-duplicates API calls across identical texts.

翻譯 新增以 OpenAI embedding 為後端的語意向量搜尋。向量內嵌於每筆 `Entry`,透過 AOF 持久化,並經由新指令 `VSEARCH` / `VSIM` / `VGET` 查詢;以內容雜湊為 key 的 `__torii:embed:*` 快取可跨相同文字去重 API 呼叫。

Changes

FEAT

  • Add core/openai package — standalone text-embedding-3-small client with Client.Dim / Client.Model accessors, singleton initialization, and .env loading via godotenv
  • Add Entry.Vector []float32 field with base64 little-endian float32 codec (core/store/vector.go); AOFRecord.Vector *string is replayed and re-emitted during compaction for lossless restart
  • Add embedding cache under __torii:embed:<sha256(model|dim|text)> (core/store/vcache.go); d != currentDim is treated as MISS, and KEYS / FIND / QUERY / VSEARCH skip the __torii:* namespace so internal keys never leak
  • Add SET <key> <value> [NX|XX] [<seconds>] VECTOR — main key writes synchronously, a background goroutine attaches the embedding, and Store.Close() joins sync.WaitGroup before compacting AOF
  • Add VSEARCH <text> [MATCH <pattern>] [LIMIT <n>] — top-K cosine similarity via linear scan + min-heap (default k = 10), with glob-based key filtering; query vector goes through the embedding cache so cache HIT avoids the API
  • Add VSIM <key1> <key2> — cosine similarity between two stored vectors, (nil) on any missing vector, explicit dim-mismatch error
  • Add VGET <key> — returns the stored vector as a JSON array (debug helper), returns a defensive copy to prevent callers from mutating Entry.Vector
翻譯
  • 新增 core/openai 套件 — 獨立的 text-embedding-3-small client,提供 Client.Dim / Client.Model 存取器、singleton 初始化,並透過 godotenv 載入 .env
  • 新增 Entry.Vector []float32 欄位搭配 base64 little-endian float32 編解碼(core/store/vector.go);AOFRecord.Vector *string 會在 replay 與 compact 時還原/重新寫回,確保重啟無損
  • 新增 __torii:embed:<sha256(model|dim|text)> 的 embedding 快取(core/store/vcache.go);d != currentDim 視為 MISS,KEYS / FIND / QUERY / VSEARCH 全面跳過 __torii:* 前綴,內部 key 不會外露
  • 新增 SET <key> <value> [NX|XX] [<seconds>] VECTOR — 主 key 同步寫入、背景 goroutine 補 embedding,Store.Close()sync.WaitGroup 等待排空後再做 AOF compaction
  • 新增 VSEARCH <text> [MATCH <pattern>] [LIMIT <n>] — 線性掃描 + min-heap 的 top-K cosine 相似度查詢(預設 k = 10),支援 glob key 過濾;查詢向量走 embedding 快取,HIT 時不打 API
  • 新增 VSIM <key1> <key2> — 回傳兩個 key 的向量 cosine 相似度,任一缺向量回 (nil),維度不符給明確錯誤訊息
  • 新增 VGET <key> — 以 JSON 陣列輸出 stored vector(除錯用),回傳深拷貝避免外部修改汙染 Entry.Vector

UPDATE

  • Plain Set() now clears Entry.Vector on overwrite — a key re-set without the VECTOR flag drops the stale embedding since the underlying text has changed
  • Extend AOF with addToAOFWithVector / writeAOF shared helpers; vector field is only emitted when len(vec) > 0 to preserve backward compatibility with existing records
翻譯
  • Set() 在覆寫時會清空 Entry.Vector — 同一 key 若未帶 VECTOR flag 重新 SET,代表底層文字已變,舊 embedding 需一併作廢
  • AOF 擴充出 addToAOFWithVector / writeAOF 共用 helper;僅在 len(vec) > 0 才寫出 vector 欄位,保持與既有紀錄的向後相容

Files Changed

File Status Tag
core/openai/client.go Added FEAT
core/openai/client_test.go Added FEAT
core/store/vector.go Added FEAT
core/store/vcache.go Added FEAT
core/store/vsearch.go Added FEAT
core/store/vsim.go Added FEAT
core/store/set.go Modified FEAT, UPDATE
core/store/exec.go Modified FEAT
core/store/aof.go Modified FEAT, UPDATE
core/store/store.go Modified FEAT
core/store/find.go Modified FEAT
core/store/keys.go Modified FEAT
core/store/query.go Modified FEAT
README.md Modified DOC
doc/README.zh.md Added DOC
doc/architecture.md Added DOC
doc/architecture.zh.md Added DOC
doc/doc.md Added DOC
doc/doc.zh.md Added DOC
go.mod / go.sum Modified CHORE
Makefile Modified CHORE
.gitignore Modified CHORE
LICENSE Added CHORE

Generated by SKILL

v0.4.5

16 Apr 03:35

Choose a tag to compare

v0.4.4 -> v0.4.5

Summary

Split Entry.parseCached into a write-path parseAndCache and a read-path cached to eliminate a data race between concurrent RLock holders, and fix GetField which previously called the cache accessor with no lock held.

翻譯 將 `Entry.parseCached` 拆分為寫路徑 `parseAndCache` 與讀路徑 `cached`,消除並發 RLock 持有者之間的 data race,並修復 `GetField` 先前在完全無鎖狀態下呼叫快取存取器的問題。

Changes

FIX

  • Split Entry.parseCached() into parseAndCache() (write-path, mutates e.parsed, must be called under write lock or single-threaded init) and cached() (pure read, returns (nil, false) on miss; safe under RLock)
  • Fix GetField data race: previously called parseCached() after Get() had already released the RLock — now holds the db RLock across lookup, cache read, and WalkKeys to prevent concurrent map mutation by SetField / IncrField / DelField
翻譯
  • Entry.parseCached() 拆分為 parseAndCache()(寫路徑,會寫入 e.parsed,必須持寫鎖或單執行緒初始化時呼叫)與 cached()(純讀,miss 時回傳 (nil, false);RLock 下安全)
  • 修復 GetField 的 data race:先前在 Get() 已釋放 RLock 後才呼叫 parseCached() — 現在整個查找、快取讀取、WalkKeys 流程皆在 db RLock 下進行,避免 SetField / IncrField / DelField 並發修改共用 map

REFACTOR

  • Update all write-path callers (Set, SetField, IncrField, DelField, AOF replay) to use parseAndCache()
  • Update read-path callers (Query, GetField) to use cached(), relying on the invariant that every write path warms e.parsed before releasing the write lock
翻譯
  • 將所有寫路徑(SetSetFieldIncrFieldDelField、AOF replay)改為呼叫 parseAndCache()
  • 將讀路徑(QueryGetField)改為呼叫 cached(),仰賴「每個寫路徑在釋放寫鎖前已預熱 e.parsed」此不變量

Files Changed

File Status Tag
core/store/set.go Modified FIX, REFACTOR
core/store/get.go Modified FIX
core/store/query.go Modified REFACTOR
core/store/setField.go Modified REFACTOR
core/store/incrField.go Modified REFACTOR
core/store/del.go Modified REFACTOR
core/store/aof.go Modified REFACTOR
CLAUDE.md Modified DOC

Generated by SKILL

v0.4.4

16 Apr 02:12

Choose a tag to compare

v0.4.3 -> v0.4.4

Summary

Cache parsed JSON objects in Entry to eliminate repeated json.Unmarshal on read/query hot paths, and make Entry.value unexported to enforce cache consistency at compile time.

翻譯 在 Entry 中快取已解析的 JSON 物件,消除讀取與查詢熱路徑的重複 json.Unmarshal,並將 Entry.value 私有化以在編譯期強制快取一致性。

Changes

PERF

  • Cache deserialized JSON in Entry.parsed field, populated on Set() and AOF replay via parseCached()
  • Replace per-call json.Unmarshal with cached lookup in GetField, SetField, IncrField, DelField, and Query
  • Trade-off: JSON entries use ~2x memory (raw string + parsed object), non-JSON entries have zero overhead
翻譯
  • Entry.parsed 欄位快取已解析的 JSON,透過 parseCached()Set() 和 AOF replay 時填充
  • GetFieldSetFieldIncrFieldDelFieldQuery 中的逐次 json.Unmarshal 替換為快取查詢
  • 代價:JSON entry 記憶體約 2 倍(raw string + parsed object),非 JSON entry 零開銷

REFACTOR

  • Make Entry.Value unexported (value) — all reads go through Value(), all writes through setValue() or setParsed()
  • Replace json.Marshal(entry) with entry.JSON() across all file-write paths (8 call sites)
  • Remove encoding/json import from get.go, query.go, incr.go, incrField.go, del.go, setField.go, ttl.go
翻譯
  • Entry.Value 改為私有(value)— 所有讀取走 Value(),所有寫入走 setValue()setParsed()
  • 所有寫檔路徑(8 處)的 json.Marshal(entry) 替換為 entry.JSON()
  • 移除 get.goquery.goincr.goincrField.godel.gosetField.gottl.go 中不再使用的 encoding/json import

Files Changed

File Status Tag
core/store/set.go Modified PERF, REFACTOR
core/store/get.go Modified PERF, REFACTOR
core/store/query.go Modified PERF, REFACTOR
core/store/setField.go Modified PERF, REFACTOR
core/store/incrField.go Modified PERF, REFACTOR
core/store/del.go Modified PERF, REFACTOR
core/store/incr.go Modified REFACTOR
core/store/ttl.go Modified REFACTOR
core/store/aof.go Modified PERF, REFACTOR
core/store/exec.go Modified REFACTOR
core/store/find.go Modified REFACTOR
README.md Modified DOC
CLAUDE.md Added DOC
core/store/query_test.go Added TEST
core/store/find_bench_test.go Added TEST

Generated by SKILL

v0.4.3

10 Apr 10:44

Choose a tag to compare

v0.4.2 -> v0.4.3

Summary

Switch AOF compaction trigger from line count to byte size with a 1MB floor, so compaction reflects actual disk waste and replay cost instead of record count.

翻譯 將 AOF 壓縮觸發條件從行數改為位元組大小並設定 1MB 門檻,讓壓縮時機貼近實際磁碟浪費與 replay 成本,而非單純的記錄筆數。

Changes

PERF

  • Replace line-count-based compaction trigger with byte-size tracking (aofSize / aofSizeBaseline)
  • Set 1MB minimum size floor before compaction can fire, preventing thrashing on small databases
  • Trigger compaction when aofSize >= max(baseline, 1MB) * 2, giving compaction cost proper amortization
  • Update replayAOF to accumulate byte size instead of line count, and reuse scanner bytes to avoid extra allocations
翻譯
  • 以位元組追蹤 (aofSize / aofSizeBaseline) 取代基於行數的壓縮觸發條件
  • 設定 1MB 最小門檻,避免小型資料庫頻繁壓縮造成 thrashing
  • 觸發條件改為 aofSize >= max(baseline, 1MB) * 2,讓壓縮成本得到合理攤銷
  • replayAOF 改以位元組累計大小,並直接重用 scanner bytes 減少額外配置

Files Changed

File Status Tag
core/store/aof.go Modified PERF
core/store/store.go Modified PERF

Generated by SKILL

v0.4.2

10 Apr 09:24

Choose a tag to compare

v0.4.1 -> v0.4.2

Summary

Trigger AOF compaction inline on writes when the inflation ratio (AOF lines / live keys) reaches 2x, with a minimum live-key threshold to avoid wasteful rewrites on small databases.

翻譯 在寫入路徑中依膨脹比(AOF 行數 / 活躍 key 數)達到 2 倍時自動觸發 compaction,並設定最小活躍 key 門檻避免小型資料庫進行無意義的重寫。

Changes

PERF

  • Track AOF line count per database and trigger compaction inline in addToAOF when aofLines >= live * 2 and live >= 1024, bounding AOF growth for long-running processes without requiring restart
  • Return line count from replayAOF to initialize the counter on lazy DB load
  • Reset aofLines to the post-compaction line count so the inflation baseline adapts as the working set grows
翻譯
  • 每個 database 追蹤 AOF 行數,在 addToAOF 中於 aofLines >= live * 2live >= 1024 時直接觸發 compaction,讓長時間運行的 process 不需重啟即可控制 AOF 成長
  • replayAOF 回傳行數,用於 lazy DB 載入時初始化計數器
  • Compaction 完成後將 aofLines 重設為壓縮後的行數,讓膨脹比的基準隨工作集自適應調整

Files Changed

File Status Tag
core/store/aof.go Modified PERF
core/store/store.go Modified PERF

Generated by SKILL

v0.4.1

09 Apr 18:39

Choose a tag to compare

v0.4.0 -> v0.4.1

Summary

Extract shared core struct to enable Session support, add lazy DB replay and parallel compaction for faster startup/shutdown, replace channel-based cancellation with context.Context, support custom storage path via New(path), and sync Expire/Persist changes to JSON cache files.

翻譯 抽離共用 core 結構體支援 Session、新增延遲載入與並行 compact 加速啟動與關閉、以 context.Context 取代 channel 取消機制、支援自訂儲存路徑、修正 Expire/Persist 未同步寫入 JSON 快取檔案。

Changes

FEAT

  • Add Session type sharing core db routing, allowing independent db index per session
  • Add custom storage path support via New(path ...string) with directory validation
翻譯
  • 新增 Session 類型共用 core db 路由,每個 session 可獨立切換 db index
  • 透過 New(path ...string) 支援自訂儲存路徑,含目錄驗證

FIX

  • Sync Expire/ExpireAt/Persist changes to JSON cache files, matching Set/Del behavior
  • Remove expired key's JSON cache file in cleanExpired, consistent with Del
翻譯
  • 修正 Expire/ExpireAt/Persist 變更未同步寫入 JSON 快取檔案,與 Set/Del 行為一致
  • 清除過期 key 時同步刪除對應 JSON 快取檔案,與 Del 行為一致

REFACTOR

  • Extract core struct with DB(), Current(), Select() methods shared by Store and Session
  • Move all command methods from *Store receiver to *core receiver
翻譯
  • 抽離 core 結構體,DB()Current()Select()StoreSession 共用
  • 所有命令方法從 *Store receiver 遷移至 *core receiver

PERF

  • Add lazy DB replay via sync.Once, only loading AOF on first access instead of all 16 at startup
  • Parallelize compact() in Close() using goroutines, reducing shutdown time to slowest single DB
  • Replace unbuffered cleanCh channel with context.Context cancellation, eliminating potential deadlock on shutdown
翻譯
  • 透過 sync.Once 延遲載入 DB,首次存取才 replay AOF,啟動不再掃描全部 16 個 DB
  • Close() 中以 goroutine 並行執行 compact(),關閉耗時降為最慢單一 DB
  • context.Context 取消機制取代無緩衝 channel,消除關閉時潛在 deadlock

Files Changed

File Status Tag
core/store/store.go Modified FEAT
core/store/ttl.go Modified FIX
core/store/exec.go Modified REFACTOR
core/store/get.go Modified REFACTOR
core/store/set.go Modified REFACTOR
core/store/del.go Modified REFACTOR
core/store/find.go Modified REFACTOR
core/store/query.go Modified REFACTOR
core/store/keys.go Modified REFACTOR
core/store/incr.go Modified REFACTOR
core/store/incrField.go Modified REFACTOR
core/store/setField.go Modified REFACTOR
cmd/test/main.go Modified UPDATE

Generated by SKILL

v0.4.0

09 Apr 08:24

Choose a tag to compare

v0.3.0 -> v0.4.0

Summary

Add NE (not equal) comparison operator, extract a dedicated filter package with an infix expression parser supporting AND/OR/NOT logical operators, and add concurrent slice scan for FIND/QUERY when data exceeds 1024 entries.

翻譯 新增 NE(不等於)比較運算子,提取獨立的 filter 套件支援 AND/OR/NOT 中綴表達式解析,並為 FIND/QUERY 新增超過 1024 筆時的切塊並發掃描。

Changes

FEAT

  • Add NE (not equal) comparison operator for FIND and QUERY commands
  • Add infix expression parser supporting AND, OR, NOT logical operators and parenthesized grouping for QUERY command
  • Add operator aliases: GTE/>=, LTE/<=, != for extended comparison syntax
翻譯
  • 為 FIND 和 QUERY 命令新增 NE(不等於)比較運算子
  • 為 QUERY 命令新增支援 AND、OR、NOT 邏輯運算子及括號分組的中綴表達式解析器
  • 新增運算子別名:GTE/>=、LTE/<=、!= 擴展比較語法

PERF

  • Add concurrent slice scan (sliceScan) for FIND and QUERY, auto-sharding into 1024-entry chunks when data exceeds threshold
翻譯
  • 為 FIND 和 QUERY 新增切塊並發掃描(sliceScan),超過 1024 筆時自動分片並行處理

REFACTOR

  • Extract filter package from find.go with Filter interface, Cond struct, and operator/matching logic
  • Separate filter components into dedicated files: filter.go, operation.go, parser.go, extension.go
  • Simplify Query method signature from field/op/value parameters to single Filter interface
翻譯
  • 從 find.go 提取 filter 套件,包含 Filter 介面、Cond 結構及運算子/匹配邏輯
  • 將 filter 元件分離為獨立檔案:filter.go、operation.go、parser.go、extension.go
  • 簡化 Query 方法簽章,從 field/op/value 參數改為單一 Filter 介面

Files Changed

File Status Tag
core/store/filter/filter.go Added FEAT
core/store/filter/operation.go Added FEAT
core/store/filter/parser.go Added FEAT
core/store/filter/extension.go Added FEAT
core/store/exec.go Modified REFACTOR
core/store/find.go Modified PERF
core/store/query.go Modified PERF

Generated by SKILL

v0.3.0

08 Apr 12:24

Choose a tag to compare

v0.2.0 -> v0.3.0

Summary

Add FIND/QUERY commands with comparison operators and LIMIT support, extend dot-notation to EXIST/TYPE, and consolidate file structure.

翻譯 新增 FIND/QUERY 指令支援比較運算子與 LIMIT,擴展 dot-notation 至 EXIST/TYPE,並整併檔案結構

Changes

FEAT

  • Add FIND command with value search across all keys (EQ/GT/GE/LT/LE/LIKE)
  • Add QUERY command for JSON sub-field conditional search with dot-notation
  • Add LIMIT support for FIND and QUERY commands
  • Add time-based sorting (UpdatedAt > CreatedAt, newest first) for FIND and QUERY results
  • Add dot-notation support for EXIST and TYPE commands (ExistField, TypeField)
翻譯
  • 新增 FIND 指令,支援全域值搜尋(EQ/GT/GE/LT/LE/LIKE)
  • 新增 QUERY 指令,支援 JSON 子欄位條件查詢與 dot-notation
  • 新增 FIND 與 QUERY 的 LIMIT 參數
  • 新增 FIND 與 QUERY 依時間排序(UpdatedAt > CreatedAt,新到舊)
  • 新增 EXIST 與 TYPE 的 dot-notation 巢狀欄位查詢

REFACTOR

  • Extract WalkKeys to utils for shared use across GetField, IncrField, and Query
  • Merge compact.go into aof.go and delField.go into del.go
  • Merge getField.go into get.go
  • Extract showList helper and parseLimit in exec.go
翻譯
  • 抽離 WalkKeys 至 utils 供 GetField、IncrField、Query 共用
  • 合併 compact.go 至 aof.go,delField.go 至 del.go
  • 合併 getField.go 至 get.go
  • 抽離 exec.go 中的 showList 與 parseLimit 輔助函式

Files Changed

File Status Tag
core/store/find.go Added FEAT
core/store/query.go Added FEAT
core/store/exec.go Modified FEAT
core/store/get.go Modified FEAT
core/store/del.go Modified REFACTOR
core/store/aof.go Modified REFACTOR
core/store/incrField.go Modified REFACTOR
core/utils/utils.go Modified REFACTOR
core/store/compact.go Deleted REFACTOR
core/store/delField.go Deleted REFACTOR
core/store/getField.go Deleted REFACTOR
README.md Modified DOC

Generated by SKILL

v0.2.0

08 Apr 07:40

Choose a tag to compare

v0.1.0 -> v0.2.0

Summary

Add document field operations (get/set/del/incr with dot-notation), KEYS glob matching, INCR numeric increment, AOF compaction on close, and shared utility package.

翻譯 新增文件欄位操作(dot-notation 的 get/set/del/incr)、KEYS glob 匹配、INCR 數值遞增、關閉時 AOF 壓縮,以及共用工具套件。

Changes

FEAT

  • Add dot-notation nested field access for GET/SET/DEL/INCR commands
  • Add KEYS command with glob pattern matching
  • Add INCR command supporting both standalone keys and nested JSON fields
  • Add AOF compaction on close with atomic file writes
翻譯
  • 新增 dot-notation 巢狀欄位存取,支援 GET/SET/DEL/INCR 指令
  • 新增 KEYS 指令,支援 glob 模式匹配
  • 新增 INCR 指令,支援獨立 key 與巢狀 JSON 欄位數值遞增
  • 新增關閉時 AOF 壓縮,採用 atomic file write

FIX

  • Fix AOF compaction skipped when no writes occur in current session
翻譯
  • 修正當前 session 無寫入時 AOF compaction 被跳過的問題

REFACTOR

  • Extract WriteFile, Atov, Vtoa, Vtof to core/utils package as shared utilities
翻譯
  • 將 WriteFile、Atov、Vtoa、Vtof 抽至 core/utils 套件作為共用工具

Files Changed

File Status Tag
core/store/getField.go Added FEAT
core/store/setField.go Added FEAT
core/store/delField.go Added FEAT
core/store/incr.go Added FEAT
core/store/incrField.go Added FEAT
core/store/keys.go Added FEAT
core/store/compact.go Added FEAT
core/store/exec.go Modified FEAT
core/store/set.go Modified REFACTOR
core/store/store.go Modified FIX
core/utils/utils.go Added REFACTOR
cmd/test/main.go Modified FIX

Generated by SKILL