Releases: pardnchiu/ToriiDB
v0.5.1
v0.5.0 -> v0.5.1
Summary
Add OS keychain as a secondary source for OPENAI_API_KEY. Lookup order is env (including .env) first, keychain fallback second — darwin uses security, linux uses secret-tool, other platforms read $HOME/.secrets.
翻譯
新增 OS keychain 作為 `OPENAI_API_KEY` 的備援來源。查找順序為 env(含 `.env`)優先、keychain 次之 — darwin 走 `security`、linux 走 `secret-tool`、其他平台讀 `$HOME/.secrets`。Changes
FEAT
- Add keychain fallback for
OPENAI_API_KEYinopenai.New()viagithub.com/pardnchiu/go-utils/filesystem/keychain; service nameToriiDB, fallback pathos.UserHomeDir()(falls back to.on lookup error). Env takes precedence — keychain is only consulted when env is empty.
翻譯
- 於
openai.New()以github.com/pardnchiu/go-utils/filesystem/keychain為OPENAI_API_KEY加入 keychain 後備;service 名稱為ToriiDB,fallback 路徑為os.UserHomeDir()(查詢失敗時退回.)。env 優先,僅在 env 為空時才讀取 keychain。
Files Changed
| File | Status | Tag |
|---|---|---|
core/openai/client.go |
Modified | FEAT |
Generated by SKILL
v0.5.0
v0.4.5 -> v0.5.0
Summary
Add semantic vector search backed by OpenAI embeddings. Vectors are inlined on each Entry, persisted through AOF, and queried via new VSEARCH / VSIM / VGET commands; a content-addressed embedding cache under __torii:embed:* de-duplicates API calls across identical texts.
翻譯
新增以 OpenAI embedding 為後端的語意向量搜尋。向量內嵌於每筆 `Entry`,透過 AOF 持久化,並經由新指令 `VSEARCH` / `VSIM` / `VGET` 查詢;以內容雜湊為 key 的 `__torii:embed:*` 快取可跨相同文字去重 API 呼叫。Changes
FEAT
- Add
core/openaipackage — standalonetext-embedding-3-smallclient withClient.Dim/Client.Modelaccessors, singleton initialization, and.envloading viagodotenv - Add
Entry.Vector []float32field with base64 little-endian float32 codec (core/store/vector.go);AOFRecord.Vector *stringis replayed and re-emitted during compaction for lossless restart - Add embedding cache under
__torii:embed:<sha256(model|dim|text)>(core/store/vcache.go);d != currentDimis treated as MISS, andKEYS/FIND/QUERY/VSEARCHskip the__torii:*namespace so internal keys never leak - Add
SET <key> <value> [NX|XX] [<seconds>] VECTOR— main key writes synchronously, a background goroutine attaches the embedding, andStore.Close()joinssync.WaitGroupbefore compacting AOF - Add
VSEARCH <text> [MATCH <pattern>] [LIMIT <n>]— top-K cosine similarity via linear scan + min-heap (defaultk = 10), with glob-based key filtering; query vector goes through the embedding cache so cache HIT avoids the API - Add
VSIM <key1> <key2>— cosine similarity between two stored vectors,(nil)on any missing vector, explicit dim-mismatch error - Add
VGET <key>— returns the stored vector as a JSON array (debug helper), returns a defensive copy to prevent callers from mutatingEntry.Vector
翻譯
- 新增
core/openai套件 — 獨立的text-embedding-3-smallclient,提供Client.Dim/Client.Model存取器、singleton 初始化,並透過godotenv載入.env - 新增
Entry.Vector []float32欄位搭配 base64 little-endian float32 編解碼(core/store/vector.go);AOFRecord.Vector *string會在 replay 與 compact 時還原/重新寫回,確保重啟無損 - 新增
__torii:embed:<sha256(model|dim|text)>的 embedding 快取(core/store/vcache.go);d != currentDim視為 MISS,KEYS/FIND/QUERY/VSEARCH全面跳過__torii:*前綴,內部 key 不會外露 - 新增
SET <key> <value> [NX|XX] [<seconds>] VECTOR— 主 key 同步寫入、背景 goroutine 補 embedding,Store.Close()以sync.WaitGroup等待排空後再做 AOF compaction - 新增
VSEARCH <text> [MATCH <pattern>] [LIMIT <n>]— 線性掃描 + min-heap 的 top-K cosine 相似度查詢(預設k = 10),支援 glob key 過濾;查詢向量走 embedding 快取,HIT 時不打 API - 新增
VSIM <key1> <key2>— 回傳兩個 key 的向量 cosine 相似度,任一缺向量回(nil),維度不符給明確錯誤訊息 - 新增
VGET <key>— 以 JSON 陣列輸出 stored vector(除錯用),回傳深拷貝避免外部修改汙染Entry.Vector
UPDATE
- Plain
Set()now clearsEntry.Vectoron overwrite — a key re-set without theVECTORflag drops the stale embedding since the underlying text has changed - Extend AOF with
addToAOFWithVector/writeAOFshared helpers;vectorfield is only emitted whenlen(vec) > 0to preserve backward compatibility with existing records
翻譯
- 純
Set()在覆寫時會清空Entry.Vector— 同一 key 若未帶VECTORflag 重新 SET,代表底層文字已變,舊 embedding 需一併作廢 - AOF 擴充出
addToAOFWithVector/writeAOF共用 helper;僅在len(vec) > 0才寫出vector欄位,保持與既有紀錄的向後相容
Files Changed
| File | Status | Tag |
|---|---|---|
core/openai/client.go |
Added | FEAT |
core/openai/client_test.go |
Added | FEAT |
core/store/vector.go |
Added | FEAT |
core/store/vcache.go |
Added | FEAT |
core/store/vsearch.go |
Added | FEAT |
core/store/vsim.go |
Added | FEAT |
core/store/set.go |
Modified | FEAT, UPDATE |
core/store/exec.go |
Modified | FEAT |
core/store/aof.go |
Modified | FEAT, UPDATE |
core/store/store.go |
Modified | FEAT |
core/store/find.go |
Modified | FEAT |
core/store/keys.go |
Modified | FEAT |
core/store/query.go |
Modified | FEAT |
README.md |
Modified | DOC |
doc/README.zh.md |
Added | DOC |
doc/architecture.md |
Added | DOC |
doc/architecture.zh.md |
Added | DOC |
doc/doc.md |
Added | DOC |
doc/doc.zh.md |
Added | DOC |
go.mod / go.sum |
Modified | CHORE |
Makefile |
Modified | CHORE |
.gitignore |
Modified | CHORE |
LICENSE |
Added | CHORE |
Generated by SKILL
v0.4.5
v0.4.4 -> v0.4.5
Summary
Split Entry.parseCached into a write-path parseAndCache and a read-path cached to eliminate a data race between concurrent RLock holders, and fix GetField which previously called the cache accessor with no lock held.
翻譯
將 `Entry.parseCached` 拆分為寫路徑 `parseAndCache` 與讀路徑 `cached`,消除並發 RLock 持有者之間的 data race,並修復 `GetField` 先前在完全無鎖狀態下呼叫快取存取器的問題。Changes
FIX
- Split
Entry.parseCached()intoparseAndCache()(write-path, mutatese.parsed, must be called under write lock or single-threaded init) andcached()(pure read, returns(nil, false)on miss; safe under RLock) - Fix
GetFielddata race: previously calledparseCached()afterGet()had already released the RLock — now holds the db RLock across lookup, cache read, andWalkKeysto prevent concurrent map mutation bySetField/IncrField/DelField
翻譯
- 將
Entry.parseCached()拆分為parseAndCache()(寫路徑,會寫入e.parsed,必須持寫鎖或單執行緒初始化時呼叫)與cached()(純讀,miss 時回傳(nil, false);RLock 下安全) - 修復
GetField的 data race:先前在Get()已釋放 RLock 後才呼叫parseCached()— 現在整個查找、快取讀取、WalkKeys流程皆在 db RLock 下進行,避免SetField/IncrField/DelField並發修改共用 map
REFACTOR
- Update all write-path callers (
Set,SetField,IncrField,DelField, AOF replay) to useparseAndCache() - Update read-path callers (
Query,GetField) to usecached(), relying on the invariant that every write path warmse.parsedbefore releasing the write lock
翻譯
- 將所有寫路徑(
Set、SetField、IncrField、DelField、AOF replay)改為呼叫parseAndCache() - 將讀路徑(
Query、GetField)改為呼叫cached(),仰賴「每個寫路徑在釋放寫鎖前已預熱e.parsed」此不變量
Files Changed
| File | Status | Tag |
|---|---|---|
core/store/set.go |
Modified | FIX, REFACTOR |
core/store/get.go |
Modified | FIX |
core/store/query.go |
Modified | REFACTOR |
core/store/setField.go |
Modified | REFACTOR |
core/store/incrField.go |
Modified | REFACTOR |
core/store/del.go |
Modified | REFACTOR |
core/store/aof.go |
Modified | REFACTOR |
CLAUDE.md |
Modified | DOC |
Generated by SKILL
v0.4.4
v0.4.3 -> v0.4.4
Summary
Cache parsed JSON objects in Entry to eliminate repeated json.Unmarshal on read/query hot paths, and make Entry.value unexported to enforce cache consistency at compile time.
翻譯
在 Entry 中快取已解析的 JSON 物件,消除讀取與查詢熱路徑的重複 json.Unmarshal,並將 Entry.value 私有化以在編譯期強制快取一致性。Changes
PERF
- Cache deserialized JSON in
Entry.parsedfield, populated onSet()and AOF replay viaparseCached() - Replace per-call
json.Unmarshalwith cached lookup inGetField,SetField,IncrField,DelField, andQuery - Trade-off: JSON entries use ~2x memory (raw string + parsed object), non-JSON entries have zero overhead
翻譯
- 在
Entry.parsed欄位快取已解析的 JSON,透過parseCached()在Set()和 AOF replay 時填充 - 將
GetField、SetField、IncrField、DelField、Query中的逐次json.Unmarshal替換為快取查詢 - 代價:JSON entry 記憶體約 2 倍(raw string + parsed object),非 JSON entry 零開銷
REFACTOR
- Make
Entry.Valueunexported (value) — all reads go throughValue(), all writes throughsetValue()orsetParsed() - Replace
json.Marshal(entry)withentry.JSON()across all file-write paths (8 call sites) - Remove
encoding/jsonimport fromget.go,query.go,incr.go,incrField.go,del.go,setField.go,ttl.go
翻譯
- 將
Entry.Value改為私有(value)— 所有讀取走Value(),所有寫入走setValue()或setParsed() - 所有寫檔路徑(8 處)的
json.Marshal(entry)替換為entry.JSON() - 移除
get.go、query.go、incr.go、incrField.go、del.go、setField.go、ttl.go中不再使用的encoding/jsonimport
Files Changed
| File | Status | Tag |
|---|---|---|
core/store/set.go |
Modified | PERF, REFACTOR |
core/store/get.go |
Modified | PERF, REFACTOR |
core/store/query.go |
Modified | PERF, REFACTOR |
core/store/setField.go |
Modified | PERF, REFACTOR |
core/store/incrField.go |
Modified | PERF, REFACTOR |
core/store/del.go |
Modified | PERF, REFACTOR |
core/store/incr.go |
Modified | REFACTOR |
core/store/ttl.go |
Modified | REFACTOR |
core/store/aof.go |
Modified | PERF, REFACTOR |
core/store/exec.go |
Modified | REFACTOR |
core/store/find.go |
Modified | REFACTOR |
README.md |
Modified | DOC |
CLAUDE.md |
Added | DOC |
core/store/query_test.go |
Added | TEST |
core/store/find_bench_test.go |
Added | TEST |
Generated by SKILL
v0.4.3
v0.4.2 -> v0.4.3
Summary
Switch AOF compaction trigger from line count to byte size with a 1MB floor, so compaction reflects actual disk waste and replay cost instead of record count.
翻譯
將 AOF 壓縮觸發條件從行數改為位元組大小並設定 1MB 門檻,讓壓縮時機貼近實際磁碟浪費與 replay 成本,而非單純的記錄筆數。Changes
PERF
- Replace line-count-based compaction trigger with byte-size tracking (
aofSize/aofSizeBaseline) - Set 1MB minimum size floor before compaction can fire, preventing thrashing on small databases
- Trigger compaction when
aofSize >= max(baseline, 1MB) * 2, giving compaction cost proper amortization - Update
replayAOFto accumulate byte size instead of line count, and reuse scanner bytes to avoid extra allocations
翻譯
- 以位元組追蹤 (
aofSize/aofSizeBaseline) 取代基於行數的壓縮觸發條件 - 設定 1MB 最小門檻,避免小型資料庫頻繁壓縮造成 thrashing
- 觸發條件改為
aofSize >= max(baseline, 1MB) * 2,讓壓縮成本得到合理攤銷 replayAOF改以位元組累計大小,並直接重用 scanner bytes 減少額外配置
Files Changed
| File | Status | Tag |
|---|---|---|
core/store/aof.go |
Modified | PERF |
core/store/store.go |
Modified | PERF |
Generated by SKILL
v0.4.2
v0.4.1 -> v0.4.2
Summary
Trigger AOF compaction inline on writes when the inflation ratio (AOF lines / live keys) reaches 2x, with a minimum live-key threshold to avoid wasteful rewrites on small databases.
翻譯
在寫入路徑中依膨脹比(AOF 行數 / 活躍 key 數)達到 2 倍時自動觸發 compaction,並設定最小活躍 key 門檻避免小型資料庫進行無意義的重寫。Changes
PERF
- Track AOF line count per database and trigger compaction inline in
addToAOFwhenaofLines >= live * 2andlive >= 1024, bounding AOF growth for long-running processes without requiring restart - Return line count from
replayAOFto initialize the counter on lazy DB load - Reset
aofLinesto the post-compaction line count so the inflation baseline adapts as the working set grows
翻譯
- 每個 database 追蹤 AOF 行數,在
addToAOF中於aofLines >= live * 2且live >= 1024時直接觸發 compaction,讓長時間運行的 process 不需重啟即可控制 AOF 成長 replayAOF回傳行數,用於 lazy DB 載入時初始化計數器- Compaction 完成後將
aofLines重設為壓縮後的行數,讓膨脹比的基準隨工作集自適應調整
Files Changed
| File | Status | Tag |
|---|---|---|
core/store/aof.go |
Modified | PERF |
core/store/store.go |
Modified | PERF |
Generated by SKILL
v0.4.1
v0.4.0 -> v0.4.1
Summary
Extract shared core struct to enable Session support, add lazy DB replay and parallel compaction for faster startup/shutdown, replace channel-based cancellation with context.Context, support custom storage path via New(path), and sync Expire/Persist changes to JSON cache files.
翻譯
抽離共用 core 結構體支援 Session、新增延遲載入與並行 compact 加速啟動與關閉、以 context.Context 取代 channel 取消機制、支援自訂儲存路徑、修正 Expire/Persist 未同步寫入 JSON 快取檔案。Changes
FEAT
- Add
Sessiontype sharingcoredb routing, allowing independent db index per session - Add custom storage path support via
New(path ...string)with directory validation
翻譯
- 新增
Session類型共用coredb 路由,每個 session 可獨立切換 db index - 透過
New(path ...string)支援自訂儲存路徑,含目錄驗證
FIX
- Sync Expire/ExpireAt/Persist changes to JSON cache files, matching Set/Del behavior
- Remove expired key's JSON cache file in
cleanExpired, consistent withDel
翻譯
- 修正 Expire/ExpireAt/Persist 變更未同步寫入 JSON 快取檔案,與 Set/Del 行為一致
- 清除過期 key 時同步刪除對應 JSON 快取檔案,與 Del 行為一致
REFACTOR
- Extract
corestruct withDB(),Current(),Select()methods shared byStoreandSession - Move all command methods from
*Storereceiver to*corereceiver
翻譯
- 抽離
core結構體,DB()、Current()、Select()供Store與Session共用 - 所有命令方法從
*Storereceiver 遷移至*corereceiver
PERF
- Add lazy DB replay via
sync.Once, only loading AOF on first access instead of all 16 at startup - Parallelize
compact()inClose()using goroutines, reducing shutdown time to slowest single DB - Replace unbuffered
cleanChchannel withcontext.Contextcancellation, eliminating potential deadlock on shutdown
翻譯
- 透過
sync.Once延遲載入 DB,首次存取才 replay AOF,啟動不再掃描全部 16 個 DB Close()中以 goroutine 並行執行compact(),關閉耗時降為最慢單一 DB- 以
context.Context取消機制取代無緩衝 channel,消除關閉時潛在 deadlock
Files Changed
| File | Status | Tag |
|---|---|---|
core/store/store.go |
Modified | FEAT |
core/store/ttl.go |
Modified | FIX |
core/store/exec.go |
Modified | REFACTOR |
core/store/get.go |
Modified | REFACTOR |
core/store/set.go |
Modified | REFACTOR |
core/store/del.go |
Modified | REFACTOR |
core/store/find.go |
Modified | REFACTOR |
core/store/query.go |
Modified | REFACTOR |
core/store/keys.go |
Modified | REFACTOR |
core/store/incr.go |
Modified | REFACTOR |
core/store/incrField.go |
Modified | REFACTOR |
core/store/setField.go |
Modified | REFACTOR |
cmd/test/main.go |
Modified | UPDATE |
Generated by SKILL
v0.4.0
v0.3.0 -> v0.4.0
Summary
Add NE (not equal) comparison operator, extract a dedicated filter package with an infix expression parser supporting AND/OR/NOT logical operators, and add concurrent slice scan for FIND/QUERY when data exceeds 1024 entries.
翻譯
新增 NE(不等於)比較運算子,提取獨立的 filter 套件支援 AND/OR/NOT 中綴表達式解析,並為 FIND/QUERY 新增超過 1024 筆時的切塊並發掃描。Changes
FEAT
- Add NE (not equal) comparison operator for FIND and QUERY commands
- Add infix expression parser supporting AND, OR, NOT logical operators and parenthesized grouping for QUERY command
- Add operator aliases: GTE/>=, LTE/<=, != for extended comparison syntax
翻譯
- 為 FIND 和 QUERY 命令新增 NE(不等於)比較運算子
- 為 QUERY 命令新增支援 AND、OR、NOT 邏輯運算子及括號分組的中綴表達式解析器
- 新增運算子別名:GTE/>=、LTE/<=、!= 擴展比較語法
PERF
- Add concurrent slice scan (sliceScan) for FIND and QUERY, auto-sharding into 1024-entry chunks when data exceeds threshold
翻譯
- 為 FIND 和 QUERY 新增切塊並發掃描(sliceScan),超過 1024 筆時自動分片並行處理
REFACTOR
- Extract filter package from find.go with Filter interface, Cond struct, and operator/matching logic
- Separate filter components into dedicated files: filter.go, operation.go, parser.go, extension.go
- Simplify Query method signature from field/op/value parameters to single Filter interface
翻譯
- 從 find.go 提取 filter 套件,包含 Filter 介面、Cond 結構及運算子/匹配邏輯
- 將 filter 元件分離為獨立檔案:filter.go、operation.go、parser.go、extension.go
- 簡化 Query 方法簽章,從 field/op/value 參數改為單一 Filter 介面
Files Changed
| File | Status | Tag |
|---|---|---|
core/store/filter/filter.go |
Added | FEAT |
core/store/filter/operation.go |
Added | FEAT |
core/store/filter/parser.go |
Added | FEAT |
core/store/filter/extension.go |
Added | FEAT |
core/store/exec.go |
Modified | REFACTOR |
core/store/find.go |
Modified | PERF |
core/store/query.go |
Modified | PERF |
Generated by SKILL
v0.3.0
v0.2.0 -> v0.3.0
Summary
Add FIND/QUERY commands with comparison operators and LIMIT support, extend dot-notation to EXIST/TYPE, and consolidate file structure.
翻譯
新增 FIND/QUERY 指令支援比較運算子與 LIMIT,擴展 dot-notation 至 EXIST/TYPE,並整併檔案結構Changes
FEAT
- Add FIND command with value search across all keys (EQ/GT/GE/LT/LE/LIKE)
- Add QUERY command for JSON sub-field conditional search with dot-notation
- Add LIMIT support for FIND and QUERY commands
- Add time-based sorting (UpdatedAt > CreatedAt, newest first) for FIND and QUERY results
- Add dot-notation support for EXIST and TYPE commands (ExistField, TypeField)
翻譯
- 新增 FIND 指令,支援全域值搜尋(EQ/GT/GE/LT/LE/LIKE)
- 新增 QUERY 指令,支援 JSON 子欄位條件查詢與 dot-notation
- 新增 FIND 與 QUERY 的 LIMIT 參數
- 新增 FIND 與 QUERY 依時間排序(UpdatedAt > CreatedAt,新到舊)
- 新增 EXIST 與 TYPE 的 dot-notation 巢狀欄位查詢
REFACTOR
- Extract WalkKeys to utils for shared use across GetField, IncrField, and Query
- Merge compact.go into aof.go and delField.go into del.go
- Merge getField.go into get.go
- Extract showList helper and parseLimit in exec.go
翻譯
- 抽離 WalkKeys 至 utils 供 GetField、IncrField、Query 共用
- 合併 compact.go 至 aof.go,delField.go 至 del.go
- 合併 getField.go 至 get.go
- 抽離 exec.go 中的 showList 與 parseLimit 輔助函式
Files Changed
| File | Status | Tag |
|---|---|---|
core/store/find.go |
Added | FEAT |
core/store/query.go |
Added | FEAT |
core/store/exec.go |
Modified | FEAT |
core/store/get.go |
Modified | FEAT |
core/store/del.go |
Modified | REFACTOR |
core/store/aof.go |
Modified | REFACTOR |
core/store/incrField.go |
Modified | REFACTOR |
core/utils/utils.go |
Modified | REFACTOR |
core/store/compact.go |
Deleted | REFACTOR |
core/store/delField.go |
Deleted | REFACTOR |
core/store/getField.go |
Deleted | REFACTOR |
README.md |
Modified | DOC |
Generated by SKILL
v0.2.0
v0.1.0 -> v0.2.0
Summary
Add document field operations (get/set/del/incr with dot-notation), KEYS glob matching, INCR numeric increment, AOF compaction on close, and shared utility package.
翻譯
新增文件欄位操作(dot-notation 的 get/set/del/incr)、KEYS glob 匹配、INCR 數值遞增、關閉時 AOF 壓縮,以及共用工具套件。Changes
FEAT
- Add dot-notation nested field access for GET/SET/DEL/INCR commands
- Add KEYS command with glob pattern matching
- Add INCR command supporting both standalone keys and nested JSON fields
- Add AOF compaction on close with atomic file writes
翻譯
- 新增 dot-notation 巢狀欄位存取,支援 GET/SET/DEL/INCR 指令
- 新增 KEYS 指令,支援 glob 模式匹配
- 新增 INCR 指令,支援獨立 key 與巢狀 JSON 欄位數值遞增
- 新增關閉時 AOF 壓縮,採用 atomic file write
FIX
- Fix AOF compaction skipped when no writes occur in current session
翻譯
- 修正當前 session 無寫入時 AOF compaction 被跳過的問題
REFACTOR
- Extract WriteFile, Atov, Vtoa, Vtof to core/utils package as shared utilities
翻譯
- 將 WriteFile、Atov、Vtoa、Vtof 抽至 core/utils 套件作為共用工具
Files Changed
| File | Status | Tag |
|---|---|---|
core/store/getField.go |
Added | FEAT |
core/store/setField.go |
Added | FEAT |
core/store/delField.go |
Added | FEAT |
core/store/incr.go |
Added | FEAT |
core/store/incrField.go |
Added | FEAT |
core/store/keys.go |
Added | FEAT |
core/store/compact.go |
Added | FEAT |
core/store/exec.go |
Modified | FEAT |
core/store/set.go |
Modified | REFACTOR |
core/store/store.go |
Modified | FIX |
core/utils/utils.go |
Added | REFACTOR |
cmd/test/main.go |
Modified | FIX |
Generated by SKILL