Create STREAM_UPDATE Index

[daiiz]

インデックス作成

$ sh create-stream-index.sh

完了！

---
createTime: '2023-06-07T08:25:30.020154Z'
displayName: Search index for my text embeddings
etag: AMEw9yOrLAoJ7yfEObYYe5b6rMqJEqtP8yH_zcDsFxxXg_LveK9xfTS5Helvqm7YOumL
indexStats:
  shardsCount: 1
  vectorsCount: '10'
indexUpdateMethod: STREAM_UPDATE
metadata:
  config:
    algorithmConfig:
      treeAhConfig: {}
    approximateNeighborsCount: 100
    dimensions: 1536
    shardSize: SHARD_SIZE_SMALL
metadataSchemaUri: gs://google-cloud-aiplatform/schema/matchingengine/metadata/nearest_neighbor_search_1.0.0.yaml
name: projects/349...locations/us-central1/indexes/541...
updateTime: '2023-06-07T09:10:37.723917Z'

[daiiz]

入力テキストのEmbeddingを得るスクリプト

$ node tools/gen-embedding.js こんにちは > "./sampledata/text/$(uuidgen).json"

[daiiz]

https://openai.com/blog/new-and-improved-embedding-model

Smaller embedding size. The new embeddings have only 1536 dimensions, one-eighth the size of davinci-001 embeddings, making the new embeddings more cost effective in working with vector databases.

[daiiz]

メモ

初期データを与えるGCSバケットではサブディレクトリを作ってはいけない。

{
  "error": {
    "code": 400,
    "message": "The given root directory `gs://my-demo-embbeddings2/index_dir3/` contains unexpected subdirectory (e.g. `text/03E64EE5-9BD1-4185-99F2-32334821E844.json`). No subdirectory is allowed except \"delete/\".",
    "status": "FAILED_PRECONDITION"
  }
}

[daiiz]

次: IndexEndpointにデプロイしたIndexを差し替えるには？

[daiiz]

メモ

着々と課金が始まっている