Add vector_search_indexes resource (direct engine) #5123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open

janniklasrose wants to merge 47 commits into main from janniklasrose/vs-index

NEXT_CHANGELOG.md

-Original file line number
+Diff line change
@@ Expand Up / @@ -9,4 +9,6 @@ @@
     ### Bundles
     * Make sure warnings asking for approval are understood by agents ([#5239](https://github.com/databricks/cli/pull/5239))
+    * Added `vector_search_indexes` as a bundle resource (direct engine only). Supports UC grants and prompts for confirmation on recreate or delete since both are destructive ([#5123](https://github.com/databricks/cli/pull/5123)).
     ### Dependency updates

acceptance/bundle/deployment/bind/vector_search_index/databricks.yml.tmpl

-Original file line number
+Diff line change
@@ -0,0 +1,15 @@
+    bundle:
+      name: test-bundle-$UNIQUE_NAME
+    sync:
+      paths: []
+    resources:
+      vector_search_indexes:
+        index1:
+          name: $INDEX_NAME
+          endpoint_name: $ENDPOINT_NAME
+          primary_key: id
+          index_type: DIRECT_ACCESS
+          direct_access_index_spec:
+            schema_json: '{"columns":[{"name":"id","type":"integer"}]}'

acceptance/bundle/deployment/bind/vector_search_index/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

acceptance/bundle/deployment/bind/vector_search_index/output.txt

-Original file line number
+Diff line change
@@ -0,0 +1,54 @@
+    >>> [CLI] vector-search-endpoints create-endpoint test-vs-endpoint-[UNIQUE_NAME] STANDARD
+    {
+      "name": "test-vs-endpoint-[UNIQUE_NAME]",
+      "endpoint_type": "STANDARD"
+    }
+    >>> [CLI] vector-search-indexes create-index --json {"name":"main.default.test_vs_index_[UNIQUE_NAME]","endpoint_name":"test-vs-endpoint-[UNIQUE_NAME]","primary_key":"id","index_type":"DIRECT_ACCESS","direct_access_index_spec":{"schema_json":"{\"columns\":[{\"name\":\"id\",\"type\":\"integer\"}]}"}}
+    {
+      "name": "main.default.test_vs_index_[UNIQUE_NAME]",
+      "endpoint_name": "test-vs-endpoint-[UNIQUE_NAME]",
+      "index_type": "DIRECT_ACCESS",
+      "primary_key": "id"
+    }
+    >>> [CLI] bundle deployment bind index1 main.default.test_vs_index_[UNIQUE_NAME] --auto-approve
+    Updating deployment state...
+    Successfully bound vector_search_index with an id 'main.default.test_vs_index_[UNIQUE_NAME]'
+    Run 'bundle deploy' to deploy changes to your workspace
+    >>> [CLI] bundle deploy --auto-approve
+    Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
+    Deploying resources...
+    Updating deployment state...
+    Deployment complete!
+    >>> [CLI] vector-search-indexes get-index main.default.test_vs_index_[UNIQUE_NAME]
+    {
+      "name": "main.default.test_vs_index_[UNIQUE_NAME]",
+      "endpoint_name": "test-vs-endpoint-[UNIQUE_NAME]",
+      "index_type": "DIRECT_ACCESS",
+      "primary_key": "id"
+    }
+    >>> [CLI] bundle deployment unbind index1
+    Updating deployment state...
+    >>> [CLI] bundle destroy --auto-approve
+    All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default
+    Deleting files...
+    Destroy complete!
+    >>> [CLI] vector-search-indexes get-index main.default.test_vs_index_[UNIQUE_NAME]
+    {
+      "name": "main.default.test_vs_index_[UNIQUE_NAME]",
+      "endpoint_name": "test-vs-endpoint-[UNIQUE_NAME]",
+      "index_type": "DIRECT_ACCESS",
+      "primary_key": "id"
+    }
+    >>> [CLI] vector-search-indexes delete-index main.default.test_vs_index_[UNIQUE_NAME]
+    >>> [CLI] vector-search-endpoints delete-endpoint test-vs-endpoint-[UNIQUE_NAME]

acceptance/bundle/deployment/bind/vector_search_index/script

-Original file line number
+Diff line change
@@ -0,0 +1,27 @@
+    ENDPOINT_NAME="test-vs-endpoint-$UNIQUE_NAME"
+    INDEX_NAME="main.default.test_vs_index_$UNIQUE_NAME"
+    export ENDPOINT_NAME INDEX_NAME
+    envsubst < databricks.yml.tmpl > databricks.yml
+    cleanup() {
+        trace $CLI vector-search-indexes delete-index "${INDEX_NAME}"
+        trace $CLI vector-search-endpoints delete-endpoint "${ENDPOINT_NAME}"
+    }
+    trap cleanup EXIT
+    trace $CLI vector-search-endpoints create-endpoint "${ENDPOINT_NAME}" STANDARD | jq '{name, endpoint_type}'
+    trace $CLI vector-search-indexes create-index --json "{\"name\":\"${INDEX_NAME}\",\"endpoint_name\":\"${ENDPOINT_NAME}\",\"primary_key\":\"id\",\"index_type\":\"DIRECT_ACCESS\",\"direct_access_index_spec\":{\"schema_json\":\"{\\\"columns\\\":[{\\\"name\\\":\\\"id\\\",\\\"type\\\":\\\"integer\\\"}]}\"}}" | jq '{name, endpoint_name, index_type, primary_key}'
+    trace $CLI bundle deployment bind index1 "${INDEX_NAME}" --auto-approve
+    trace $CLI bundle deploy --auto-approve
+    trace $CLI vector-search-indexes get-index "${INDEX_NAME}" | jq '{name, endpoint_name, index_type, primary_key}'
+    trace $CLI bundle deployment unbind index1
+    trace $CLI bundle destroy --auto-approve
+    # Read the pre-defined index again (expecting it still exists and is not deleted):
+    trace $CLI vector-search-indexes get-index "${INDEX_NAME}" | jq '{name, endpoint_name, index_type, primary_key}'

acceptance/bundle/deployment/bind/vector_search_index/test.toml

-Original file line number
+Diff line change
@@ -0,0 +1,11 @@
+    Local = true
+    Cloud = true
+    RequiresUnityCatalog = true
+    Ignore = [
+        ".databricks",
+        "databricks.yml",
+    ]
+    [EnvMatrix]
+      DATABRICKS_BUNDLE_ENGINE = ["direct"]

acceptance/bundle/invariant/configs/vector_search_index.yml.tmpl

-Original file line number
+Diff line change
@@ -0,0 +1,17 @@
+    bundle:
+      name: test-bundle-$UNIQUE_NAME
+    resources:
+      vector_search_endpoints:
+        bar:
+          name: test-endpoint-$UNIQUE_NAME
+          endpoint_type: STANDARD
+      vector_search_indexes:
+        foo:
+          name: test-index-$UNIQUE_NAME
+          endpoint_name: ${resources.vector_search_endpoints.bar.name}
+          primary_key: id
+          index_type: DELTA_SYNC
+          delta_sync_index_spec:
+            source_table: main.default.source_$UNIQUE_NAME
+            pipeline_type: TRIGGERED

acceptance/bundle/invariant/continue_293/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

acceptance/bundle/invariant/migrate/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

acceptance/bundle/invariant/migrate/test.toml

-Original file line number
+Diff line change
@@ -1,5 +1,6 @@
-    # vector_search_endpoints has no terraform converter
+    # vector_search_endpoints and vector_search_indexes have no terraform converter
     EnvMatrixExclude.no_vector_search_endpoint = ["INPUT_CONFIG=vector_search_endpoint.yml.tmpl"]
+    EnvMatrixExclude.no_vector_search_index = ["INPUT_CONFIG=vector_search_index.yml.tmpl"]
     # Error: Catalog resources are only supported with direct deployment mode
     EnvMatrixExclude.no_catalog = ["INPUT_CONFIG=catalog.yml.tmpl"]
@@ Expand Down @@

acceptance/bundle/invariant/no_drift/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

acceptance/bundle/invariant/test.toml

-Original file line number
+Diff line change
@@ Expand Up / @@ -56,6 +56,7 @@ EnvMatrix.INPUT_CONFIG = [ @@
         "sql_warehouse.yml.tmpl",
         "synced_database_table.yml.tmpl",
         "vector_search_endpoint.yml.tmpl",
+        "vector_search_index.yml.tmpl",
         "volume.yml.tmpl",
         "volume_external.yml.tmpl",
     ]
@@ Expand Down @@

acceptance/bundle/refschema/out.fields.txt

-Original file line number
+Diff line change
@@ Expand Up @@
     resources.vector_search_endpoints.*.permissions[*].level	iam.PermissionLevel	ALL
     resources.vector_search_endpoints.*.permissions[*].service_principal_name	string	ALL
     resources.vector_search_endpoints.*.permissions[*].user_name	string	ALL
+    resources.vector_search_indexes.*.creator	string	REMOTE
+    resources.vector_search_indexes.*.delta_sync_index_spec	*vectorsearch.DeltaSyncVectorIndexSpecRequest	INPUT	STATE
+    resources.vector_search_indexes.*.delta_sync_index_spec	*vectorsearch.DeltaSyncVectorIndexSpecResponse	REMOTE
+    resources.vector_search_indexes.*.delta_sync_index_spec.columns_to_index	[]string	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.columns_to_index[*]	string	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.columns_to_sync	[]string	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.columns_to_sync[*]	string	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.embedding_source_columns	[]vectorsearch.EmbeddingSourceColumn	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.embedding_source_columns[*]	vectorsearch.EmbeddingSourceColumn	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.embedding_source_columns[*].embedding_model_endpoint_name	string	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.embedding_source_columns[*].model_endpoint_name_for_query	string	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.embedding_source_columns[*].name	string	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.embedding_vector_columns	[]vectorsearch.EmbeddingVectorColumn	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.embedding_vector_columns[*]	vectorsearch.EmbeddingVectorColumn	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.embedding_vector_columns[*].embedding_dimension	int	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.embedding_vector_columns[*].name	string	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.embedding_writeback_table	string	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.pipeline_id	string	REMOTE
+    resources.vector_search_indexes.*.delta_sync_index_spec.pipeline_type	vectorsearch.PipelineType	ALL
+    resources.vector_search_indexes.*.delta_sync_index_spec.source_table	string	ALL
+    resources.vector_search_indexes.*.direct_access_index_spec	*vectorsearch.DirectAccessVectorIndexSpec	ALL
+    resources.vector_search_indexes.*.direct_access_index_spec.embedding_source_columns	[]vectorsearch.EmbeddingSourceColumn	ALL
+    resources.vector_search_indexes.*.direct_access_index_spec.embedding_source_columns[*]	vectorsearch.EmbeddingSourceColumn	ALL
+    resources.vector_search_indexes.*.direct_access_index_spec.embedding_source_columns[*].embedding_model_endpoint_name	string	ALL
+    resources.vector_search_indexes.*.direct_access_index_spec.embedding_source_columns[*].model_endpoint_name_for_query	string	ALL
+    resources.vector_search_indexes.*.direct_access_index_spec.embedding_source_columns[*].name	string	ALL
+    resources.vector_search_indexes.*.direct_access_index_spec.embedding_vector_columns	[]vectorsearch.EmbeddingVectorColumn	ALL
+    resources.vector_search_indexes.*.direct_access_index_spec.embedding_vector_columns[*]	vectorsearch.EmbeddingVectorColumn	ALL
+    resources.vector_search_indexes.*.direct_access_index_spec.embedding_vector_columns[*].embedding_dimension	int	ALL
+    resources.vector_search_indexes.*.direct_access_index_spec.embedding_vector_columns[*].name	string	ALL
+    resources.vector_search_indexes.*.direct_access_index_spec.schema_json	string	ALL
+    resources.vector_search_indexes.*.endpoint_name	string	ALL
+    resources.vector_search_indexes.*.endpoint_uuid	string	REMOTE	STATE
+    resources.vector_search_indexes.*.id	string	INPUT
+    resources.vector_search_indexes.*.index_subtype	vectorsearch.IndexSubtype	ALL
+    resources.vector_search_indexes.*.index_type	vectorsearch.VectorIndexType	ALL
+    resources.vector_search_indexes.*.lifecycle	resources.Lifecycle	INPUT
+    resources.vector_search_indexes.*.lifecycle.prevent_destroy	bool	INPUT
+    resources.vector_search_indexes.*.modified_status	string	INPUT
+    resources.vector_search_indexes.*.name	string	ALL
+    resources.vector_search_indexes.*.primary_key	string	ALL
+    resources.vector_search_indexes.*.status	*vectorsearch.VectorIndexStatus	REMOTE
+    resources.vector_search_indexes.*.status.index_url	string	REMOTE
+    resources.vector_search_indexes.*.status.indexed_row_count	int64	REMOTE
+    resources.vector_search_indexes.*.status.message	string	REMOTE
+    resources.vector_search_indexes.*.status.ready	bool	REMOTE
+    resources.vector_search_indexes.*.url	string	INPUT
+    resources.vector_search_indexes.*.grants.full_name	string	ALL
+    resources.vector_search_indexes.*.grants.securable_type	string	ALL
+    resources.vector_search_indexes.*.grants[*]	catalog.PrivilegeAssignment	ALL
+    resources.vector_search_indexes.*.grants[*].principal	string	ALL
+    resources.vector_search_indexes.*.grants[*].privileges	[]catalog.Privilege	ALL
+    resources.vector_search_indexes.*.grants[*].privileges[*]	catalog.Privilege	ALL
     resources.volumes.*.access_point	string	REMOTE
     resources.volumes.*.browse_only	bool	REMOTE
     resources.volumes.*.catalog_name	string	ALL
@@ Expand Down @@

acceptance/bundle/resources/vector_search_endpoints/drift/budget_policy/output.txt

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -12,11 +12,9 @@ Deployment complete!
  
      "effective_budget_policy_id": "remote-policy"

    }

    === Plan detects drift and proposes update

    === budget_policy_id drift is ignored (effective vs requested mismatch)

    >>> [CLI] bundle plan

    update vector_search_endpoints.my_endpoint

    Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged

    Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged

    >>> [CLI] bundle destroy --auto-approve

    The following resources will be deleted:

acceptance/bundle/resources/vector_search_endpoints/drift/budget_policy/script

            
                      Original file line number
                      Diff line number
                      Diff line change
                  
    @@ -14,5 +14,5 @@ endpoint_name="vs-endpoint-${UNIQUE_NAME}"
  
    title "Simulate remote drift: set budget_policy_id outside the bundle"

    trace $CLI vector-search-endpoints update-endpoint-budget-policy "${endpoint_name}" "remote-policy"

    title "Plan detects drift and proposes update"

    trace $CLI bundle plan | contains.py "Plan: 0 to add, 1 to change, 0 to delete, 0 unchanged"

    title "budget_policy_id drift is ignored (effective vs requested mismatch)"

    trace $CLI bundle plan | contains.py "Plan: 0 to add, 0 to change, 0 to delete, 1 unchanged"

acceptance/bundle/resources/vector_search_indexes/basic/databricks.yml.tmpl

-Original file line number
+Diff line change
@@ -0,0 +1,20 @@
+    bundle:
+      name: deploy-vs-index-$UNIQUE_NAME
+    sync:
+      paths: []
+    resources:
+      vector_search_endpoints:
+        my_endpoint:
+          name: vs-endpoint-$UNIQUE_NAME
+          endpoint_type: STANDARD
+      vector_search_indexes:
+        my_index:
+          name: vs-index-$UNIQUE_NAME
+          endpoint_name: ${resources.vector_search_endpoints.my_endpoint.name}
+          primary_key: id
+          index_type: DELTA_SYNC
+          delta_sync_index_spec:
+            source_table: main.default.source_$UNIQUE_NAME
+            pipeline_type: TRIGGERED

acceptance/bundle/resources/vector_search_indexes/basic/out.requests.direct.json

-Original file line number
+Diff line change
@@ -0,0 +1,14 @@
+    {
+      "method": "POST",
+      "path": "/api/2.0/vector-search/indexes",
+      "body": {
+        "delta_sync_index_spec": {
+          "pipeline_type": "TRIGGERED",
+          "source_table": "main.default.source_[UNIQUE_NAME]"
+        },
+        "endpoint_name": "vs-endpoint-[UNIQUE_NAME]",
+        "index_type": "DELTA_SYNC",
+        "name": "vs-index-[UNIQUE_NAME]",
+        "primary_key": "id"
+      }
+    }

acceptance/bundle/resources/vector_search_indexes/basic/out.test.toml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

acceptance/bundle/resources/vector_search_indexes/basic/output.txt

-Original file line number
+Diff line change
@@ -0,0 +1,72 @@
+    >>> [CLI] bundle validate
+    Name: deploy-vs-index-[UNIQUE_NAME]
+    Target: default
+    Workspace:
+      User: [USERNAME]
+      Path: /Workspace/Users/[USERNAME]/.bundle/deploy-vs-index-[UNIQUE_NAME]/default
+    Validation OK!
+    >>> [CLI] bundle summary
+    Name: deploy-vs-index-[UNIQUE_NAME]
+    Target: default
+    Workspace:
+      User: [USERNAME]
+      Path: /Workspace/Users/[USERNAME]/.bundle/deploy-vs-index-[UNIQUE_NAME]/default
+    Resources:
+      Vector Search Endpoints:
+        my_endpoint:
+          Name: vs-endpoint-[UNIQUE_NAME]
+          URL:  [DATABRICKS_URL]/compute/vector-search/vs-endpoint-[UNIQUE_NAME]?o=[NUMID]
+      Vector Search Indexes:
+        my_index:
+          Name: vs-index-[UNIQUE_NAME]
+          URL:  (not deployed)
+    >>> [CLI] bundle deploy
+    Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/deploy-vs-index-[UNIQUE_NAME]/default/files...
+    Deploying resources...
+    Updating deployment state...
+    Deployment complete!
+    >>> [CLI] vector-search-indexes get-index vs-index-[UNIQUE_NAME]
+    {
+      "name": "vs-index-[UNIQUE_NAME]",
+      "endpoint_name": "vs-endpoint-[UNIQUE_NAME]",
+      "index_type": "DELTA_SYNC",
+      "primary_key": "id"
+    }
+    >>> [CLI] bundle summary
+    Name: deploy-vs-index-[UNIQUE_NAME]
+    Target: default
+    Workspace:
+      User: [USERNAME]
+      Path: /Workspace/Users/[USERNAME]/.bundle/deploy-vs-index-[UNIQUE_NAME]/default
+    Resources:
+      Vector Search Endpoints:
+        my_endpoint:
+          Name: vs-endpoint-[UNIQUE_NAME]
+          URL:  [DATABRICKS_URL]/compute/vector-search/vs-endpoint-[UNIQUE_NAME]?o=[NUMID]
+      Vector Search Indexes:
+        my_index:
+          Name: vs-index-[UNIQUE_NAME]
+          URL:  (not deployed)
+    >>> print_requests.py //vector-search/indexes
+    >>> [CLI] bundle destroy --auto-approve
+    The following resources will be deleted:
+      delete resources.vector_search_endpoints.my_endpoint
+      delete resources.vector_search_indexes.my_index
+    This action will result in the deletion of the following Vector Search indexes.
+    For Delta Sync indexes, the source Delta Table is preserved but the embedding pipeline is removed.
+    For Direct Access indexes, all upserted vectors are permanently lost:
+      delete resources.vector_search_indexes.my_index
+    All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/deploy-vs-index-[UNIQUE_NAME]/default
+    Deleting files...
+    Destroy complete!

acceptance/bundle/resources/vector_search_indexes/basic/script

-Original file line number
+Diff line change
@@ -0,0 +1,22 @@
+    envsubst < databricks.yml.tmpl > databricks.yml
+    cleanup() {
+        trace $CLI bundle destroy --auto-approve
+        rm -f out.requests.txt
+    }
+    trap cleanup EXIT
+    trace $CLI bundle validate
+    trace $CLI bundle summary
+    rm -f out.requests.txt
+    trace $CLI bundle deploy
+    # Get index details
+    index_name="vs-index-${UNIQUE_NAME}"
+    trace $CLI vector-search-indexes get-index "${index_name}" | jq '{name, endpoint_name, index_type, primary_key}'
+    trace $CLI bundle summary
+    trace print_requests.py //vector-search/indexes > out.requests.$DATABRICKS_BUNDLE_ENGINE.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add vector_search_indexes resource (direct engine) #5123

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Add vector_search_indexes resource (direct engine) #5123

Are you sure you want to change the base?

Uh oh!

Add vector_search_indexes resource (direct engine) #5123

Uh oh!

Uh oh!

Diff view

Diff view

There are no files selected for viewing

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!