Refactor Spanner embedding generation to run ML.PREDICT outside write transaction

[dwnoble]

Summary

This change refactors the Spanner embedding ingestion helper to separate remote machine learning inference (ML.PREDICT) from Spanner read-write transactions.

Problem

Previously, the embedding generator executed INSERT OR UPDATE ... SELECT FROM ML.PREDICT(...) within a Spanner read-write transaction. Because ML.PREDICT makes external network calls to Vertex AI, the write transaction remained open for a prolonged period (~85 seconds in a typical bulk ingestion batch of 1,000 nodes).

During bulk imports (such as the undata dataset), Spanner frequently runs high-priority background range splits and directory migrations (_MoveRange). The long-held write locks from our transaction repeatedly conflicted with these range moves, causing Spanner to abort the user transaction with a 409 conflict:
Transaction was aborted. It was wounded by a higher priority transaction with tag "_MoveRange" due to conflict on keys with prefix [<internal>], column state in table split.

Solution

1. Uncouple Prediction from Transaction: Run the ML.PREDICT query as a read-only query outside the read-write transaction using a database snapshot. This fetches the generated embeddings into memory without holding Spanner locks.
2. Fast DML Write: Write the retrieved embeddings to the database in a short, local read-write transaction using batch parameter arrays. The transaction duration is reduced from ~85 seconds to <50 milliseconds, eliminating range split conflicts.
3. DDL and Test Cleanups:
   • Removed unused secondary index NodeEmbeddingLabelIndex on embedding_label.
   • Updated integration and unit tests in app_test.py and utils/embeddings_test.py to match the two-stage execution model."

[codacy-production]

Not up to standards ⛔

🔴 Issues 1 medium · 1 minor

Alerts:
⚠ 2 issues (≤ 0 issues of at least minor severity)

Results:
2 new issues

Category	Results
Security	1 medium
CodeStyle	1 minor

View in Codacy

🟢 Metrics 4 complexity

Metric	Results
Complexity	4

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.}

[gemini-code-assist]

Code Review

This pull request refactors the embedding generation process in _generate_embeddings_partitioned by running ML.PREDICT as a read-only query outside of the read-write transaction, followed by writing the results in a separate, short transaction to reduce transaction overhead. Additionally, the unused embedding_label_index has been removed from the schema, configuration, Spanner client, and tests. Feedback suggests filtering out any None embeddings returned by ML.PREDICT before performing the write transaction to avoid unnecessary write operations.

[gmechali]

Thank you Dan! (pls wait for Xiao's review too)