feat(semantic): support high-dimensional embeddings

[chrisolszewski]

Closes #49

Summary

Raises the embedding dimension cap from 1024 to 4096 so larger models like
text-embedding-3-large (3072) can be indexed through the external backends.
The cache format and all defaults stay the same.

What changed

• semantic_index.rs: MAX_DIMENSION goes from 1024 to 4096. Dimension
  checking is consolidated into one validate_embedding_dimension() that runs
  both when a backend returns a batch and when a cache is read back.
• Tests: build, persist, reload, and search at 4096; reject 0 and 4097 on both
  build and deserialization.

Why just the cap and one check

This PR intentionally stays small: a constant change plus a single boundary
check. Bigger questions like aggregate memory ceilings and corrupt-cache
hardening are left out so they can be handled on their own if you want them.

Validation

• cargo fmt --check
• cargo test -p agent-file-tools --test semantic_validation_test
• cargo test -p agent-file-tools semantic_index

---

Summary by cubic

Raises the embedding dimension cap to 4096 to support high‑dimensional models like text-embedding-3-large (3072) and common 4096‑d local models. Cache format and defaults are unchanged.

• New Features

  • Increase MAX_DIMENSION from 1024 to 4096.
  • Allow 4096‑d embeddings during build and read; reject 0 and >4096 with clear errors.

• Refactors

  • Add validate_embedding_dimension() and use it in batch validation and deserialization.

^{Written for commit 32f7f25. Summary will update on new commits. Review in cubic}

[cubic-dev-ai]

No issues found across 2 files

_{Re-trigger cubic}