feat(embeddings): generic per-input token cap across all providers

[nicoloboschi]

Summary

There were two separate embedding-truncation knobs — a token-based
HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MAX_INPUT_TOKENS wired only into the
LiteLLM SDK provider (Bedrock Titan, #2501), and a proposed char-based
OpenAI-only cap (#3091). This unifies them into one generic,
provider-agnostic flag and moves the truncation to the single choke point
that every embedding path flows through.

What changed

• New flag HINDSIGHT_API_EMBEDDINGS_MAX_INPUT_TOKENS (config
  embeddings_max_input_tokens), off by default, applies to every provider
  (local, OpenAI-compatible/llama.cpp, LiteLLM SDK/Bedrock, Cohere, Gemini…).
  Token-based (tiktoken cl100k_base, approximate — set with headroom).
• Backward compatible: the old
  HINDSIGHT_API_EMBEDDINGS_LITELLM_SDK_MAX_INPUT_TOKENS is kept as a
  deprecated alias and folded into the generic name at load time (generic wins
  when both are set).
• Truncation moved before the provider: it now runs once in
  generate_embeddings_batch (the universal boundary for retain, recall
  queries, consolidation, and import) instead of inside a single provider's
  encode(). Removed the per-provider max_input_tokens from
  LiteLLMSDKEmbeddings.
• truncate_to_tokens moved to token_encoding.py and now returns a
  TokenTruncation dataclass (no tuple return).

Why

Remote providers with a fixed input-token limit (Bedrock Titan V2's hard 8192
cap, or a self-hosted llama.cpp /v1/embeddings server) reject an oversized
input with a permanent 4xx rather than truncating server-side the way local
SentenceTransformers does. One oversized memory then fails the whole
retain/recall batch. The cap is the escape hatch — and it belongs to every
provider, not just LiteLLM SDK.

Validation

• tests/test_embeddings_max_input_tokens.py (new): central truncation +
  warning, cap-applied-before-backend, no-cap-passes-verbatim, and config
  parsing (generic env, deprecated alias, precedence, default-disabled).
• LiteLLM-SDK truncation tests migrated out of
  test_litellm_sdk_embeddings.py into the central path.
• lint.sh ✓, ty check ✓, targeted tests green.

Docs

configuration.md moves the row into the general Embeddings section, notes it
applies to all providers, and records the deprecated alias. .env.example
(and the bundled hindsight-embed copy) updated.