add openAI embedding support

[ishaanxgupta]

No description provided.

[github-actions]

	Fails
🚫	🔐 This PR modifies sensitive files: src/config/settings.py. These require review by a core maintainer (@ishaanxgupta or @ved015) before merging.
🚫	📋 PR description is too short. Please describe: What changed and Why Any relevant issue links (Closes #NNN) Steps to test manually

	Warnings
⚠️	🧪 Source files in src/ were modified but no test files changed. Please add or update tests to cover your changes.

	Messages
📖	📝 No CHANGELOG.md update detected. If this PR introduces a user-visible change, please add an entry.
📖	✅ Targeting main. Please squash commits before merging to keep the git history clean.

Generated by 🚫 dangerJS against aa86886

[github-actions]

✅ Staging Deployment Report

Item	Value
Branch	develop
Commit	7e959f6
Environment	Staging
Health	http://13.232.74.176:8001/health
API Docs	http://13.232.74.176:8001/docs
Smoke Tests	success

🟢 Staging is live and healthy! Test your changes at the staging URL above.

Ready to ship? Comment /promote on this PR to merge to main and deploy to production.

[gemini-code-assist]

Code Review

This pull request integrates OpenAI embedding models into the ingestion pipeline, adding support for models like text-embedding-3-small and large. Key changes include a new lazy-loaded OpenAI client, auto-detection of OpenAI models, and logic to handle native dimension reduction and analytics tracking. Review feedback identified a potential AttributeError when tracking token usage if the API response lacks usage data, suggesting a safer way to access these attributes to ensure robustness.

[gemini-code-assist]

Accessing response.usage.total_tokens directly via getattr on response.usage will raise an AttributeError if response.usage is None. While the OpenAI API typically returns usage information, it's safer to handle cases where it might be missing, especially when using OpenAI-compatible proxies or different library versions that might not guarantee the presence of the usage object.

    usage = getattr(response, "usage", None)
    input_tokens = getattr(usage, "total_tokens", 0) if usage else len(text.split())