fix: local package dep fix

[jamescalam]

User description

Due to constraints in current fastembed dependencies, I have temporarily dropped the package so that HuggingFace and other local package dependencies can still function. In the meantime user's can install fastembed separately to use the fastembed encoders.

See related issue in fastembed repo here

---

PR Type

Bug fix, Enhancement

---

Description

• Updated the version number from 0.0.44 to 0.0.45.
• Removed fastembed dependency to resolve local package conflicts.
• Added tokenizers as a new dependency.
• Adjusted version constraints for transformers to ensure compatibility.

---

Changes walkthrough 📝

	Relevant files
Enhancement	__init__.py Bump version number to 0.0.45                                                        semantic_router/init.py Updated the version number from 0.0.44 to 0.0.45. +1/-1
Bug fix	pyproject.toml Update dependencies and version in pyproject.toml                pyproject.toml Updated the version number from 0.0.44 to 0.0.45. Removed fastembed dependency. Added tokenizers dependency. Adjusted version constraints for transformers. +4/-5

---

💡 PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

[github-actions]

PR Description updated to latest commit (d3468ff)

[github-actions]

PR Review 🔍

(Review updated until commit d3468ff)

⏱️ Estimated effort to review [1-5]	2, because the changes are straightforward and involve version updates and dependency modifications. The PR is small and the changes are clear, making it relatively easy to review.
🧪 Relevant tests	No
⚡ Possible issues	Dependency Compatibility: Removing fastembed and adding tokenizers might introduce compatibility issues with existing code that relies on fastembed. It's crucial to ensure that all functionalities work as expected without fastembed.
🔒 Security concerns	No

Code feedback:

---

relevant file	pyproject.toml
suggestion	Consider specifying a more precise version range for the tokenizers dependency to avoid potential future compatibility issues with other packages. For example, you could use tokenizers = "^0.19.0" to ensure compatibility with minor updates but not major changes that could break functionality. [important]
relevant line	tokenizers = {version = ">=0.19", optional = true}

---

relevant file	pyproject.toml
suggestion	Since fastembed is removed from the dependencies, it's important to ensure that no residual code or references to fastembed remain in the project that could cause runtime errors. A thorough search in the codebase for any fastembed imports or usage would be prudent. [important]
relevant line	-fastembed = {version = "^0.2.4", optional = true, python = "<3.12"}

---

relevant file	pyproject.toml
suggestion	It might be beneficial to add a comment next to the transformers dependency explaining why the version constraint was changed to >=4.36.2. This can help maintainers understand the reason for this change, especially if it's related to compatibility or specific features needed from the newer versions. [medium]
relevant line	transformers = {version = ">=4.36.2", optional = true}

---

relevant file	semantic_router/__init__.py
suggestion	Ensure that all references to the version number in the codebase, documentation, and possibly CI/CD configurations are updated to reflect the new version 0.0.45. This helps in maintaining consistency and avoiding issues during deployment or package releases. [medium]
relevant line	__version__ = "0.0.45"

---

[github-actions]

PR Description updated to latest commit (d3468ff)

[github-actions]

Persistent review updated to latest commit d3468ff

[github-actions]

PR Code Suggestions ✨

Category	Suggestion	Score
Possible issue	Adjust the version constraints for transformers and tokenizers to be more restrictive to avoid potential compatibility issues Ensure that the version constraints for transformers and tokenizers are consistent with the rest of the dependencies. The current change introduces a more permissive version constraint for transformers and tokenizers, which might lead to compatibility issues with other packages or the project's environment. pyproject.toml [29-30] -transformers = {version = ">=4.36.2", optional = true} -tokenizers = {version = ">=0.19", optional = true} +transformers = {version = "^4.36.2", optional = true} +tokenizers = {version = "^0.19", optional = true} Suggestion importance[1-10]: 9 Why: The suggestion addresses a potential compatibility issue by making the version constraints more restrictive, which is crucial for maintaining a stable environment.	9
Maintainability	Implement dynamic versioning by fetching the package version directly from the project's metadata Consider using a more dynamic approach for version management to avoid manual updates. This can be done by reading the version from a single source like pyproject.toml or another configuration file, which helps in maintaining consistency across the project. semantic_router/init.py [7] -__version__ = "0.0.45" +from importlib.metadata import version +__version__ = version("semantic-router") Suggestion importance[1-10]: 8 Why: This suggestion improves maintainability by centralizing version management, reducing the risk of inconsistencies across the project.	8
Enhancement	Add fastembed back to the local extras to ensure no loss of functionality Consider adding back fastembed or providing a replacement if it was intentionally removed. The removal of fastembed from the dependencies and extras without a clear replacement might lead to missing functionality that was previously available. pyproject.toml [46] -local = ["torch", "transformers", "tokenizers", "llama-cpp-python"] +local = ["torch", "transformers", "tokenizers", "llama-cpp-python", "fastembed"] Suggestion importance[1-10]: 7 Why: The suggestion is valid as it ensures no loss of functionality by re-adding fastembed, but it may not be necessary if the removal was intentional and the functionality is no longer needed.	7
Best practice	Ensure essential dependencies are not marked as optional if they are critical for the package's operation Verify the necessity of marking transformers and tokenizers as optional dependencies. If these packages are essential for the local extras functionality, consider removing the optional = true flag to ensure they are installed with the package. pyproject.toml [29-30] -transformers = {version = ">=4.36.2", optional = true} -tokenizers = {version = ">=0.19", optional = true} +transformers = {version = ">=4.36.2", optional = false} +tokenizers = {version = ">=0.19", optional = false} Suggestion importance[1-10]: 6 Why: The suggestion is reasonable for ensuring critical dependencies are always installed, but it might not be necessary if the packages are truly optional for certain use cases.	6