fix: James/encoder disallowed special

[jamescalam]

Type

enhancement, bug_fix

---

Description

• Updated the version of the semantic-router package to 0.0.37 in both init.py and pyproject.toml.
• Enhanced the text encoding in semantic_router/encoders/openai.py by replacing the encode method with encode_ordinary for faster processing without disallowed special tokens.

---

Changes walkthrough

	Relevant files
Configuration changes	__init__.py Update package version in __init__.py                                        semantic_router/init.py Updated the version number from 0.0.36 to 0.0.37. +1/-1      pyproject.toml Update project version in pyproject.toml                                  pyproject.toml Updated the version number from 0.0.36 to 0.0.37. +1/-1
Enhancement	openai.py Optimize text encoding by using encode_ordinary                    semantic_router/encoders/openai.py Changed the method of encoding text to use encode_ordinary instead of encode with no disallowed special tokens. +2/-1

---

✨ PR-Agent usage:
Comment /help on the PR to get a list of all available PR-Agent tools and their descriptions

[github-actions]

PR Description updated to latest commit (cc1430d)

[github-actions]

PR Review

⏱️ Estimated effort to review [1-5]	2, because the changes are straightforward and limited to version updates and a method replacement in the encoding process. The logic is not complex, and the impact is localized to specific parts of the code.
🧪 Relevant tests	No
🔍 Possible issues	Possible Bug: The replacement of encode with encode_ordinary in semantic_router/encoders/openai.py might not handle special tokens as the original method did, potentially altering the behavior of the encoder.
🔒 Security concerns	No

Code feedback:

---

relevant file	semantic_router/encoders/openai.py
suggestion	Ensure that encode_ordinary method properly handles edge cases or special conditions previously managed by encode. This might involve adding checks or conditions that were implicitly handled by the encode method. [important]
relevant line	tokens = self._token_encoder.encode_ordinary(text)

---

---

✨ Review tool usage guide:

---

Overview:
The review tool scans the PR code changes, and generates a PR review which includes several types of feedbacks, such as possible PR issues, security threats and relevant test in the PR. More feedbacks can be added by configuring the tool.

The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on any PR.

• When commenting, to edit configurations related to the review tool (pr_reviewer section), use the following template:

/review --pr_reviewer.some_config1=... --pr_reviewer.some_config2=...

• With a configuration file, use the following template:

[pr_reviewer]
some_config1=...
some_config2=...

See the review usage page for a comprehensive guide on using this tool.

[github-actions]

Failed to generate code suggestions for PR

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.39%. Comparing base (eb9ebe3) to head (cc1430d).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #263   +/-   ##
=======================================
  Coverage   80.39%   80.39%           
=======================================
  Files          44       44           
  Lines        2443     2443           
=======================================
  Hits         1964     1964           
  Misses        479      479           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.