Skip to content
This repository has been archived by the owner on Aug 10, 2023. It is now read-only.

fix for crashes in tiktoken due to message being empty and not being … #1458

Merged
merged 5 commits into from
Jul 8, 2023
Merged

fix for crashes in tiktoken due to message being empty and not being … #1458

merged 5 commits into from
Jul 8, 2023

Conversation

MrMatch246
Copy link
Contributor

…handled down the line.

  File "/home/user/REPOS/rAIversing/venv3/lib/python3.11/site-packages/revChatGPT/V3.py", line 390, in ask
    full_response: str = "".join(response)
                         ^^^^^^^^^^^^^^^^^
  File "/home/user/REPOS/rAIversing/venv3/lib/python3.11/site-packages/revChatGPT/V3.py", line 197, in ask_stream
    self.__truncate_conversation(convo_id=convo_id)
  File "/home/user/REPOS/rAIversing/venv3/lib/python3.11/site-packages/revChatGPT/V3.py", line 143, in __truncate_conversation
    self.get_token_count(convo_id) > self.truncate_limit
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/REPOS/rAIversing/venv3/lib/python3.11/site-packages/revChatGPT/V3.py", line 169, in get_token_count
    num_tokens += len(encoding.encode(value))
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/REPOS/rAIversing/venv3/lib/python3.11/site-packages/tiktoken/core.py", line 116, in encode
    if match := _special_token_regex(disallowed_special).search(text):
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: expected string or buffer

acheong08
acheong08 suggested changes Jul 7, 2023
View reviewed changes
src/revChatGPT/V3.py Outdated
@@ -166,7 +166,8 @@ def get_token_count(self, convo_id: str = "default") -> int:
# every message follows <im_start>{role/name}\n{content}<im_end>\n
num_tokens += 5
for key, value in message.items():
num_tokens += len(encoding.encode(value))
if not(value is None or value == ""):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be simplified to if value:

@acheong08 acheong08 merged commit 2126821 into acheong08:main Jul 8, 2023
5 checks passed
acheong08 pushed a commit that referenced this pull request Jul 8, 2023
Revert "fix for crashes in tiktoken due to message being empty and no…
ea9cff6 
…t being … (#1458)"

This reverts commit 2126821.
@acheong08 acheong08 mentioned this pull request Jul 8, 2023
Merged
acheong08 pushed a commit that referenced this pull request Jul 8, 2023
Revert "fix for crashes in tiktoken due to message being empty and no…
4f8b2f6 
…t being … (#1458)" (#1459)

This reverts commit 2126821.
@acheong08
Copy link
Owner

Something went wrong when merging

acheong08 pushed a commit that referenced this pull request Jul 8, 2023
reapply #1458
3714d36
@acheong08
Copy link
Owner

merged

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@acheong08 acheong08 acheong08 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@MrMatch246 @acheong08