Resolved #4: Handles multi-byte characters with url-retrieve (i.e., without the external curl command) #29
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
March 30, 2023 17:34
If the user provides the OpenAPI key via a function, as is the case by default if the user puts credentials in an auth-sources resource like .authinfo or .authinfo.gpg, then it is necessary to encode the function's returned value into utf-8 before passing it onward to build the HTTP request. This commit ensures that happen. Why is this necessary, given that the API key contains only alphanumeric characters and therefore should be byte-for-byte the same in utf-8 as in us-acii? I don't know. It may because emacs's url-http.el library concats many strings together, and they all need to be identically encoded before they can be combined correctly. Whatever the reason, this fix works and allows you to send prompts which include Unicode characters that require multibyte encodings in UTF-8
This changes response parsing so that the response body is decoded as UTF-8 and then parsed as JSON, rather than the other way around. This fixes the handling of responses that contain Unicode characters which are encoded with multiple bytes in UTF-8, such as emojis.
|
This is intended to resolve #4 |
|
Appears to work for me too. This is great! I'm also on Emacs 28.2 on Linux, I'm going to make the assumption here that explicitly encoding all parts of the request as utf-8 is the conservative approach and thus unlikely to cause issues in other versions or platforms.
Ah, I can see why this could be a problem. Pushed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
These two commits fix handling of multi-byte characters in the prompt send to OpenAI and in the response received from it.
As a result, you can now use non-Latin alphabets like Greek, or emojis.
To make requests work, it was necessary to encode the loaded API key as utf-8, for reasons that remain obscure, but may have to do with some subtlety concerning how the
concatfunction combines multibyte and non-multibyte strings. To make replies work, it was necessary to decode the response body as utf-8 before, not after, parsing it as json.I have tested this lightly on Emacs 28.1 running on Linux.