-
Notifications
You must be signed in to change notification settings - Fork 13.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add encoding parameter to TextLoader #2250
Conversation
This merge request proposes changes to the TextLoader class to make it more flexible and robust when handling text files with different encodings. The current implementation of TextLoader does not provide a way to specify the encoding of the text file being read. As a result, it might lead to incorrect handling of files with non-default encodings, causing issues with the content.
Similar to #2223 but little more user control. |
|
||
def load(self) -> List[Document]: | ||
"""Load from file path.""" | ||
with open(self.file_path, encoding="utf-8") as f: | ||
with open(self.file_path, encoding=self.encoding) as f: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Addition of 'r' parameter fixes the UnicodeDecodeError
. Could you also add that acc. to #2223
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
From first glance, I believe mode='r' is the default, so I do not know why that would cause a problem. https://docs.python.org/3/library/functions.html#open
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That is true. The updated version 0.0.129 did fix that issue. I am not sure why I had that issue in Windows. btw, would it be better if the default encoding is 'utf-8' as opposed to None.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks!
This merge request proposes changes to the TextLoader class to make it more flexible and robust when handling text files with different encodings. The current implementation of TextLoader does not provide a way to specify the encoding of the text file being read. As a result, it might lead to incorrect handling of files with non-default encodings, causing issues with loading the content.
Benefits: