Add a notice when indexing flow text is only accept Bytes. Not str #355
Comments
Hi, using bytes to represent documents seems counter-intuitive, let me explain why. In the very early version of GNES, we did send text in vanilla
One follow-up question you may have is, if every income data is in bytes, how can GNES know what is what and how to deserialize these bytes to the correct modality? The answer is the Preprocessor. It will deserialize the bytes into the correct modal and Python data type. Note how the class attribute gnes/gnes/preprocessor/base.py Lines 41 to 54 in b4d2c8c
For example, using a As a summary, let me repeat the whole procedure again.
In short, a GNES flow/stack without a If you feel like this idea need to be known more for others, welcome to make a PR about this, either via Python |
Hi @hanxiao , I see that problem when use Thanks |
Hi. I found an issue when use Flow to indexing text as type str it will encounter horrible message:
This caused by passing the string not bytes.
so to solve this you need to convert/encode your str type to bytes.
I will make PR for this soon
The text was updated successfully, but these errors were encountered: