feat: Request for Enhanced Text Segmentation and Compression Features #241
Description
(Content translated by AI)
1、Text Segmentation for Download:
I would like to suggest adding a feature that allows for text segmentation during downloads. Given the context capacity limitations of the model, we could set a threshold to automatically split long texts into manageable segments. Additionally, it would be helpful to include friendly content prompts at the beginning of each segmented section to guide users through the content.
2、Text Compression for Redundant Content:
I have noticed that a significant portion of the content contains repetitive "[Non-text file]" entries. It would be beneficial to implement a compression feature that consolidates these blank or redundant entries into a more concise representation, rather than repeating them multiple times. This would improve the overall readability and efficiency of the content.
英文内容来自AI翻译，下面是原文。
功能建议
1、希望可以增加字数分段下载功能，因为模型有上下文容量限制，所以我们可以设置一个阈值，对长文本进行分段，并且在分割文本的头部加入一些友好的内容补充提示。
2、多余的文本压缩，我发现内容中有大部分 "[Non-text file]" 的文本,我觉得可以把这些空白内容的文件进行一个整合表达，而非重复表示，达到一个压缩内容的效果。