Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

streamHandler 返回的 streamData 存在截断问题 #542

Closed
liby opened this issue May 26, 2023 · 10 comments
Closed

streamHandler 返回的 streamData 存在截断问题 #542

liby opened this issue May 26, 2023 · 10 comments
Labels
bug Something isn't working

Comments

@liby
Copy link

liby commented May 26, 2023

运行环境

  • Device: iMac(Retina 5K, 27-inch, 2020)
  • macOS: 13.3.1 (22E261)
  • Bob Version: 1.8.0

描述问题

在请求经过代理的 OpenAI API 时，streamData.text 出现截断的问题，模板项目是 chatbot-ui，相关的代码在 chatbot-ui/blob/main/pages/api/chat.ts

image

通过 Console 看到失败的日志 log，log 是在这一行下面打印的

image

通过抓包看到 Response 返回的数据没有问题：

image
Response 
data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"role":"assistant"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"我们"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"记录"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"您"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"的"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"API"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"使用"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"数据"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"，"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"例如"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"调"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"用"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"次"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"数"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"和"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"使用"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"的"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"API"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"令"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"牌"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"。"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"但"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"是"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"，"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"我们"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"不"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"会"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"存"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"储"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"或"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"访"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"问"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"发送"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"到"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"API"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"的"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"请求"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"内容"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{"content":"。"},"index":0,"finish_reason":null}]}

data: {"id":"chatcmpl-7KJM8RQ9fDHPB7WbhCevAGbbRRiGW","object":"chat.completion.chunk","created":1685075320,"model":"gpt-3.5-turbo-0301","choices":[{"delta":{},"index":0,"finish_reason":"stop"}]}

data: [DONE]
@liby liby added the bug Something isn't working label May 26, 2023
@liby liby changed the title streamHandler 返回的 streamData 存在问题 streamHandler 返回的 streamData 存在截断问题 May 26, 2023
@ripperhe
Copy link
Owner

@liby 这个原因应该是代理服务器没处理好，把流式数据截断返回了。（把一条 data 从中间截断了）

Bob 只会把服务器返回的数据原封不动抛给插件，如果服务器已经截断了，Bob 这里返回的数据也是截断的。

从抓包 response 看不出来问题，因为被截断的流式数据最终会合并在一起展示，所以看不出来。但是从你日志可以看出来服务器将一条 data 分成了两部分返回。

可以参考下 #504，可能是同样的问题

image

@liby
Copy link
Author

liby commented May 26, 2023 •
edited
Loading

@ripperhe

看起来有这个可能，不过这个项目并没有使用 ningx 相关的显式配置，而是采用的 Vercel edge function 的默认配置。

尝试在其他桌面客户端比如 OpenCat、Openai Translator 上使用也没有遇到类似截断的问题，所以才猜测是不是 Bob 在处理数据时存在问题

@ripperhe
Copy link
Owner

@liby

这个出现概率高吗？

如果其他客户端都没有问题的话，我后面也检查下是不是 Bob 网络库的问题，不过这个可能较小。

@liby
Copy link
Author

liby commented May 26, 2023

如果超过 3 个单词以上 80% 的几率会出现截断

@ripperhe
Copy link
Owner

ripperhe commented Jun 1, 2023

@liby 考虑了下，难免会有截断的情况，参考 openai 官方代码做个兼容逻辑吧

  • 如果新的 stream text 末尾没有 \n，则视为数据被截断了
    • 把本次准备解析的数据的最后一行删掉
    • 并且把删掉的最后一行用 buffer 变量缓存起来
    • 拼接到下一次的 stream text 的头部
  • 否则，视为数据完整，重置 buffer 变量

https://github.com/openai/openai-cookbook/blob/main/apps/file-q-and-a/nextjs/src/services/openai.ts

image

@liby
Copy link
Author

liby commented Jun 1, 2023 •
edited
Loading

感谢提供思路，已解决

@liby liby closed this as completed Jun 1, 2023
@ripperhe ripperhe mentioned this issue Jun 1, 2023
Closed
@liby
Copy link
Author

liby commented Nov 30, 2023 •
edited
Loading

@ripperhe 最近从同事那里得到反馈说这个问题又出现了，Debug 了一下，发现部署在 Vercel Edge function 上 的 OpenAI API 返回的流数据除了截断的问题以外，还有顺序随机的问题（

比如我尝试翻译："Too easy! This will connect to the OpenAI API and return the response to the client. The only thing we need to do now is modify the response to be a stream." 得到结果：「OpenAI API并将响。现在应为流。」
这时候查看日志：

info	18:28:26.676511+0800	Bob	[I] [yetone.openai.translator]: index 1: data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"Open"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"AI"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" A
info	18:28:26.717190+0800	Bob	[I] [yetone.openai.translator]: index 2: PI"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"并"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"将"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"响
info	18:28:26.771152+0800	Bob	[I] [yetone.openai.translator]: index 3: "},"finish_reason":null}]}
info	18:28:26.887237+0800	Bob	[I] [yetone.openai.translator]: index 4: data: {"id":"chatcmpl-8QYZP
info	18:28:26.887370+0800	Bob	[I] [yetone.openai.translator]: index 5: lxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"。"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"现"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"在
info	18:28:26.960846+0800	Bob	[I] [yetone.openai.translator]: index 6: "},"finish_reason":null}]}
info	18:28:27.145579+0800	Bob	[I] [yetone.openai.translator]: index 7: 响"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"应"},"finish_reason":null}]}
info	18:28:27.145919+0800	Bob	[I] [yetone.openai.translator]: index 8: data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"为"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"流"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"
info	18:28:27.146450+0800	Bob	[I] [yetone.openai.translator]: index 9: 。"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

而且这个问题还不是稳定复现的；且有些时候还会出现在数据流中间就会出现 data: [DONE] 数据的情况（概率极低）。

我 Debug 了一天，没有什么头绪，想请教一下作者有什么好的建议嘛 😢

@ripperhe
Copy link
Owner

ripperhe commented Dec 1, 2023

@ripperhe 最近从同事那里得到反馈说这个问题又出现了，Debug 了一下，发现部署在 Vercel Edge function 上 的 OpenAI API 返回的流数据除了截断的问题以外，还有顺序随机的问题（

比如我尝试翻译："Too easy! This will connect to the OpenAI API and return the response to the client. The only thing we need to do now is modify the response to be a stream." 得到结果：「OpenAI API并将响。现在应为流。」 这时候查看日志：

info	18:28:26.676511+0800	Bob	[I] [yetone.openai.translator]: index 1: data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"Open"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"AI"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":" A
info	18:28:26.717190+0800	Bob	[I] [yetone.openai.translator]: index 2: PI"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"并"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"将"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"响
info	18:28:26.771152+0800	Bob	[I] [yetone.openai.translator]: index 3: "},"finish_reason":null}]}
info	18:28:26.887237+0800	Bob	[I] [yetone.openai.translator]: index 4: data: {"id":"chatcmpl-8QYZP
info	18:28:26.887370+0800	Bob	[I] [yetone.openai.translator]: index 5: lxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"。"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"现"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"在
info	18:28:26.960846+0800	Bob	[I] [yetone.openai.translator]: index 6: "},"finish_reason":null}]}
info	18:28:27.145579+0800	Bob	[I] [yetone.openai.translator]: index 7: 响"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"应"},"finish_reason":null}]}
info	18:28:27.145919+0800	Bob	[I] [yetone.openai.translator]: index 8: data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"为"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"流"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{"content":"
info	18:28:27.146450+0800	Bob	[I] [yetone.openai.translator]: index 9: 。"},"finish_reason":null}]}

data: {"id":"chatcmpl-8QYZPlxerdJll72kSFpO2rPVDBO9x","object":"chat.completion.chunk","created":1701340107,"model":"gpt-3.5-turbo-0301","system_fingerprint":null,"choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

而且这个问题还不是稳定复现的；且有些时候还会出现在数据流中间就会出现 data: [DONE] 数据的情况（概率极低）。

我 Debug 了一天，没有什么头绪，想请教一下作者有什么好的建议嘛 😢

所以服务器返回的数据就是乱的对吧，而且每一段数据里面也没有可以用来排序的参数，我也没啥思路 😂，有见到其他 App 有这种问题吗？

有没有抓包验证过 Bob 日志和服务器实际返回的顺序是一致的，不过 Bob 用的是主流的网络框架，应该不存在把顺序搞乱这种问题。

@liby
Copy link
Author

liby commented Dec 1, 2023 •
edited
Loading

有没有抓包验证过 Bob 日志和服务器实际返回的顺序是一致的，不过 Bob 用的是主流的网络框架，应该不存在把顺序搞乱这种问题

Charles 抓包看了没问题，不过就像你之前讲过的，因为截断的流式数据最终会合并在一起展示，所以抓包工具看不出来问题也是正常的。

但是从 Console 输出的 log（在代码里手动打印返回的 streamData.text）能看出来，部署在 Vercel Edge function 的 OpenAI API 返回的数据就是有被随机打乱和截断的问题，应该跟 Bob 无关。

顺着这个问题我找到了一个 issue：中文字符截断导致乱码问题，虽然问题导致的结果不同，但是导致问题的原因是高度相似的，我还试了一下，中译英返回的流数据也存在被截断的情况，但尝试了很多次都没有流数据乱序的问题，翻译结果也是对的。

比如我将「有没有抓包验证过 Bob 日志和服务器实际返回的顺序是一致的，不过 Bob 用的是主流的网络框架，应该不存在把顺序搞乱这种问题。」翻译成英文，在 Console 中看到的日志：

info	10:02:27.317675+0800	Bob	[I] [yetone.openai.translator]: index 0: data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":"Have"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" you"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" ve<…>
info	10:02:27.392592+0800	Bob	[I] [yetone.openai.translator]: index 1: data: {"id":"chatcmpl-8Qn9J
info	10:02:27.392694+0800	Bob	[I] [yetone.openai.translator]: index 2: flcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" that"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" the"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" order"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" of"},"finish_reason":null}]}

data: {"id"<…>
info	10:02:27.438997+0800	Bob	[I] [yetone.openai.translator]: index 3: "},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" logs"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" and"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" th
info	10:02:27.492865+0800	Bob	[I] [yetone.openai.translator]: index 4: e"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" actual"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" server"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" response
info	10:02:27.585989+0800	Bob	[I] [yetone.openai.translator]: index 5: s"},"finish_reason":null}]}
info	10:02:27.586229+0800	Bob	[I] [yetone.openai.translator]: index 6: data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" are"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" consistent"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" by
info	10:02:27.618299+0800	Bob	[I] [yetone.openai.translator]: index 7: "},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" capturing"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" packets"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":"?
info	10:02:27.652914+0800	Bob	[I] [yetone.openai.translator]: index 8: "},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" However"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":","},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":"
info	10:02:27.722039+0800	Bob	[I] [yetone.openai.translator]: index 9: Bob"},"finish_reason":null}]}
info	10:02:27.722357+0800	Bob	[I] [yetone.openai.translator]: index 10: data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" is"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" using"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" a
info	10:02:27.731580+0800	Bob	[I] [yetone.openai.translator]: index 11: "},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" mainstream"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" network"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" frame
info	10:02:27.771918+0800	Bob	[I] [yetone.openai.translator]: index 12: work"},"finish_reason":null}]}
info	10:02:27.772186+0800	Bob	[I] [yetone.openai.translator]: index 13: data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":","},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" so"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" the
info	10:02:27.829998+0800	Bob	[I] [yetone.openai.translator]: index 14: re"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" should"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" be"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" n
info	10:02:27.870390+0800	Bob	[I] [yetone.openai.translator]: index 15: o"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" problem"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" with"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" mes
info	10:02:27.896538+0800	Bob	[I] [yetone.openai.translator]: index 16: sing"},"finish_reason":null}]}
info	10:02:27.896716+0800	Bob	[I] [yetone.openai.translator]: index 17: data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" up"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" the"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":" order"},"finish_reason":null}]}

data: {"id":"chatcmpl-8Qn9JflcINoizIE7KF69aGcuh44Md","object":"chat.completion.chunk","created":1701396149,"model":"gpt-3.5-turbo-1106","system_fingerprint":"fp_eeff13170a","choices":[{"index":0,"delta":{"content":"."},"finish_reas<…>
info	10:02:27.896998+0800	Bob	[I] [yetone.openai.translator]: index 18:

虽然有些 log 被省略展示了，甚至最后也没有看到 data: [DONE]，但是翻译结果是没问题的：
image

@ripperhe
Copy link
Owner

ripperhe commented Dec 2, 2023

确实，抓包结果里面数据最终会合并在一起，以致于看不出来数据传输过程是否有「截断」，不过数据返回的顺序都有问题的话，那我理解抓包里面的结果合并起来之后也应该是错乱的。

那目前看起来「截断」能兼容，但是「顺序错乱」无解。我也没啥思路，后面有啥思路再同步给你。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ripperhe @liby