Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Inference doesn't show whole text during incremental updates. #2354

Closed
kpoeppel opened this issue Apr 6, 2023 · 7 comments · Fixed by #2458
Closed

Inference doesn't show whole text during incremental updates. #2354

kpoeppel opened this issue Apr 6, 2023 · 7 comments · Fixed by #2458
Labels
bug Something isn't working inference website
Milestone
Inference Pipelin...

Comments

@kpoeppel
Copy link
Contributor

kpoeppel commented Apr 6, 2023

In the currently deployed version on open-assistant.io/chat, the incremental inference updates start from some later tokens.

  1. During generation:
    Screenshot at 2023-04-06 19-23-25
  2. Final response:
    Screenshot at 2023-04-06 19-23-31

In case this isn't resolved yet, I can try to tackle this.
@notmd
Copy link
Collaborator

notmd commented Apr 6, 2023

Which browser are you using?

@kpoeppel
Copy link
Contributor Author

kpoeppel commented Apr 6, 2023

Firefox, version 111.0.1, Linux Mint

@andreaskoepf
Copy link
Collaborator

Firefox

Can confirm this defect for Firefox (same version) on Ubuntu.

@andreaskoepf andreaskoepf added bug Something isn't working website inference labels Apr 6, 2023
@andreaskoepf andreaskoepf added this to the Inference Pipeline MVP milestone Apr 6, 2023
@echo0x22
Copy link
Contributor

echo0x22 commented Apr 6, 2023

The problem is also observed in Chromium on Android and Chromium on Linux

@olliestanley olliestanley mentioned this issue Apr 7, 2023
Closed
@olliestanley
Copy link
Collaborator

Another user in #2374 reported that this seems to occur when there is a comma near the start - all text before the comma is not visible until the generation is complete.

@Jmshaver
Copy link

Jmshaver commented Apr 7, 2023

Similar problem with Chromium on Mac. All text before an apostrophe was not visible until the entire generation was complete

@planetoryd
Copy link

planetoryd commented Apr 8, 2023 •
edited
Loading

This problem is especially apparent when talking in Chinese

Image

Image

Firefox 111.0.1 on Kubuntu

kpoeppel added a commit to kpoeppel/Open-Assistant that referenced this issue Apr 11, 2023
@kpoeppel
Fix LAION-AI#2354, introduce line-buffering + yield multiple objects …
890092f 
…from list in chat_stream.ts
@kpoeppel kpoeppel mentioned this issue Apr 11, 2023
Merged
@notmd notmd linked a pull request Apr 11, 2023 that will close this issue
Fix #2354, introduce line-buffering + yield multiple objects from list in chat_stream.ts #2458
Merged
AbdBarho pushed a commit to kpoeppel/Open-Assistant that referenced this issue Apr 14, 2023
@kpoeppel @AbdBarho
Fix LAION-AI#2354, introduce line-buffering + yield multiple objects …
3e924f0 
…from list in chat_stream.ts
AbdBarho pushed a commit that referenced this issue Apr 14, 2023
@kpoeppel
Fix #2354, introduce line-buffering + yield multiple objects from lis…
322c785 
…t in chat_stream.ts (#2458)

I found two bugs in chat_stream.ts, one is the possibility of incomplete
lines, so the `iteratorSSE` has to buffer those. The second is the main
cause for issue #2354, the Object.fromEntries dictionary creator
iteratively overwrites 'data' keys, so the yield has to be wrapped in a
loop for yielding all lines.

This PR should resolve issue #2354
@olliestanley olliestanley mentioned this issue Apr 14, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working inference website
Projects
Open-Assistant
Archived in project
Milestone
Inference Pipeline MVP
7 participants
@olliestanley @andreaskoepf @notmd @kpoeppel @planetoryd @Jmshaver @echo0x22