Skip to content

Refine/Add thinking text and refine streaming chat #438

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
iceljc wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Refine/Add thinking text and refine streaming chat #438

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ceacedd
refine thinking section
Apr 7, 2026
c6ae958
add stop streaming
Apr 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@
"svelte-codemirror-editor": "^2.1.0",
"svelte-collapse": "^0.1.2",
"svelte-file-dropzone": "^2.0.2",
"svelte-hero-icons": "^5.2.0",
"svelte-i18n": "^4.0.0",
"svelte-json-tree": "^2.2.0",
"svelte-jsoneditor": "^3.11.0",
Expand Down
8 changes: 5 additions & 3 deletions src/lib/common/spinners/Loader.svelte
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,20 +6,22 @@
* disableDefaultStyles?: boolean,
* containerClasses?: string,
* containerStyles?: string,
* size?: number
* size?: number,
* color?: string
* }}
*/
let {
disableDefaultStyles = false,
containerClasses = '',
containerStyles = '',
size = 100
size = 100,
color = 'var(--bs-primary)'
} = $props();
</script>

<div
class="{disableDefaultStyles ? '' : 'loader'} {containerClasses}"
style={`${containerStyles}`}
>
<Circle {size} color="var(--bs-primary)" unit="px" duration="1s" />
<Circle {size} {color} unit="px" duration="1s" />
</div>
1 change: 1 addition & 0 deletions src/lib/helpers/types/conversationTypes.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,7 @@ IRichContent.prototype.language;
* @property {boolean} is_dummy
* @property {boolean} is_appended
* @property {string} [indication]
* @property {any} [meta_data]
*/

/**
Expand Down
3 changes: 2 additions & 1 deletion src/lib/scss/custom/pages/_chat.scss
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,8 @@
.dropdown-menu {
box-shadow: $box-shadow;
border: 1px solid var(--#{$prefix}border-color);
top: 10px !important;
top: auto !important;
bottom: 100% !important;
}
}

Expand Down
1 change: 1 addition & 0 deletions src/lib/services/api-endpoints.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ export const endpoints = {
conversationMessageDeletionUrl: `${host}/conversation/{conversationId}/message/{messageId}`,
conversationMessageUpdateUrl: `${host}/conversation/{conversationId}/update-message`,
conversationTagsUpdateUrl: `${host}/conversation/{conversationId}/update-tags`,
stopStreamingUrl: `${host}/conversation/{conversationId}/stop-streaming`,
fileUploadUrl: `${host}/agent/{agentId}/conversation/{conversationId}/upload`,
pinConversationUrl: `${host}/agent/{agentId}/conversation/{conversationId}/dashboard`,
conversationStateSearchKeysUrl: `${host}/conversation/state/keys`,
Expand Down
19 changes: 17 additions & 2 deletions src/lib/services/conversation-service.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,8 +100,9 @@ export async function getDialogs(conversationId, count = 100) {
* @param {string} conversationId - The conversation id
* @param {string} text - The text message sent to CSR
* @param {import('$conversationTypes').MessageData?} data - Additional data
* @param {boolean} isStreamingMsg - whether it is a streaming message
*/
export async function sendMessageToHub(agentId, conversationId, text, data = null) {
export async function sendMessageToHub(agentId, conversationId, text, data = null, isStreamingMsg = false) {
let url = replaceUrl(endpoints.conversationMessageUrl, {
agentId: agentId,
conversationId: conversationId
Expand All @@ -113,7 +114,8 @@ export async function sendMessageToHub(agentId, conversationId, text, data = nul
text: text,
states: totalStates,
postback: data?.postback,
input_message_id: data?.inputMessageId
input_message_id: data?.inputMessageId,
is_streaming_msg: isStreamingMsg
}).then(response => {
resolve(response?.data);
}).catch(err => {
Expand Down Expand Up @@ -293,6 +295,19 @@ export async function getAddressOptions(text) {
return response.data;
}

/**
* Stop streaming in a conversation
* @param {string} conversationId The conversation id
* @returns {Promise<{success: boolean}>}
*/
export async function stopStreaming(conversationId) {
let url = replaceUrl(endpoints.stopStreamingUrl, {
conversationId: conversationId
});
const response = await axios.post(url);
return response.data;
}

/** @type {AbortController} */
let controller = new AbortController();

Expand Down
101 changes: 78 additions & 23 deletions src/routes/chat/[agentId]/[conversationId]/chat-box.svelte
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@
uploadConversationFiles,
getAddressOptions,
pinConversationToDashboard,
stopStreaming as stopStreamingApi,
} from '$lib/services/conversation-service.js';
import {
PUBLIC_LIVECHAT_ENTRY_ICON,
Expand Down Expand Up @@ -200,7 +201,6 @@
let isListening = $state(false);
let isLite = $state(false);
let isFrame = $state(false);
let loadTextEditor = $state(false);
let autoScrollLog = $state(false);
let loadChatUtils = $state(false);
let disableSpeech = $state(false);
Expand All @@ -212,12 +212,15 @@
let copyClicked = $state(false);
let isStreaming = $state(false);
let isHandlingQueue = $state(false);
let isStopStreamClicked = $state(false);

let loadEditor = $derived(!isSendingMsg && !isThinking && loadTextEditor && messageQueue.length === 0);
let disableAction = $derived(!ADMIN_ROLES.includes(currentUser?.role || '') && currentUser?.id !== conversationUser?.id || !AgentExtensions.chatable(agent));
// let loadEditor = $derived(!isSendingMsg && !isThinking && loadTextEditor && messageQueue.length === 0);
let loadEditor = true;
let disableAction = $derived(!ADMIN_ROLES.includes(currentUser?.role || '')
&& currentUser?.id !== conversationUser?.id
|| !AgentExtensions.chatable(agent));

$effect(() => {
loadTextEditor = true;
if (loadEditor) {
focusChatTextArea();
}
Expand Down Expand Up @@ -578,7 +581,11 @@
dialogs.push({
...message,
is_chat_message: false,
is_dummy: true
is_dummy: true,
meta_data: {
...(message.meta_data || {}),
thinking_text: message.meta_data?.thinking_text || ''
}
});
}
refresh();
Expand All @@ -595,6 +602,13 @@
&& lastMsg?.is_dummy
) {
setTimeout(() => {
const thinkingText = message.meta_data?.thinking_text || '';
if (thinkingText) {
if (!dialogs[dialogs.length - 1].meta_data) {
dialogs[dialogs.length - 1].meta_data = { thinking_text: '' };
}
dialogs[dialogs.length - 1].meta_data.thinking_text += thinkingText;
}
dialogs[dialogs.length - 1].text += message.text;
refreshDialogs();
Comment on lines +605 to 613
Copy link
Copy Markdown

@qodo-code-review qodo-code-review bot Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

2. Streaming update can throw 🐞 Bug ☼ Reliability

onReceiveLlmStreamMessage() mutates dialogs[dialogs.length - 1] inside a deferred setTimeout
callback without checking that the last dialog still exists and is the expected dummy assistant
message. If dialogs is truncated/emptied concurrently, this can throw a runtime TypeError and break
streaming rendering.
Agent Prompt 
### Issue description
The streaming handler assumes `dialogs[dialogs.length - 1]` exists and is mutable, but that may be false if dialogs changes between scheduling the `setTimeout` and running it.

### Issue Context
`setTimeout(..., 0)` introduces a race window; other actions/events (e.g., truncation on message deletion) can change `dialogs` before the callback executes.

### Fix Focus Areas
- src/routes/chat/[agentId]/[conversationId]/chat-box.svelte[595-615]
- src/routes/chat/[agentId]/[conversationId]/chat-box.svelte[1239-1245]

### Suggested fix
- In the `setTimeout` callback, capture and validate the target message before mutating:
  - e.g., store `const targetId = message.message_id;` and then find the matching dummy message by id/uuid, or re-check that the last item is still `is_dummy` assistant with the same `message_id`.
- Add bounds checks:
  - `const last = dialogs.at(-1); if (!last) return;`
  - Ensure `last.text` and `last.meta_data.thinking_text` are initialized before appending.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

}, 0);
Expand Down Expand Up @@ -624,6 +638,18 @@
}

try {
const thinkingText = item.meta_data?.thinking_text || '';
if (thinkingText) {
if (!dialogs[dialogs.length - 1].meta_data) {
dialogs[dialogs.length - 1].meta_data = { thinking_text: '' };
}
for (const tt of thinkingText) {
dialogs[dialogs.length - 1].meta_data.thinking_text += tt;
refreshDialogs();
await delay(10);
}
}

for (const char of item.text) {
dialogs[dialogs.length - 1].text += char;
refreshDialogs();
Expand All @@ -642,6 +668,22 @@
refresh();
}

function stopStreaming() {
isStopStreamClicked = true;
// @ts-ignore
stopStreamingApi(page.params.conversationId).then((res) => {
if (res?.success) {
isStreaming = false;
isThinking = false;
isSendingMsg = false;
messageQueue = [];
isHandlingQueue = false;
refresh();
}
isStopStreamClicked = false;
});
Comment on lines +671 to +684
Copy link
Copy Markdown

@qodo-code-review qodo-code-review bot Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Stop click can get stuck 🐞 Bug ☼ Reliability

In chat-box.svelte, stopStreaming() only resets isStopStreamClicked inside the .then() handler, so
if stopStreamingApi rejects the UI can remain in a permanently “clicked” state with no Stop button
recovery. This can effectively dead-end the streaming control until a refresh.
Agent Prompt 
### Issue description
`stopStreaming()` sets `isStopStreamClicked = true` but never resets it if `stopStreamingApi(...)` rejects. This can leave the UI stuck in a state where the stop control never returns.

### Issue Context
The function currently uses `.then(...)` only; any thrown error or rejected promise skips resetting UI state.

### Fix Focus Areas
- src/routes/chat/[agentId]/[conversationId]/chat-box.svelte[671-685]

### Suggested fix
- Convert to `async/await` with `try { ... } catch (e) { ... } finally { isStopStreamClicked = false; }`, or add `.catch(...).finally(...)`.
- Ensure error cases either restore the Stop button (by resetting `isStopStreamClicked`) or present an error message.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

}

/** @param {import('$conversationTypes').ChatResponseModel} message */
function onIndicationReceived(message) {
isThinking = true;
Expand Down Expand Up @@ -757,8 +799,7 @@
...data,
postback: postback,
states: [
...data?.states || [],
{ key: "use_stream_message", value: PUBLIC_LIVECHAT_STREAM_ENABLED }
...data?.states || []
]
};

Expand Down Expand Up @@ -801,7 +842,7 @@
}

// @ts-ignore
await sendMessageToHub(agentId, convId, msgText, messageData);
await sendMessageToHub(agentId, convId, msgText, messageData, PUBLIC_LIVECHAT_STREAM_ENABLED === "true");
deleteMessageDraft();
isSendingMsg = false;
}
Expand Down Expand Up @@ -1973,7 +2014,7 @@
{#if message.sender.role == UserRole.Client}
<img src="images/users/user-dummy.jpg" class="rounded-circle avatar-sm" style="margin-bottom: -15px;" alt="avatar">
{:else}
{@const isShowIcon = (message?.rich_content?.message?.text || message?.text) || message?.uuid !== lastBotMsg?.uuid}
{@const isShowIcon = (message?.rich_content?.message?.text || message?.text || message?.meta_data?.thinking_text) || message?.uuid !== lastBotMsg?.uuid}
<img
class="rounded-circle avatar-sm"
style={`display: ${isShowIcon ? 'block' : 'none'}; margin-bottom: -15px;`}
Expand All @@ -1983,7 +2024,7 @@
{/if}
</div>
<div class="msg-container">
<RcMessage containerClasses={'bot-msg'} markdownClasses={'markdown-dark text-dark'} message={message} />
<RcMessage containerClasses={'bot-msg'} markdownClasses={'markdown-dark text-dark'} message={message} isStreaming={isStreaming || isThinking} />
{#if message?.message_id === lastBotMsg?.message_id && message?.uuid === lastBotMsg?.uuid}
{@const isStreamEnd = (message?.rich_content?.message?.text || message?.text) && !isStreaming && !isHandlingQueue && !isThinking}
<div style={`display: ${isStreamEnd ? 'flex' : 'none'}; gap: 10px; flex-wrap: wrap; margin-top: 5px;`}>
Expand Down Expand Up @@ -2150,7 +2191,7 @@
<ChatFileUploader
accept={'.png,.jpg,.jpeg'}
containerClasses={'line-align-center text-primary chat-util-item'}
disabled={disableAction}
disabled={isSendingMsg || isThinking || disableAction}
onfiledroped={() => refresh()}
>
<span>
Expand All @@ -2164,7 +2205,7 @@
<ChatFileUploader
accept={'.pdf,.xlsx,.xls,.csv'}
containerClasses={'line-align-center text-primary chat-util-item'}
disabled={disableAction}
disabled={isSendingMsg || isThinking || disableAction}
onfiledroped={() => refresh()}
>
<span>
Expand All @@ -2178,7 +2219,7 @@
<ChatFileUploader
accept={'.wav,.mp3'}
containerClasses={'line-align-center text-primary chat-util-item'}
disabled={disableAction}
disabled={isSendingMsg || isThinking || disableAction}
onfiledroped={() => refresh()}
>
<span>
Expand All @@ -2196,21 +2237,35 @@
onclick={() => toggleBigMessageModal()}
/>
{#if PUBLIC_LIVECHAT_FILES_ENABLED === 'true'}
<ChatUtil disabled={disableAction} onclick={() => loadChatUtils = true} />
<ChatUtil
disabled={isSendingMsg || isThinking || disableAction}
onclick={() => loadChatUtils = true}
/>
{/if}
</div>
</div>
</div>
<div class="col-auto">
<button
type="submit"
class={`btn btn-rounded chat-send waves-effect waves-light ${mode === TRAINING_MODE ? 'btn-danger' : 'btn-primary'}`}
disabled={!_.trim(text) || isSendingMsg || isThinking || disableAction}
onclick={() => sentTextMessage()}
>
<span class="d-none d-md-inline-block me-2">Send</span>
<i class="mdi mdi-send"></i>
</button>
{#if !isStopStreamClicked && isStreaming && PUBLIC_LIVECHAT_STREAM_ENABLED === 'true'}
<button
type="button"
class="btn btn-rounded chat-send waves-effect waves-light btn-danger"
aria-label="Stop streaming"
onclick={() => stopStreaming()}
>
<i class="mdi mdi-stop"></i>
</button>
{:else}
<button
type="submit"
class={`btn btn-rounded chat-send waves-effect waves-light ${mode === TRAINING_MODE ? 'btn-danger' : 'btn-primary'}`}
disabled={!_.trim(text) || isSendingMsg || isThinking || disableAction}
onclick={() => sentTextMessage()}
>
<span class="d-none d-md-inline-block me-2">Send</span>
<i class="mdi mdi-send"></i>
</button>
{/if}
</div>
</div>
</div>
Expand Down
Loading
Loading