fix: drain entire queue per flush cycle

[heyitsaamir]

Removes the 10-item batch cap in HttpStream flush. Previously, each flush only processed 10 queued items, causing a long streaming tail when the LLM finishes faster than chunks are sent to Teams.

[heyitsaamir]

This change is part of the following stack:

• feat: graceful stream cancellation on 403 #513
  • fix: drain entire queue per flush cycle #520 ◀

_{Change managed by git-spice.}

[rajan-chari]

Interesting. Some thoughts as to why this limit might be there and also rebuttals. TLDR. seems like a clear win to me.

1. Perhaps there was concern about this text getting too long?
             this.text += activity. Text;
   However we just keep adding to it afik so not clear why limit it to 10.

2. Keep the user waiting while we append strings
   String append operation is orders of magnitude faster than the streaming so the user does not have to wait

Verdict: PR seems like a clear win.

Other future improvement: I remember seeing something about 1/sec update limit. However, origin for this seems unclear. Changing that to 1/500 ms would also streaming responsiveness benefits.

[rajan-chari]

would love to hear from @aacebo

[rajan-chari]

Thanks @heyitsaamir

[heyitsaamir]

@copilot can you go through the history of that file and figure out why that chunk limit was added? Use old PRs if needed too.

[Copilot]

@copilot can you go through the history of that file and figure out why that chunk limit was added? Use old PRs if needed too.

Here's what the git history shows:

The 10-item cap was never an intentional design decision — it was a simplification that slipped in during PR #256 ("fix streaming retry logic", commit ada46913) by @aacebo.

Original code (from the very first commit 7f38971d): The flush used a proportional limit — processing ~10% of the queue per flush cycle:

const size = Math.round(this.queue.length / 10);
while (this.queue.length && i <= size) {

In PR #256 (commit ada46913), the primary fix was correcting how promises are passed to the retry helper: promises.retry(this.send(activity)) → promises.retry(() => this.send(activity)) (a real bug — reused rejected promises). As part of that refactor, the dynamic Math.round(this.queue.length / 10) was silently replaced with a hardcoded i < 10:

-    const size = Math.round(this.queue.length / 10);
-    while (this.queue.length && i <= size) {
+    while (this.queue.length && i < 10) {

This turned a proportional 10%-per-flush rule into a fixed hard cap of 10 items — even if the queue had 200+ items. The original proportional logic would have also had a tail problem (it would always leave ~90% behind), but the fixed cap made it worse and more predictable in a bad way.

Since each flush cycle consolidates all dequeued items into a single TypingActivity chunk, there's no good reason to cap how much of the queue is drained per cycle. This PR removes that cap entirely.

The base branch was changed.