feat(dart_frog): allow disabling Response buffer output

[luiscib3r]

Status

READY

Description

Allows to configure the shelf.io.buffer_output parameter in the context of the shelf.Response to allow returning a streamed response as explained in this issue: dart-lang/shelf#54 (comment)

Type of Change

• ✨ New feature (non-breaking change which adds functionality)
• 🛠️ Bug fix (non-breaking change which fixes an issue)
• ❌ Breaking change (fix or feature that would cause existing functionality to change)
• 🧹 Code refactor
• ✅ Build configuration change
• 📝 Documentation
• 🗑️ Chore

[luiscib3r]

Hi. I am using dart frog to build a service that communicates with the OpenAI/Together AI api. So these APIs allow you to obtain the response from the LLM in streaming. A response with 'Content-Type': 'text/event-stream; charset=utf-8' is used to send each token as soon as it is generated. I need to resend that streamed response using dart frog. I created a little example code:

import 'package:dart_frog/dart_frog.dart';

Response onRequest(RequestContext context) {
  return Response.stream(
    body: stream(),
    headers: {
      'Content-Type': 'text/event-stream; charset=utf-8',
      'X-Content-Type-Options': 'nosniff',
    },
    useBufferOutput: false,
  );
}

Stream<List<int>> stream() async* {
  const message = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. '
      'Sed non risus. Suspendisse lectus tortor, dignissim sit amet, '
      'adipiscing nec, ultricies sed, dolor. Cras elementum ultrices diam. '
      'Maecenas ligula massa, varius a, semper congue, euismod non, mi. '
      'Proin porttitor, orci nec nonummy molestie, enim est eleifend mi, '
      'non fermentum diam nisl sit amet erat. Duis semper. Duis arcu massa, '
      'scelerisque vitae, consequat in, pretium a, enim. Pellentesque congue. '
      'Ut in risus volutpat libero pharetra tempor. Cras vestibulum bibendum '
      'augue. Praesent egestas leo in pede. Praesent blandit odio eu enim. '
      'Pellentesque sed dui ut augue blandit sodales. Vestibulum ante ipsum '
      'primis in faucibus orci luctus et ultrices posuere cubilia Curae; '
      'Aliquam nibh. Mauris ac mauris sed pede pellentesque fermentum. '
      'Maecenas adipiscing ante non diam sodales hendrerit.';

  final words = message.split(' ');

  for (final word in words) {
    yield 'data: $word\n\n'.codeUnits;
    await Future<void>.delayed(const Duration(milliseconds: 300));
  }
}

Without setting the useBufferOutput parameter to false, the response is not returned until the stream has been closed. When the useBufferOutput is false then the response is sent immediately and keeps transmitting new events from the stream until it closes.