How do I respond to stream results via API - Grape

[datpmt]

Recently I have been using the ChatGPT API and I've found a very useful method of sending information called streaming. Instead of sending a long piece of text all at once, ChatGPT sends it in small chunks until it's finished. For example, the text "Hello! How can I assist you today?" would be divided as follows:
0.1s: Hello
0.2s: !
0.3s: How
...
0.9s: today
1s: ?
Thank you for everyone's contributions!

# /api/v1/chat_gpt
module API
  module V1
    class AttachmentStream
      attr_reader :response
      def initialize(response)
        @response = response
      end

      def each
        yield response.read_body
      end
    end
    class ChatGpt < Grape::API
      helpers API::V1::Helpers::Authentication
      helpers API::V1::Helpers::ChatGpt
      resource :chat_gpt do
        desc 'Chat with ChatGPT'
        params do
          use :authorization_token
          requires :messages, type: Array[Hash], desc: 'Chat Content'
          optional :max_tokens, type: Integer
          optional :stream, type: Boolean, default: false
        end
        post do
          content_type 'text/event-stream'
          url = 'https://api.openai.com/v1/chat/completions'
          uri = URI(url)

          data = {
            model: 'gpt-3.5-turbo',
            stream: true,
            messages: [
              {
                  "role": "user",
                  "content": "hi"
              }
            ]
          }

          Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http|
            request = Net::HTTP::Post.new(uri.path)

            # Set any request headers if required
            request['Content-Type'] = 'application/json'
            request['Authorization'] = "Bearer #{ENV.fetch('GPT_KEY')}"
            # Set the request body if required
            request.body = data.to_json

            http.request(request) do |response|
              stream AttachmentStream.new(response)
            end
          end
        end
      end
    end
  end
end

• They seem to be returning at the same time :<
  
• Via OPENAI
  

[dblock]

Can you please elaborate a little? I am not sure what "respond to stream results" means. Are you trying to do something on the server or client? What is not working in the code you've provided?

[datpmt]

I want the results returned from my API to be the same as the results returned from OpenAI.
The first screenshot is the return from my API. And it is returning at the same time, not sequentially until the end of the transfer process like OpenAI (2nd screenshot). I have researched many articles about streaming but I haven't found any that can help me complete it. Thank you for your interest!

[dblock]

I see. Start by doing this without Grape.

Does Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http| actually consume responses in parallel? According to https://stackoverflow.com/questions/35495016/stream-the-response-body-of-an-http-get-to-an-http-post-with-ruby it may not. Start by showing code that is 100% performing a streaming request/response with OpenAPI in Ruby, then we can figure out how Grape should respond without collecting that data as a stream as well.

[datpmt]

hi @dblock
I'm sure about Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http| actually consuming responses in parallel. According to https://stackoverflow.com/questions/41306082/ruby-nethttp-read-the-header-before-the-body-without-head-request

net/http supports streaming, you can use this to read the header before the body.

But I haven't seen any article saying it can respond to streaming endpoint so I need your help.

[datpmt]

And Grape Readme.md has this one:

But it also seems to return at the same time, I don't know what to do it returns one piece every second

# /api/v1/chat_gpt
module API
  module V1
    class MyStream
      def each
        3.times do
          sleep 1
          yield "data: #{{ time: Time.zone.now }.to_json}\n\n"
        end
      end
    end

    class ChatGpt < Grape::API
      resource :chat_gpt do
        post do
          content_type 'text/event-stream'
          stream MyStream.new
          status 200
        end
      end
    end
  end
end

[dblock]

I'm sure about Net::HTTP.start(uri.host, uri.port, use_ssl: true) do |http| actually consuming responses in parallel.

Care to show a working example with OpenAPI in pure ruby?

But it also seems to return at the same time, I don't know what to do it returns one piece every second

I don't know either. Let's figure it out. Looks like @urkle contributed much of the streaming implementation in #1520, maybe he is around to help?

[dblock]

I know what the problem is @datpmt - what web server are you using? I got it working with Puma, but not Webrick.
I have a working sample in ruby-grape/grape-on-rack#56.

Let us know if this helps?

rack/rack#396 (comment)

[datpmt]

Thank you for your help @dblock !
I use Puma too and I work with the Rails app (~> 6.1.5) not using either rack or rackup

When I create the new file test.rb (code bellow) and start the server by command line:

ruby test.rb

# test.rb
require 'grape'
require 'rack'
require 'rack/handler/puma'

class MyStream
  def each
    3.times do
      yield "data: #{{ time: Time.now }.to_json}\n\n"
      sleep 1
    end
  end
end

class ChatGpt < Grape::API
  resource :chat_gpt do
    get do
      stream MyStream.new
      content_type 'text/event-stream'
      status 200
    end
  end
end

Rack::Handler::Puma.run ChatGpt.new

It working well

Screen.Recording.2023-11-09.at.13.50.02.mov

But when I start my main project (with Rails app) by command line:

rails s -p 3005

Code in my Rails app:

# /api/v1/chat_gpt
module API
  module V1
    class MyStream
      def each
        3.times do
          yield "data: #{{ time: Time.zone.now }.to_json}\n\n"
          sleep 1
        end
      end
    end

    class ChatGpt < Grape::API
      resource :chat_gpt do
        get do
          stream MyStream.new
          content_type 'text/event-stream'
          status 200
        end
      end
    end
  end
end

There seems to be no difference. I tried adding gem 'rack' to my Gemfile without success.

Screen.Recording.2023-11-09.at.13.54.33.mov

I think the problem is related to Rails or Rack, do you have any ideas?
Thank you again for your interest!

[dblock]

Let's try to isolate it. Care to port my sample that I added in ruby-grape/grape-on-rack#56 to https://github.com/ruby-grape/grape-on-rails? Make a (non-working) PR?

[datpmt]

hi @dblock
If you have time, I need the sample at https://github.com/ruby-grape/grape-on-rails
Thank you again for your interest!

[dblock]

If you have time, I need the sample at https://github.com/ruby-grape/grape-on-rails

I do not, but you should find time to write and, and I will find time to dig into why it doesn't work

[dblock]

I am going to close the issue here since it's clearly not a Grape problem, as grape returns all the right things per spec and works on Puma.

[datpmt]

According to rails/rails#38780 (comment), this is not a Rails problem but an inconsistent Rack version.

[dblock]

Did upgrading Rack to >2.2 or <=2.1 fix it in your application?

[datpmt]

Did upgrading Rack to >2.2 or <=2.1 fix it in your application?

Downgrade rack (2.2.4) to rack (2.1.4.3) it works fine!