Ability to stream entities

[raulpopadineti]

Hi,

We'd like to build an API that is able to paginate entities and send them in chunks.

A really raw way of sending data in chunks would be:

require 'grape'

module Stream
  class API < Grape::API
    prefix 'api'
    format :txt
    default_format :txt

    class Streamer
      def initialize(pages:, page_size:, latency:)
        @pages = pages
        @page_size = page_size
        @latency = latency
      end

      def each
        (1..@pages).lazy.each do |p|
          yield "#{p}\n" * @page_size
          sleep(@latency)
        end
      end
    end

    resources :users do
      params do
        optional :pages, type: Integer, default: 10
        optional :page_size, type: Integer, default: 10
        optional :latency, type: Float, default: 1.0
      end

      get do
        status 200
        p = declared(params)
        stream Streamer.new(pages: p.pages, page_size: p.page_size, latency: p.latency)
      end
    end
  end
end

With this approach I get the deprecation warning:

[DEPRECATION] Argument as FileStreamer-like object is deprecated. Use path to file instead.

because Grape assumes the response is a file.

Will you support streaming of a Grape::Entity or any other non-file objects in the future?

[dblock]

I think the deprecation is just for files. It's still supporting a pure streaming interface. You should look at the code a bit more in depth and see what's missing? Generally I don't see why we wouldn't want this, so I'll leave this open.

[lfidnl]

Hello. I guess, now stream method can stream only files. Maybe we should to do:

1. Add option stream: true to file method. This option will replace current stream method.
2. Rewrite stream method in such a way that it can receive object which respond to each method and streams it.

[raulpopadineti]

@dblock @lfidnl I've managed to stream JSON by polishing a bit on the Streamer class I added:

class Streamer
      def initialize(entity_class:, enum:, root:, options:)
        @entity_class = entity_class
        @enum = enum
        @root = root
        @options = options
      end

      def each
        yield "{\n  \"#{@root}\": [\n    "
        first = true
        @enum.lazy.each do |object|
          buffer = ''
          buffer << ",\n  " unless first
          first = false
          data = represent_entity(object, @options).as_json
          buffer << JSON.pretty_generate(data)[1..-2].strip
          yield buffer
          # sleep 1
        end
        yield "  \n  ]\n}\n"
      end

      def represent_entity(objects, options = {})
        if objects.respond_to?(:to_ary)
          inner = objects.to_ary.map { |object| @entity_class.new(object, options.reverse_merge(collection: true)).presented }
        else
          inner = @entity_class.new(objects, options).presented
        end
        inner
      end
    end

Basically I overwritten the default represent method because the root_element is added in the each method. I'm also passing the objects as an Enumerator with lazy loading for memory optimization.

Let me know if you have any other suggestion or how can I improve it so I can wrap it all up and submit a PR.

[lfidnl]

@raulpopadineti one question: is it works now with deprecation warning, or not?

[raulpopadineti]

@lfidnl it works with deprecation warning.

[lfidnl]

@raulpopadineti good, I think, @dblock should approve my suggestion (or change it) and anyone can create PR for fix this issue :)

[dblock]

I think what you suggest is sensible, would like to see a PR.

[raulpopadineti]

@dblock to which solution are you referring?

[dblock]

I meant what @lfidnl says. But you're all looking at code, I suggest trying something and PRing it.

[code-bunny]

For those interested that may use JSON API, i have took the ideas here and built a basic gem https://github.com/Cabbit/grape_json_api_streamer for grape that i'll expand upon as my projects get more complex.

[dblock]

@cabbit contribute that to http://www.ruby-grape.org/projects/ pls.

[dm1try]

fixed in the referenced PR