Airrecord find_each

[JacksonRiso]

Hi team!

I am not experienced enough to write the proper tests for this and open a pull request, but I wanted to share the community my Rails inspired find_each for Airrecord. Instead of looping through the pagination and returning potentially thousands of records in one big array, this allows you to grab one page, process each record in that page, grab the next page, etc.

Example usage

Table.find_each(view: 'blah') do |record|
      puts record.id
end

For big queries and low memory servers it can really help!

Here's my config/airrecord.rb if anyone wants to add this to their application, and of course I would love to see this moved into a pull request.

Airrecord::Table.class_eval do
  def self.find_each(filter: nil, sort: nil, view: nil, offset: nil, paginate: true, fields: nil, max_records: nil, page_size: nil, &block)
    options = {}
    options[:filterByFormula] = filter if filter

    if sort
      options[:sort] = sort.map do |field, direction|
        { field: field.to_s, direction: direction }
      end
    end

    options[:view] = view if view
    options[:offset] = offset if offset
    options[:fields] = fields if fields
    options[:maxRecords] = max_records if max_records
    options[:pageSize] = page_size if page_size

    path = "/v0/#{base_key}/#{client.escape(table_name)}/listRecords"
    response = client.connection.post(path, options.to_json, { 'Content-Type' => 'application/json' })
    parsed_response = client.parse(response.body)

    if response.success?
      records = parsed_response['records']
      records = records.map do |record|
        new(record['fields'], id: record['id'], created_at: record['createdTime'])
      end
      records.each(&block) if block_given?
      if paginate && parsed_response['offset']
        find_each(
          filter: filter,
          sort: sort,
          view: view,
          paginate: paginate,
          fields: fields,
          offset: parsed_response['offset'],
          max_records: max_records,
          page_size: page_size,
          &block
        )
      end

    else
      client.handle_error(response.status, parsed_response)
    end
  end
end

Cheers!

[cloudsbird]

Interesting.. Let me test if there's a time I will create PR for this one

[papercrane-elijah]

I hope you can add this, we are currently experiencing this exact bottleneck, and a paginated result would work so much better

[Meekohi]

Part of me wonders if we could just expose the last parsed_response["offset"] so that people can "roll their own" flow (I can imagine for example someone might want a .detect like run a block until a condition is met then bail out). The records interface already accepts an offset e.g. Things.records(pagination:false, offset:'whatever') but because AirTable's offset tokens are generated on the fly you need to know what the last one was. Maybe something like

do
  goodThing = Things.records(pagination:false, offset: Things.last_offset).find { 
    |thing| isVeryGoodThing(thing) 
  }
while Things.last_offset && goodThing.nil?