Implementing ETL destinations

Thibaut Barrère edited this page Jan 18, 2018 · 3 revisions

Like sources, destinations are classes that you are providing. Destinations must implement:

  • a constructor (to which Kiba will pass the provided arguments in the DSL)
  • a write(row) method that will be called for each non-dismissed row
  • an optional close method that will be called, if present, at the end of the processing (useful to tear down resources such as connections)

Here is an example destination:

require 'csv'

# simple destination assuming all rows have the same fields
class MyCsvDestination
  attr_reader :output_file

  def initialize(output_file)
    @output_file = output_file
  end

  def write(row)
    @csv ||= CSV.open(output_file, 'w')
    unless @headers_written
      @headers_written = true
      @csv << row.keys
    end
    @csv << row.values
  end

  def close
    @csv.close
  end
end

Note that you do not have to pass the row back as a result (unlike on transforms), because the return of write(row) on a destination is currently ignored.

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.