Skip to content

Commit

Permalink
fix additional row processors to ignore blank rows
Browse files Browse the repository at this point in the history
  • Loading branch information
cqr committed Oct 21, 2010
1 parent 158d4f0 commit 26cb46d
Show file tree
Hide file tree
Showing 7 changed files with 6 additions and 2 deletions.
1 change: 1 addition & 0 deletions lib/etl/processor/check_unique_processor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@ def compound_key_constraints
# Process the row. This implementation will only return a row if it
# it's key combination has not already been seen.
def process(row)
return row unless row
key = (keys.collect { |k| row[k] }).join('|')
unless compound_key_constraints[key]
compound_key_constraints[key] = 1
Expand Down
1 change: 1 addition & 0 deletions lib/etl/processor/copy_field_processor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ module Processor #:nodoc:
class CopyFieldProcessor < ETL::Processor::RowProcessor
# Process the given row
def process(row)
return row unless row
destination = (configuration[:destination] || configuration[:dest])
source_value = row[configuration[:source]]
case source_value
Expand Down
1 change: 1 addition & 0 deletions lib/etl/processor/hierarchy_exploder_processor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ def initialize(control, configuration={})

# Process the row expanding it into hierarchy values
def process(row)
return row unless row
rows = []
target = configuration[:target]
table = configuration[:table]
Expand Down
1 change: 1 addition & 0 deletions lib/etl/processor/rename_processor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ module Processor #:nodoc:
# * <tt>:dest</tt>: The destination field name
class RenameProcessor < ETL::Processor::RowProcessor
def process(row)
return row unless row
source_value = row[configuration[:source]]
case source_value
when Numeric
Expand Down
1 change: 1 addition & 0 deletions lib/etl/processor/require_non_blank_processor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ def initialize(control, configuration)

# Process the row.
def process(row)
return row unless row
fields.each { |field| return if row[field].blank? }
row
end
Expand Down
1 change: 1 addition & 0 deletions lib/etl/processor/sequence_processor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ module Processor #:nodoc:
# * <tt>:dest</tt>: The destination field name
class SequenceProcessor < ETL::Processor::RowProcessor
def process(row)
return row unless row
sequences[configuration[:context]] ||= 0
row[configuration[:dest]] = sequences[configuration[:context]] += 1
row
Expand Down
2 changes: 0 additions & 2 deletions lib/etl/processor/surrogate_key_processor.rb
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,7 @@ def initialize(control, configuration)
# Add a surrogate key to the row
def process(row)
if row
#puts "processing row #{row.inspect}"
@surrogate_key += 1
#puts "adding surrogate key to row: #{@surrogate_key}"
row[destination] = @surrogate_key
row
end
Expand Down

1 comment on commit 26cb46d

@thbar
Copy link

@thbar thbar commented on 26cb46d Jun 10, 2011

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to self - maybe this check could be done in the caller ? I'll see how to do that.

Please sign in to comment.