Skip to content

Commit

Permalink
[GOBBLIN-267] Changed workunit creation policy to compare update time…
Browse files Browse the repository at this point in the history
… with maxLookBackDays

Closes apache#2119 from aditya1105/avro-to-orc
  • Loading branch information
aditya1105 authored and zxliucmu committed Nov 16, 2017
1 parent eb18d2c commit 0d96214
Showing 1 changed file with 4 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -386,10 +386,13 @@ protected boolean shouldCreateWorkunit(Partition sourcePartition, LongWatermark

/**
* Check if workunit needs to be created. Returns <code>true</code> If the
* <code>updateTime</code> is greater than the <code>lowWatermark</code>.
* <code>updateTime</code> is greater than the <code>lowWatermark</code> and <code>maxLookBackTime</code>
* <code>createTime</code> is not used. It exists for backward compatibility
*/
protected boolean shouldCreateWorkunit(long createTime, long updateTime, LongWatermark lowWatermark) {
if (new DateTime(updateTime).isBefore(this.maxLookBackTime)) {
return false;
}
return new DateTime(updateTime).isAfter(lowWatermark.getValue());
}

Expand Down

0 comments on commit 0d96214

Please sign in to comment.