Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[GOBBLIN-267] Changed workunit creation policy to compare update time with maxLookBackDays #2119

Closed
wants to merge 1 commit into from
Closed

[GOBBLIN-267] Changed workunit creation policy to compare update time with maxLookBackDays #2119

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
5 changes: 4 additions & 1 deletion ...t/src/main/java/org/apache/gobblin/data/management/conversion/hive/source/HiveSource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -386,10 +386,13 @@ protected boolean shouldCreateWorkunit(Partition sourcePartition, LongWatermark

/**
* Check if workunit needs to be created. Returns <code>true</code> If the
* <code>updateTime</code> is greater than the <code>lowWatermark</code>.
* <code>updateTime</code> is greater than the <code>lowWatermark</code> and <code>maxLookBackTime</code>
* <code>createTime</code> is not used. It exists for backward compatibility
*/
protected boolean shouldCreateWorkunit(long createTime, long updateTime, LongWatermark lowWatermark) {
if (new DateTime(updateTime).isBefore(this.maxLookBackTime)) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is updateTime here the same as the partition value (i.e. the time corresponding to the partition value YYYY-MM-DD-HH)? If this is the last time the partition was updated, then I think you might still have a problem. For example, imagine 100 days of partitions are registered today and maxLookBackTime is 5 days ago. The last time all 100 partitions were updated would be > maxLookBackTime and would thus still have workunits created for them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update time is controllable via policies.Following are the policies we have:

  1. DatePatternUpdateProvider
  2. HdfsBasedUpdateProvider
  3. HiveMetastoreBasedUpdateProvider

Hence using the 1 and 2 as update policies will make sure workunits won't be created in the scenario mentioned above.
If 3 is used, then both create time and update time should be taken by hive partition property and hence workunits must be created.

return false;
}
return new DateTime(updateTime).isAfter(lowWatermark.getValue());
}

Expand Down