[GOBBLIN-267] Changed workunit creation policy to compare update time with maxLookBackDays

[aditya1105]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-267

Description

• Here are some details about my PR, including screenshots (if applicable):
  HiveSource#shouldCreateWorkunit() now compares update time with maxLookBackDays

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  Very trivial change/fix

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[aditya1105]

@abti, can you please review this pr

[erwa]

Is updateTime here the same as the partition value (i.e. the time corresponding to the partition value YYYY-MM-DD-HH)? If this is the last time the partition was updated, then I think you might still have a problem. For example, imagine 100 days of partitions are registered today and maxLookBackTime is 5 days ago. The last time all 100 partitions were updated would be > maxLookBackTime and would thus still have workunits created for them.

[aditya1105]

Update time is controllable via policies.Following are the policies we have:

1. DatePatternUpdateProvider
2. HdfsBasedUpdateProvider
3. HiveMetastoreBasedUpdateProvider

Hence using the 1 and 2 as update policies will make sure workunits won't be created in the scenario mentioned above.
If 3 is used, then both create time and update time should be taken by hive partition property and hence workunits must be created.

[htran1]

+1.