Skip to content

Conversation

@lchik
Copy link
Contributor

@lchik lchik commented Jun 23, 2021

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

Description

  • Here are some details about my PR, including screenshots (if applicable):
    In MySQL Connector 8, calling ResultSet.getString(int) on a record with a zero timestamp
    returns the string "0000-00-00 00:00:00", even when the connection property
    zeroDateTimeBehavior=CONVERT_TO_NULL is set.
    This change is a workaround to return null for these timestamps when the property is set.

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

batchSize = (batchSize == 0 ? ConfigurationKeys.DEFAULT_SOURCE_FETCH_SIZE : batchSize);

String sourceConnProps = this.workUnitState.getProp(ConfigurationKeys.SOURCE_CONN_PROPERTIES);
boolean convertZeroDateTimeToNull = sourceConnProps != null
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think here we can generalize this a bit more rather than keep it only to the zeroDateTime behavior, to support the other zeroDateTime properties if there are any users. We can check that sourceConnProps contains zeroDateTimeBehavior, then parse it as timestamp and return the result. This way it can handle the ROUND case

@Will-Lo
Copy link
Contributor

Will-Lo commented Jun 23, 2021

Thanks for the fix! LGTM

Copy link
Contributor

@autumnust autumnust left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for fixing, the change itself LGTM.

Since we have org.apache.gobblin.source.jdbc.JdbcExtractorTest, can we add a unit test for the change we made here?

@codecov-commenter
Copy link

codecov-commenter commented Jun 23, 2021

Codecov Report

Merging #3319 (91d43ba) into master (3973a84) will increase coverage by 0.10%.
The diff coverage is 66.66%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master    #3319      +/-   ##
============================================
+ Coverage     46.30%   46.40%   +0.10%     
- Complexity     9972    10001      +29     
============================================
  Files          2031     2031              
  Lines         78994    79002       +8     
  Branches       8831     8834       +3     
============================================
+ Hits          36575    36661      +86     
+ Misses        39010    38938      -72     
+ Partials       3409     3403       -6     
Impacted Files Coverage Δ
.../org/apache/gobblin/source/jdbc/JdbcExtractor.java 10.54% <66.66%> (+1.13%) ⬆️
...management/partition/CopyableDatasetRequestor.java 35.48% <0.00%> (-9.68%) ⬇️
...in/java/org/apache/gobblin/cluster/HelixUtils.java 32.23% <0.00%> (-5.79%) ⬇️
...obblin/data/management/copy/CopyConfiguration.java 80.43% <0.00%> (-2.18%) ⬇️
.../org/apache/gobblin/metrics/RootMetricContext.java 78.12% <0.00%> (-1.57%) ⬇️
...nt/copy/writer/FileAwareInputStreamDataWriter.java 74.37% <0.00%> (+0.50%) ⬆️
.../org/apache/gobblin/cluster/GobblinTaskRunner.java 63.46% <0.00%> (+0.96%) ⬆️
...pache/gobblin/data/management/copy/CopySource.java 71.13% <0.00%> (+1.03%) ⬆️
...lin/elasticsearch/writer/FutureCallbackHolder.java 62.85% <0.00%> (+1.42%) ⬆️
... and 11 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3973a84...91d43ba. Read the comment docs.

Copy link
Contributor

@autumnust autumnust left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM !

@autumnust
Copy link
Contributor

Is the failing tests relevant ?
Seems this is failing: :gobblin-rest-service:gobblin-rest-api:generateDataTemplate

@lchik lchik force-pushed the zero-datestamps branch from 7af4324 to 5f85371 Compare June 25, 2021 17:40
Lemuel Chik added 3 commits June 25, 2021 10:49
In MySQL Connector 8, calling ResultSet.getString(int) on a record with a zero timestamp
returns the string "0000-00-00 00:00:00", even when the connection property
`zeroDateTimeBehavior=CONVERT_TO_NULL` is set.
This change is a workaround to return null for these timestamps when the property is set.
@lchik lchik force-pushed the zero-datestamps branch from 5f85371 to 0524093 Compare June 25, 2021 17:49
@autumnust autumnust merged commit 2407be6 into apache:master Jun 28, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants