Skip to content

Conversation

@twalthr
Copy link
Contributor

@twalthr twalthr commented Oct 30, 2018

What is the purpose of the change

This PR removes all dependencies to org.apache.commons libraries. In the past we only used a couple of methods that were partially pulled in from Hadoop causing the issues mentioned in the JIRA ticket.

Brief change log

  • Add more utility method to the Flink code base in order to reduce external dependencies

Verifying this change

This change is already covered by existing tests.
I added additional test such as EncodingUtilsTest and TypeCheckUtilsTest.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): yes
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? not applicable

Copy link
Contributor

@igalshilman igalshilman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, makes total senes to "inline" few methods and drop the dependencies 👍
Apart from few inline comments, LGTM.


private static final Base64.Decoder BASE64_DECODER = java.util.Base64.getUrlDecoder();

private static final MessageDigest MD5_MESSAGE_DIGEST = getMd5MessageDigest();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would MD5_MESSAGE_DIGEST be accessed concurrently? afaik this is not thread safe.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Usually this doesn't matter for translation (which is single threaded) but we are on the safer side if this method is thread-safe in the future.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing this.

*/
public static String repeat(final char ch, final int repeat) {
final char[] buf = new char[repeat];
for (int i = repeat - 1; i >= 0; i--) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder why the original author has decided to loop backwards :-)

}
return new String(output2);
default:
final StringBuilder buf = new StringBuilder(outputLength);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that you've just copied it, but honestly, I think that the default case is the only one that makes senes and all the complication in that method is a bit wired (as the comment above states - it is optimized for jdk1.4)
Feel free to ignore my comment, if you feel that fixing this, is out side of the scope of this PR :-)

Copy link
Contributor Author

@twalthr twalthr Oct 31, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I change this method, I would also need to introduce more extensive tests in order to verify that I did not introduce new bugs. This method can be removed soon anyway as Java 11 has a String.repeat method.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. makes senes.

@igalshilman
Copy link
Contributor

LGTM 👍

@twalthr
Copy link
Contributor Author

twalthr commented Oct 31, 2018

Thank you @igalshilman. I will merge this...

@asfgit asfgit closed this in 08d875e Oct 31, 2018
tisonkun pushed a commit to tisonkun/flink that referenced this pull request Jan 17, 2019
This commit removes all dependencies to org.apache.commons libraries in flink-table. In
the past we only used a couple of methods that were partially pulled in from Hadoop
causing the issues mentioned in the JIRA ticket.

This closes apache#6966.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants