[CELEBORN-1489] Update Flink support with authentication support#2596
Closed
mridulm wants to merge 4 commits intoapache:mainfrom
Closed
[CELEBORN-1489] Update Flink support with authentication support#2596mridulm wants to merge 4 commits intoapache:mainfrom
mridulm wants to merge 4 commits intoapache:mainfrom
Conversation
Contributor
Author
|
+CC @otterc, @SteNicholas, @RexXiong |
Contributor
Author
|
+CC @venkata91 |
mridulm
commented
Jul 1, 2024
| LifecycleManager manager = lifecycleManager; | ||
| if (null != manager) { | ||
| manager.stop(); | ||
| } |
Contributor
Author
There was a problem hiding this comment.
This is an unrelated minor fix for NPE
mridulm
commented
Jul 1, 2024
| try { | ||
| return createClient(remoteHost, remotePort); | ||
| } catch (IOException e) { | ||
| } catch (Exception e) { |
Contributor
Author
There was a problem hiding this comment.
SASL failures are not checked exceptions - we have to retry for those too.
mridulm
commented
Jul 1, 2024
| @Override | ||
| @VisibleForTesting | ||
| public TransportClientFactory getDataClientFactory() { | ||
| initializeTransportClientFactoryIfRequired(); |
Contributor
Author
There was a problem hiding this comment.
getDataClientFactory gets invoked without setupLifecycleManagerRef primarily for tests.
mridulm
commented
Jul 1, 2024
mridulm
commented
Jul 1, 2024
| } catch (IOException e) { | ||
| throw new RuntimeException("Failed to resolve path " + path, e); | ||
| } | ||
| } |
Contributor
Author
There was a problem hiding this comment.
This is strictly not required for this PR - but keeps failing in ApiBaseResourceAuthenticationSuite (with maven) without it.
mridulm
commented
Jul 1, 2024
| incCounter(ACTIVE_CONNECTION_COUNT, -1) | ||
| applicationIds.asScala.foreach(applicationId => | ||
| incCounter(ACTIVE_CONNECTION_COUNT, -1, Map(applicationLabel -> applicationId))) | ||
| if (null != applicationIds) { |
Contributor
Author
There was a problem hiding this comment.
This was causing NPE in the tests
Thanks for tagging me here. Will review the PR shortly. |
venkata91
suggested changes
Jul 2, 2024
venkata91
left a comment
venkata91
left a comment
There was a problem hiding this comment.
Overall lgtm except for some minor comments. This looks great, thanks for adding SASL auth support for Flink with Celeborn.
...mmon/src/main/java/org/apache/celeborn/plugin/flink/network/FlinkTransportClientFactory.java
Outdated
Show resolved
Hide resolved
...common/src/main/java/org/apache/celeborn/plugin/flink/readclient/FlinkShuffleClientImpl.java
Outdated
Show resolved
Hide resolved
...common/src/main/java/org/apache/celeborn/plugin/flink/readclient/FlinkShuffleClientImpl.java
Outdated
Show resolved
Hide resolved
...common/src/main/java/org/apache/celeborn/plugin/flink/readclient/FlinkShuffleClientImpl.java
Outdated
Show resolved
Hide resolved
Contributor
Author
|
Thanks @venkata91, @RexXiong ... have addressed the comments. |
venkata91
approved these changes
Jul 3, 2024
RexXiong
approved these changes
Jul 4, 2024
Contributor
Author
|
Thanks @venkata91, @RexXiong ! |
Contributor
Author
|
Let me backport this to 0.5 branch as well @RexXiong |
mridulm
pushed a commit
to mridulm/celeborn
that referenced
this pull request
Jul 4, 2024
Merge conflicts in branch 0.5 Fix authentication support for Apache Flink. Without these changes, Apache Flink applications fail when Celeborn cluster has authentication enabled. Fixes authentication support for Apache Flink integration This is forward port + adaptation of changes we did internally (against 0.4) when testing Apache Flink applications against Celeborn cluster with authentication (and TLS) enabled. Integration test has been updated to additionally test for Flink applications with authentication enabled in Celeborn cluster. Closes apache#2596 from mridulm/fix-flink-auth-support. Authored-by: Mridul Muralidharan <mridulatgmail.com> Signed-off-by: Shuang <lvshuang.xjs@alibaba-inc.com>
mridulm
pushed a commit
that referenced
this pull request
Jul 4, 2024
…ckport to 0.5 Backport of #2596 to branch-0.5 Conflicts were in: * ApiBaseResourceAuthenticationSuite.scala * ApiBaseResourceSuite.scala Description from #2596: ### What changes were proposed in this pull request? Fix authentication support for Apache Flink. ### Why are the changes needed? Without these changes, Apache Flink applications fail when Celeborn cluster has authentication enabled. ### Does this PR introduce _any_ user-facing change? Fixes authentication support for Apache Flink integration ### How was this patch tested? This is forward port + adaptation of changes we did internally (against 0.4) when testing Apache Flink applications against Celeborn cluster with authentication (and TLS) enabled. Integration test has been updated to additionally test for Flink applications with authentication enabled in Celeborn cluster. Closes #2603 from mridulm/fix-flink-auth-support-v0.5. Authored-by: Mridul Muralidharan <mridulatgmail.com> Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Fix authentication support for Apache Flink.
Why are the changes needed?
Without these changes, Apache Flink applications fail when Celeborn cluster has authentication enabled.
Does this PR introduce any user-facing change?
Fixes authentication support for Apache Flink integration
How was this patch tested?
This is forward port + adaptation of changes we did internally (against 0.4) when testing Apache Flink applications against Celeborn cluster with authentication (and TLS) enabled.
Integration test has been updated to additionally test for Flink applications with authentication enabled in Celeborn cluster.