Skip to content

[LIVY-697] Rsc client cannot resolve the hostname of driver in yarn-cluster mode#246

Closed
runzhiwang wants to merge 1 commit into
apache:masterfrom
runzhiwang:hostname-2-ip
Closed

[LIVY-697] Rsc client cannot resolve the hostname of driver in yarn-cluster mode#246
runzhiwang wants to merge 1 commit into
apache:masterfrom
runzhiwang:hostname-2-ip

Conversation

@runzhiwang

@runzhiwang runzhiwang commented Oct 14, 2019

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

[LIVY-697] Rsc client cannot resolve the hostname of driver in yarn-cluster mode

  1. The content of Driver in /etc/hosts are as follows:
    127.0.0.1 localhost
    10.10.10.10 test_hostname

  2. The content of Driver in /etc/hostname are as follows:
    test_hostname

  3. The findLocalAddress method in livy cannot return 10.10.10.10, but return test_hostname.
    Because the test_hostname point to 10.10.10.10, which doesn't pass the check
    address.isLoopbackAddress(), so findLocalAddress return test_hostname .

  4. The rsc client cannot resolve the test_hostname, which cause rsc client cannot connect to
    driver.

  5. The findLocalAddress method in livy can return 10.10.10.10 as expected if the content
    of Driver in /etc/hosts are as follows, which is not correct in our environment.
    127.0.0.1 localhost
    127.0.0.1 test_hostname

  6. Though I can modify the findLocalAddress method to return 10.10.10.10, but it maybe cause
    error if the machine has multiple network cards. So, rsc client gets the driver ip from the
    connection.

How was this patch tested?

  1. The content of Driver in /etc/hosts are as follows:
    127.0.0.1 localhost
    127.0.0.1 test_hostname

  2. The content of Driver in /etc/hostname are as follows:
    test_hostname

  3. rsc client can get the driver ip in connection, and connect to driver successfully.

@runzhiwang runzhiwang closed this Oct 14, 2019
@runzhiwang runzhiwang reopened this Oct 14, 2019
@codecov-io

codecov-io commented Oct 14, 2019

Copy link
Copy Markdown

Codecov Report

Merging #246 into master will increase coverage by 0.04%.
The diff coverage is 100%.

Impacted file tree graph

@@             Coverage Diff              @@
##             master     #246      +/-   ##
============================================
+ Coverage     68.08%   68.12%   +0.04%     
  Complexity      939      939              
============================================
  Files           101      101              
  Lines          5853     5855       +2     
  Branches        886      886              
============================================
+ Hits           3985     3989       +4     
+ Misses         1296     1295       -1     
+ Partials        572      571       -1
Impacted Files Coverage Δ Complexity Δ
...main/java/org/apache/livy/rsc/ContextLauncher.java 66.34% <100%> (+0.32%) 13 <0> (ø) ⬇️
...ain/java/org/apache/livy/rsc/driver/RSCDriver.java 80.75% <0%> (ø) 45% <0%> (ø) ⬇️
...c/main/scala/org/apache/livy/repl/ReplDriver.scala 33.33% <0%> (+2.56%) 7% <0%> (ø) ⬇️
...cala/org/apache/livy/scalaapi/ScalaJobHandle.scala 55.88% <0%> (+2.94%) 7% <0%> (ø) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2e7d691...4a58338. Read the comment docs.

@jerryshao jerryshao left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 68.12%. Comparing base (0804c8e) to head (4a58338).
Report is 79 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master     #246      +/-   ##
============================================
- Coverage     68.45%   68.12%   -0.33%     
- Complexity      927      939      +12     
============================================
  Files           100      101       +1     
  Lines          5729     5855     +126     
  Branches        870      886      +16     
============================================
+ Hits           3922     3989      +67     
- Misses         1247     1295      +48     
- Partials        560      571      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants