Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[LIVY-969] Create docker based integration environment for local debugging #407

Merged
merged 3 commits into from
Jun 17, 2023

Conversation

ksumit
Copy link
Contributor

@ksumit ksumit commented Jun 11, 2023

What changes were proposed in this pull request?

This PR introduces a docker and docker-compose based integrated environment for local debugging. The environment consists of:

  1. Standalone spark cluster consisting of one spark master and one spark worker
  2. Livy configured to connect to this standalone cluster

Attached README explains how this integrated environment can be customized to test with private changes.

How was this patch tested?

  1. Used docker-compose up to bring up the integrated environment
  2. Logged into livy container and submitted sample spark and sql statements.
  3. Verified spark master, spark worker and livy UIs to be working, verified statements to have succeeded.
  4. Verified remote debugging to work by attaching IDE to Livy server

debugging
Add missing newlines, move customization steps towards the end in Dockerfile so we can reuse previous layers and avoid recreating it for every image build
Revert changes in module list, still update the scala-maven-plugin version to avoid issues compiling against 2.12
Upgrade spark3 version to spark 3.2.3
@ksumit ksumit requested a review from lmccay June 11, 2023 05:09
@ksumit
Copy link
Contributor Author

ksumit commented Jun 11, 2023

@lmccay @dacort @ayushtkn @askhatri Please review. Sorry earlier PR #387 got closed accidentally when I force merged to the source branch and I'm not able to reopen it. I've addressed the concerns from that PR as well.

@codecov-commenter
Copy link

codecov-commenter commented Jun 11, 2023

Codecov Report

Merging #407 (3240e68) into master (c3dd645) will decrease coverage by 36.95%.
The diff coverage is n/a.

@@              Coverage Diff              @@
##             master     #407       +/-   ##
=============================================
- Coverage     65.51%   28.57%   -36.95%     
+ Complexity      950      379      -571     
=============================================
  Files           103      103               
  Lines          6044     6044               
  Branches        911      911               
=============================================
- Hits           3960     1727     -2233     
- Misses         1537     3968     +2431     
+ Partials        547      349      -198     

see 86 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

Copy link
Contributor

@dacort dacort left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Worked great for me!

Requesting a couple minor tweaks.

dev/docker/livy-dev-base/Dockerfile Outdated Show resolved Hide resolved
dev/docker/build-images.sh Outdated Show resolved Hide resolved
pom.xml Show resolved Hide resolved
dev/docker/README.md Show resolved Hide resolved
@ksumit
Copy link
Contributor Author

ksumit commented Jun 12, 2023

Worked great for me!

Requesting a couple minor tweaks.

@dacort, I've updated as per your review comments. I did change the file permissions for build-images.sh but don't see that reflected in the updated diff.

@ksumit ksumit merged commit 314f2de into apache:master Jun 17, 2023
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants