[ZEPPELIN-6016] Rewrite and enable Livy integration tests #4743

pan3793 · 2024-03-25T08:48:54Z

What is this PR for?

I have managed to run the Livy integration tests by rewriting the Livy Server bootstrap logic.

After several attempts, I decided to use the following version combination for Livy integration test:

Python 3.7
Scala 2.11
Livy 0.8.0-incubating
Spark 2.4.8

The version combination is based on the facts:

official Livy 0.8 binary tarball is built against Spark 2.4
official Spark 2.4 binary tarball is built against Scala 2.11
Spark 2.4 supports Python 2.7, 3.4 to 3.7

This PR also rewrite the Livy Server bootstrap logic, previously, it pulls Livy deps and relies on MiniLivyCluster, which also starts a MiniHDFS and MiniYARN, which is a so heavy approach. Now I use a sub-process to launch the bin/livy-server and run Spark in local mode for testing.

Due to changes from Spark YARN mode to Local, some tests was tuned.

What type of PR is it?

Test
Refactoring

Todos

- Analyze and fix the failed tests

What is the Jira issue?

ZEPPELIN-6016

How should this be tested?

Strongly recommended: add automated unit tests for any new or changed behavior
Outline any manual steps to test the PR here.

Screenshots (if appropriate)

Questions:

Does the license files need to update? No
Is there breaking changes for older versions? No
Does this needs documentation? Updated

pan3793 · 2024-04-03T08:48:32Z

@zjffdu The Livy test has been broken for a while, and the current approach hardly depends on Livy jars, which also blocks upgrading Spark3 with Scala 2.12, are you good with this direction?

pan3793 · 2024-04-19T15:13:51Z

livy/src/test/java/org/apache/zeppelin/livy/LivyInterpreterIT.java

      throws InterpreterException {
    if (!checkPreCondition()) {
      return;
    }
    InterpreterGroup interpreterGroup = new InterpreterGroup("group_1");
    interpreterGroup.put("session_1", new ArrayList<Interpreter>());
    Properties properties2 = new Properties(properties);
-    properties2.put("zeppelin.livy.displayAppInfo", "true");


I found that Livy can not extract appId for Spark local mode, and seems no workaround without touching Livy code, I abandoned to fix it after some struggling, just delete.

pan3793 · 2024-04-19T15:34:10Z

cc @zjffdu @jongyoul this is ready for review. And if you want changelog clear, consider merging the solo Livy 0.8 upgrading first #4678, then I will rebase this one; ignore #4678 and focus on this one is fine too.

pan3793 · 2024-04-19T15:44:55Z

Also cc @huage1994

zjffdu · 2024-04-23T06:32:06Z

Thanks for the contribution, @pan3793 , LGTM

* [MINOR] Update jdbc.md Change postgres username (apache#4704) change postgres default username mysql_user/mysql_password to pg_user/pg_password. * Bump mathjax from 2.7.0 to 3.0.0 in /zeppelin-web (apache#4705) Bumps [mathjax](https://github.com/mathjax/MathJax) from 2.7.0 to 3.0.0. - [Release notes](https://github.com/mathjax/MathJax/releases) - [Commits](mathjax/MathJax@2.7.0...3.0.0) --- updated-dependencies: - dependency-name: mathjax dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump org.apache.shiro:shiro-core from 1.10.0 to 1.13.0 (apache#4703) Bumps [org.apache.shiro:shiro-core](https://github.com/apache/shiro) from 1.10.0 to 1.13.0. - [Release notes](https://github.com/apache/shiro/releases) - [Changelog](https://github.com/apache/shiro/blob/main/RELEASE-NOTES) - [Commits](apache/shiro@shiro-root-1.10.0...shiro-root-1.13.0) --- updated-dependencies: - dependency-name: org.apache.shiro:shiro-core dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump follow-redirects from 1.15.3 to 1.15.4 in /zeppelin-web (apache#4702) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.3 to 1.15.4. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.15.3...v1.15.4) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump follow-redirects from 1.15.3 to 1.15.4 in /zeppelin-web-angular (apache#4701) Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.15.3 to 1.15.4. - [Release notes](https://github.com/follow-redirects/follow-redirects/releases) - [Commits](follow-redirects/follow-redirects@v1.15.3...v1.15.4) --- updated-dependencies: - dependency-name: follow-redirects dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [MINOR] Upgrade jackson version in /zeppelin-server (apache#4635) * [HOTFIX] Disable shell interpreter (apache#4708) * Fix GCSNotebookRepoTests (apache#4711) * [HOTFIX] Check permission when updating cron information (apache#4631) * [HOTFIX] Check permission when updating cron information * [HOTFIX] Fix commented * [HOTFIX] Check permission when updating cron information * [HOTFIX] Check permission when updating cron information * [HOTFIX] Check permission when updating cron information * [MINOR] Change minimum java version to 11 in docs (apache#4710) * [MINOR] Change minimum java version to 11 in docs * [MINOR] Change minimum java version to 11 in docs * [HOTFIX] Change the link of `helium.json` from S3 to zeppelin.apache.org (apache#4713) * [ZEPPELIN-5990] Disable sensitive configuration for JDBC url (apache#4709) * [ZEPPELIN-5990] Disable sensitive configuration for JDBC url * [ZEPPELIN-5990] Disable sensitive configuration for JDBC url * [ZEPPELIN-5995] Update Kubernetes Library and hopefully fix flaky tests (apache#4712) * [MINOR] Set Snapshot version to 0.12.0-SNAPSHOT (apache#4720) * change version to 0.11.1-SNAPSHOT * change version * change to 0.12.0-SNAPSHOT * Bump org.postgresql:postgresql from 42.4.3 to 42.7.2 in /jdbc (apache#4723) Bumps [org.postgresql:postgresql](https://github.com/pgjdbc/pgjdbc) from 42.4.3 to 42.7.2. - [Release notes](https://github.com/pgjdbc/pgjdbc/releases) - [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md) - [Commits](https://github.com/pgjdbc/pgjdbc/commits) --- updated-dependencies: - dependency-name: org.postgresql:postgresql dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [HOTFIX] Escape Ldap search filters (apache#4714) * [HOTFIX] Escape envs when using `.conf` (apache#4715) * Fix the new zeppelin ui. The specific reason is that the use of excessively wide column widths resulted in the remaining fields being squeezed out of the screen, and nz table did not have a scroll bar set to display scrolling. (apache#4727) * Bump ip from 1.1.8 to 1.1.9 in /zeppelin-web (apache#4724) Bumps [ip](https://github.com/indutny/node-ip) from 1.1.8 to 1.1.9. - [Commits](indutny/node-ip@v1.1.8...v1.1.9) --- updated-dependencies: - dependency-name: ip dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump lodash from 4.17.15 to 4.17.21 in /zeppelin-web-angular (apache#4689) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.21. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.15...4.17.21) --- updated-dependencies: - dependency-name: lodash dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [HOTFIX] Escape HeliumPackage information (apache#4728) * [NO-ISSUE] Use reload4j instead of log4j (apache#4719) * [NO-ISSUE] Upgrade org.json jar (apache#4722) * [ZEPPELIN-6001] k8s images fix (apache#4729) * Update zeppelin-interpreter Dockerfile fixed ARG version; Error "bzip2: Compressed file ends unexpectedly;" fixed by adding ---no-iri flag to wget * Update zeppelin-server Dockerfile fixed ARG version; fixed JAVA_HOME env. * Update scripts/docker/zeppelin-interpreter/Dockerfile Co-authored-by: Philipp Dallig <philipp.dallig@gmail.com> --------- Co-authored-by: Philipp Dallig <philipp.dallig@gmail.com> * [ZEPPELIN-6000] Polish some files mainly in zengine (apache#4731) * some misc polish * some misc polish * [ZEPPELIN-6003] Log source info of SQL in JDBCInterpreter (apache#4732) * [ZEPPELIN-6003] Log detail info of SQL in JDBCInterpreter * Update Co-authored-by: Philipp Dallig <philipp.dallig@gmail.com> --------- Co-authored-by: Philipp Dallig <philipp.dallig@gmail.com> * [ZEPPELIN-6002] Fix completer NPE (apache#4730) Co-authored-by: Philipp Dallig <philipp.dallig@gmail.com> * [ZEPPELIN-5986] Bump Maven surefire/failsafe plugins to recover JUnit5 tests (apache#4734) * [ZEPPELIN-5986] Re-enable Junit 5 integration tests by upgrading maven plugins * Fix SparkIntegrationTest * Disable the LivyInterpreterIT * Selenium * NPE * Disable testEditOnDoubleClick * [ZEPPELIN-6005] Update Kyuubi JDBC docs (apache#4738) * Bump org.apache.commons:commons-configuration2 from 2.8.0 to 2.10.1 (apache#4740) Bumps org.apache.commons:commons-configuration2 from 2.8.0 to 2.10.1. --- updated-dependencies: - dependency-name: org.apache.commons:commons-configuration2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [MINOR] Fix missing changed versions (apache#4737) * [MINOR] Fix missing changed versions * nit * Bump com.nimbusds:nimbus-jose-jwt in /zeppelin-server (apache#4733) Bumps [com.nimbusds:nimbus-jose-jwt](https://bitbucket.org/connect2id/nimbus-jose-jwt) from 9.13 to 9.37.2. - [Changelog](https://bitbucket.org/connect2id/nimbus-jose-jwt/src/master/CHANGELOG.txt) - [Commits](https://bitbucket.org/connect2id/nimbus-jose-jwt/branches/compare/9.37.2..9.13) --- updated-dependencies: - dependency-name: com.nimbusds:nimbus-jose-jwt dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [ZEPPELIN-6007] Enhance release scripts for tar shasum commands detection (apache#4747) * [ZEPPELIN-6007] Enhance release scripts for tar shasum commands detection * fix * nit * indent * [ZEPPELIN-6008] Fix parameter usage of bokeh in test case (apache#4748) * [ZEPPELIN-6008] Pin plotly 5.19.0 * Revert "[ZEPPELIN-6008] Pin plotly 5.19.0" This reverts commit 60d9ce9. * Pin bokeh=3.3.4 * Revert "Pin bokeh=3.3.4" This reverts commit 44ccc98. * Remove usage of deprecated paramater legend * [ZEPPELIN-5969] Remove Hadoop2 and move to Hadoop3 shaded client (apache#4691) * Drop hadoop2 in github actions * Update docs * Drop hadoop2 support * Remove hadoop2 integration tests * findbugs use the same version in all modules * Use hadoop3.3 for tests * Move to scala 2.12 * Try to fix flink * Usage of metals * Remove duplicate version and groupid * Fix Flink with Hadoop3 * fix log * R * fix * fix * fix * fix * hadoop-3.3 * fix * fix * Address comments * address comments --------- Co-authored-by: Philipp Dallig <philipp.dallig@gmail.com> * [HOTFIX] Remove rendering helium description as HTML in Frontend (apache#4755) * Bump express from 4.18.2 to 4.19.2 in /zeppelin-web-angular (apache#4744) Bumps [express](https://github.com/expressjs/express) from 4.18.2 to 4.19.2. - [Release notes](https://github.com/expressjs/express/releases) - [Changelog](https://github.com/expressjs/express/blob/master/History.md) - [Commits](expressjs/express@4.18.2...4.19.2) --- updated-dependencies: - dependency-name: express dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * update scripts/docker/zeppelin/bin/Dockerfile to JDK11 (apache#4753) * update scripts/docker/zeppelin/bin/Dockerfile to JDK11 * update miniconda to py3.9 * Update Dockerfile to remove extra line * Bump some apache common libs (apache#4757) * [ZEPPELIN-6006] Remove command line applications when downloading applications (apache#4746) * Move Files with java * Use java to download external dependecies * Improve code after review * Correct Mirror-URL and compilation * [ZEPPELIN-6016] Rewrite and enable Livy integration tests (apache#4743) * wip * nit * nit * wip * wip * fix * [ZEPPELIN-5973] Bump Livy 0.8.0-incubating * nit * Spark 3.5.1 * test * fix * comment * nit * nit * nit * [ZEPPELIN-6017] Revert changes about ZEPPELIN_IDENT_STRING in ZEPPELIN-5421 * [ZEPPELIN-6015] Update ci-action plugins (apache#4759) * [ZEPPELIN-5999] Reduce instance objects from Zeppelin (apache#4726) * Remove ZeppelinConfiguration Singelton and add MiniZeppelinServer * Add ZeppelinConfiguration to Interpreter * Remove static pluginmanager and configstorage * Inject servicelocator into SessionConfiguratior * use custom serviceLocator in integration tests * Reorder code * code cleanup * Add ZeppelinConfiguration as class variable to InterpreterOption * Avoid leaking third-party libs * [ZEPPELIN-6019] Remove Submarine support (apache#4763) * [ZEPPELIN-6022] Skip decryption of credentials.json when file is empty (apache#4765) * Skip decryption when empty * Use more elegant empty json string check * [ZEPPELIN-6018] Update gRPC version from 1.51.0 to 1.55.1 for successful Apache Zeppelin build on s390x architecture (apache#4758) * Update grpc version in pom.xml for successful Apache Zeppelin build on s390x architecture In the s390x architecture, the Apache Zeppelin package builds successfully when the grpc dependency version in the pom.xml file is changed from version 1.51.0 to version 1.62.2. Therefore, I have updated the grpc version in the pom.xml file. * Update LICENSE Updated the grpc version in LICENSE from 1.51.0 to 1.62.2 * grpc-version-change-to 1.55.1 * [ZEPPELIN-6027] Enhanced Integration with Apache Kyuubi (apache#4770) * [ZEPPELIN-6027] Enhanced Integration with Apache Kyuubi * fix style * [ZEPPELIN-6028] Enhance default value assignment for ZEPPELIN_IDENT_STRING (apache#4772) * [ZEPPELIN-6029] Set COPYFILE_DISABLE=1 for macOS tar (apache#4774) * [ZEPPELIN-6029] Add --disable-copyfile for macOS tar * COPYFILE_DISABLE=1 * ODP-1644: Removed unsupported interpreter * ODP-1315: Bumpup loadash to 4.x * ODP: fix odp version * ODP-303 New UI build fix for missing projects * [ODP-1315] ODP-1644: CVE fixes * ODP-1644: Update to Spark Version * ODP-1644: Added libthrift, updated hadoop and phoenix versions * ODP-1644: Hive 4.0.0 support for JDBC * ODP-1644: Added two variables that were previously missed * ODP-1644: Fixed node/npm version issue when testing with arm64 MacOS * ODP-1644: Increased timeouts/sleeps to pass tests * ODP-1644: Increased ms in Thread.sleep * ODP-1829: Updated pom.xml files * ODP-1829: Removed uneeded tests * ODP-1829: Fixed tests failures * ODP-1829: Disabled Finicky Helium Test * ODP-1829: Increased wait time so that tests would not fail * ODP-1829: Thread.Sleep -> Thread.sleep * Zeppelin Build fixes * ODP-1644: Updated version number to 11.2.3.3.6.0-1 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: bigpear0201 <bigpear0201@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Shefali Singh <31477542+shefali163@users.noreply.github.com> Co-authored-by: Jongyoul Lee <jongyoul@gmail.com> Co-authored-by: Philipp Dallig <philipp.dallig@gmail.com> Co-authored-by: PJ Fanning <pjfanning@users.noreply.github.com> Co-authored-by: myongyun <984093369@qq.com> Co-authored-by: th1nksnow <114925852+th1nksnow@users.noreply.github.com> Co-authored-by: Manhua <kevinjmh@qq.com> Co-authored-by: Manhua <manhua@apache.org> Co-authored-by: Cheng Pan <chengpan@apache.org> Co-authored-by: Cheng Pan <pan3793@gmail.com> Co-authored-by: Gayle <9532712+flowy0@users.noreply.github.com> Co-authored-by: zeotuan <48720253+zeotuan@users.noreply.github.com> Co-authored-by: Aditi Sharma <167422409+aditi-sharma-1@users.noreply.github.com> Co-authored-by: Prabhjyot Singh <prabhjyot@acceldata.io> Co-authored-by: shubhamsharma <shubham@acceldata.io>

pan3793 changed the title ~~Rewrite and enable Livy integration tests~~ [WIP] Rewrite and enable Livy integration tests Mar 25, 2024

pan3793 force-pushed the livy-test branch from 33dd079 to e8ef677 Compare April 3, 2024 08:00

pan3793 added 9 commits April 18, 2024 23:38

wip

1870765

nit

6c5ebf8

nit

36ebd24

wip

d7beba8

wip

e4c55c7

fix

27286ad

[ZEPPELIN-5973] Bump Livy 0.8.0-incubating

1e5c8a2

nit

b6f9d8c

Spark 3.5.1

07d00d7

pan3793 force-pushed the livy-test branch from 57b2234 to 07d00d7 Compare April 18, 2024 15:39

pan3793 added 3 commits April 19, 2024 13:07

test

11136b9

fix

77f6d69

comment

de1fdc7

pan3793 commented Apr 19, 2024

View reviewed changes

pan3793 added 3 commits April 19, 2024 23:14

nit

63527c1

nit

61258a8

nit

1b380b6

pan3793 changed the title ~~[WIP] Rewrite and enable Livy integration tests~~ [ZEPPELIN-6016] Rewrite and enable Livy integration tests Apr 19, 2024

zjffdu approved these changes Apr 23, 2024

View reviewed changes

huage1994 merged commit ca2481d into apache:master Apr 23, 2024
28 checks passed

pan3793 deleted the livy-test branch April 24, 2024 06:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ZEPPELIN-6016] Rewrite and enable Livy integration tests #4743

[ZEPPELIN-6016] Rewrite and enable Livy integration tests #4743

pan3793 commented Mar 25, 2024 •

edited

Loading

pan3793 commented Apr 3, 2024 •

edited

Loading

pan3793 Apr 19, 2024

pan3793 commented Apr 19, 2024

pan3793 commented Apr 19, 2024

zjffdu commented Apr 23, 2024

[ZEPPELIN-6016] Rewrite and enable Livy integration tests #4743

[ZEPPELIN-6016] Rewrite and enable Livy integration tests #4743

Conversation

pan3793 commented Mar 25, 2024 • edited Loading

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

pan3793 commented Apr 3, 2024 • edited Loading

pan3793 Apr 19, 2024

Choose a reason for hiding this comment

pan3793 commented Apr 19, 2024

pan3793 commented Apr 19, 2024

zjffdu commented Apr 23, 2024

pan3793 commented Mar 25, 2024 •

edited

Loading

pan3793 commented Apr 3, 2024 •

edited

Loading