Skip to content

Commit

Permalink
[KYUUBI apache#1382] Sync the flink-provided profile in docs and test…
Browse files Browse the repository at this point in the history 
…ing suite comments

<!--
Thanks for sending a pull request!

Here are some tips for you:
  1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
  2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
  3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->

### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
  1. If you add a feature, you can talk about the use case of it.
  2. If you fix a bug, you can clarify why it is a bug.
-->
Sync the flink-provided profile in docs and testing suite comments.

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request

Closes apache#1382 from turboFei/flink_provided.

Closes apache#1382

3ae9696 [Fei Wang] address nit
ef5bba7 [fwang12] reorder
e9c9c52 [fwang12] save

Lead-authored-by: fwang12 <fwang12@ebay.com>
Co-authored-by: Fei Wang <fwang12@ebay.com>
Signed-off-by: Fei Wang <fwang12@ebay.com>
  • Loading branch information
@turboFei
turboFei committed Dec 1, 2021
1 parent 7127490 commit a6d2323
Show file tree
Hide file tree
Showing 8 changed files with 21 additions and 17 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/master.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:
- java: 11
spark: '3.1'
spark-hadoop: '3.2'
profiles: '-DskipTests -Pspark-provided'
profiles: '-DskipTests -Pflink-provided,spark-provided'

env:
SPARK_LOCAL_IP: localhost
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/publish-snapshot.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,14 @@ jobs:
ASF_PASSWORD: ${{ secrets.NEXUS_PW }}
run: |
rm -rf ~/.m2/repository/org/apache/kyuubi
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided,spark-3.1 -pl dev/kyuubi-extension-spark-3-1 -am
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided,spark-3.2 -pl dev/kyuubi-extension-spark-3-2 -am
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pflink-provided,spark-provided
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pflink-provided,spark-provided,spark-3.1 -pl dev/kyuubi-extension-spark-3-1 -am
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pflink-provided,spark-provided,spark-3.2 -pl dev/kyuubi-extension-spark-3-2 -am
- name: Publish snapshot - branch-1.3
if: ${{ matrix.branch == 'branch-1.3' }}
env:
ASF_USERNAME: ${{ secrets.NEXUS_USER }}
ASF_PASSWORD: ${{ secrets.NEXUS_PW }}
run: |
rm -rf ~/.m2/repository/org/apache/kyuubi
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pspark-provided,kyuubi-extension-spark-3-1
./build/mvn clean deploy -s ./build/release/asf-settings.xml -DskipTests -Pflink-provided,spark-provided,kyuubi-extension-spark-3-1
4 changes: 3 additions & 1 deletion build/dist
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ set -e
KYUUBI_HOME="$(cd "`dirname "$0"`/.."; pwd)"
DISTDIR="$KYUUBI_HOME/dist"
MAKE_TGZ=false
# TODO: add FLINK_PROVIDED option
SPARK_PROVIDED=false
NAME=none
MVN="$KYUUBI_HOME/build/mvn"
Expand All @@ -40,10 +41,11 @@ function usage {
echo ""
echo "Usage:"
echo "+--------------------------------------------------------------------------------------+"
echo "| ./build/dist [--name <custom_name>] [--tgz] [--spark-provided] <maven build options> |"
echo "| ./build/dist [--name <custom_name>] [--tgz] [--flink-provided] [--spark-provided] <maven build options> |"
echo "+--------------------------------------------------------------------------------------+"
echo "name: - custom binary name, using project version if undefined"
echo "tgz: - whether to make a whole bundled package"
echo "flink-provided: - whether to make a package without Flink binary"
echo "spark-provided: - whether to make a package without Spark binary"
echo ""
}
Expand Down
6 changes: 3 additions & 3 deletions build/release/release.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,12 +94,12 @@ upload_svn_staging() {
}

upload_nexus_staging() {
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided \
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,flink-provided,spark-provided \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml"
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided,spark-3.1 \
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,flink-provided,spark-provided,spark-3.1 \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
-pl dev/kyuubi-extension-spark-3-1 -am
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,spark-provided,spark-3.2 \
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,flink-provided,spark-provided,spark-3.2 \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
-pl dev/kyuubi-extension-spark-3-2 -am
}
Expand Down
7 changes: 4 additions & 3 deletions docs/develop_tools/distribution.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,10 +33,11 @@ For more information on usage, run `./build/dist --help`
Usage:
+--------------------------------------------------------------------------------------+
| ./build/dist [--name <custom_name>] [--tgz] [--spark-provided] <maven build options> |
| ./build/dist [--name <custom_name>] [--tgz] [--flink-provided] [--spark-provided] <maven build options> |
+--------------------------------------------------------------------------------------+
name: - custom binary name, using project version if undefined
tgz: - whether to make a whole bundled package
flink-provided: - whether to make a package without Flink binary
spark-provided: - whether to make a package without Spark binary
```

Expand All @@ -48,10 +49,10 @@ For instance,

This results in a Kyuubi distribution named `kyuubi-{version}-bin-custom-name.tgz` for you.

If you are planing to deploy Kyuubi where `spark` is provided, in other word, it's not required to bundle spark binary, use
If you are planing to deploy Kyuubi where `spark`/`flink` is provided, in other word, it's not required to bundle spark/flink binary, use

```bash
./build/dist --tgz --spark-provided
./build/dist --tgz --spark-provided --flink-provided
```

Then you will get a Kyuubi distribution without spark binary named `kyuubi-{version}-bin.tgz`.
4 changes: 2 additions & 2 deletions ...engine/src/test/scala/org/apache/kyuubi/engine/spark/udf/KyuubiDefinedFunctionSuite.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,12 @@ import org.apache.kyuubi.{KyuubiFunSuite, TestUtils}
*
* To run the entire test suite:
* {{{
* build/mvn clean install -Pspark-provided -DwildcardSuites=org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite
* build/mvn clean install -Pflink-provided,spark-provided -DwildcardSuites=org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite
* }}}
*
* To re-generate golden files for entire suite, run:
* {{{
* KYUUBI_UPDATE=1 build/mvn clean install -Pspark-provided -DwildcardSuites=org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite
* KYUUBI_UPDATE=1 build/mvn clean install -Pflink-provided,spark-provided -DwildcardSuites=org.apache.kyuubi.engine.spark.udf.KyuubiDefinedFunctionSuite
* }}}
*/
// scalastyle:on line.size.limit
Expand Down
3 changes: 2 additions & 1 deletion kyuubi-common/src/test/scala/org/apache/kyuubi/TestUtils.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,8 @@ object TestUtils {
}
reader.close()
val hint = s"$markdown out of date, please update doc with " +
s"KYUUBI_UPDATE=1 build/mvn clean install -Pspark-provided -DwildcardSuites=$agent"
s"KYUUBI_UPDATE=1 build/mvn clean install -Pflink-provided,spark-provided " +
s"-DwildcardSuites=$agent"
assert(newOutput.size === expected.size, hint)

newOutput.zip(expected).foreach { case (out, in) => assert(out === in, hint) }
Expand Down
4 changes: 2 additions & 2 deletions kyuubi-server/src/test/scala/org/apache/kyuubi/config/AllKyuubiConfiguration.scala
View file
Original file line number Diff line number Diff line change
Expand Up @@ -35,12 +35,12 @@ import org.apache.kyuubi.zookeeper.ZookeeperConf
*
* To run the entire test suite:
* {{{
* build/mvn clean install -Pspark-provided -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration
* build/mvn clean install -Pflink-provided,spark-provided -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration
* }}}
*
* To re-generate golden files for entire suite, run:
* {{{
* KYUUBI_UPDATE=1 build/mvn clean install -Pspark-provided -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration
* KYUUBI_UPDATE=1 build/mvn clean install -Pflink-provided,spark-provided -DwildcardSuites=org.apache.kyuubi.config.AllKyuubiConfiguration
* }}}
*/
// scalastyle:on line.size.limit
Expand Down

0 comments on commit a6d2323

Please sign in to comment.