[TASK][EASY] Create PySpark batch jobs tests for RESTful API

[pan3793]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before creating

• I have searched in the task list and found no similar tasks.

Mentor

• I have sufficient knowledge and experience of this task, and I volunteer to be the mentor of this task to guide contributors to complete the task.

Skill requirements

• Knowledge about Spark submit
• Basic Knowledge of Scala testing and PySpark programming

Background and Goals

Restful API supports creating Batch jobs by posting /batches requests with or without resource file uploading. Currently, it is mainly used for submitting Spark Jars. As PySpark jobs become popular approach for data exploring and processing, we need to create tests for creating PySpark jobs.

• Rest API docs for creating batches
  • https://kyuubi.readthedocs.io/en/master/client/rest/rest_api.html#post-batches
  • https://kyuubi.readthedocs.io/en/master/client/rest/rest_api.html#post-batches-with-uploading-resource
• Rest API test caes:
  • POST without uploading
    https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/BatchRestApiSuite.scala#L141
  • POST with uploading https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/test/scala/org/apache/kyuubi/server/rest/client/BatchRestApiSuite.scala#L90

Implementation steps

Add tests for creating PySpark jobs via Rest API

• create tests for creating PySpark jobs using POST /batches with file on path of storage , e.g.s local filesystem, HDFS
• create tests for creating PySpark jobs with resource file uploading (https://kyuubi.readthedocs.io/en/master/client/rest/rest_api.html#post-batches)
• create tests with Rest API client
• zip PySpark scripts and related files before if necessary

Additional context

Original reporter is @bowenliang123

[weixi62961]

@pan3793 Could you please assign it to me ? I suppose to complete this task on 27th October

[bowenliang123]

Thanks. Assigned to @weixi62961 . Looking forward to your PRs.

[weixi62961]

@bowenliang123 cc @pan3793
I want to talk about current progress. If you have any question, please tell me.

Before writing test cases, I first used curl to test some PySpark jobs on Kyuubi 1.7.1. They were mainly tested in the local machine. All these curl test cases were passed.

1. PySpark job file is hosted in HDFS

# hdfs pi.py
curl -H "Content-Type: application/json" \
-X POST \
-d '{"batchType": "PYSPARK", "resource":"hdfs:/tmp/upload/pyspark_submit_sample/pi.py",  "name": "PySpark PI", "conf": {"spark.master": "local"}, "args": [10]}' \
http://localhost:10099/api/v1/batches

2. PySpark job file is uploaded through uploading resource

# upload resource: pi.py
curl --location --request POST 'http://localhost:10099/api/v1/batches' \
--form 'batchRequest="{\"batchType\":\"PYSPARK\",\"name\":\"PySpark Pi\",\"args\":[10]}";type=application/json' \
--form 'resourceFile=@"/localpath/to/file/pi.py"'

3. PySpark job file depends on other modules

# module dependency
curl -H "Content-Type: application/json" \
-X POST \
-d '{"batchType":"PYSPARK","resource":"hdfs:/tmp/upload/pyspark_submit_sample/test_module_dependency.py","name":"PySpark Module Dependency","conf":{"spark.master":"local","spark.submit.pyFiles":"hdfs:/tmp/upload/pyspark_submit_sample/my_module.zip"}}' \
http://localhost:10099/api/v1/batches

Refer to the existing 2 test cases for Spark jars, I'll write two test case for PySpark for "POST without uploading" and "POST with uploading".
Is it OK?

In addition, due to the complexity of PySpark's package management, it is not within the scope of this topic. For more information, refer to Spark official site: Python Package Management

BTW, I have created a small project for test pyspark submit sample

[bowenliang123]

Thanks for updating the detailed progress. Small PRs for subtasks are welcomed.
Raise the first PR for case 2, which sounds like the easiest case for minimum testing. (Also helps you to utilize the auto CI test runs on github.)
POST with uploading + test py file in resource folder for testing + local standalone for Spark master.

WDYT?

[weixi62961]

WDYT?

OK， Thanks

[bowenliang123]

@weixi62961 May I have contact with you? You can send your private wechat account number to my email address liang.bowen.123@qq.com

[weixi62961]

@weixi62961 May I have contact with you? You can send your private wechat account number to my email address liang.bowen.123@qq.com

Already sent

[weixi62961]

Summary

Let‘s summarize the status of PySpark Batch Job. In my opinion, For PySpark jobs, it is recommended to submit jobs by post batches existing resources instead of uploading resources.

Problem

• up to Kyuubi 1.8.0，Rest API for creating batches in two ways：
  • https://kyuubi.readthedocs.io/en/master/client/rest/rest_api.html#post-batches
  • https://kyuubi.readthedocs.io/en/master/client/rest/rest_api.html#post-batches-with-uploading-resource
• Currently, uploading-resource can only upload one resource file, which is not enough for normal PySpark jobs.

  • kyuubi/kyuubi-server/src/main/scala/org/apache/kyuubi/server/api/v1/BatchesResource.scala

    Line 177 in ceb8453

    def openBatchSessionWithUpload(

• PySpark jobs are different from JarSpark jobs, which are usually an uber-jar/fat-jar with all dependencies. Normal PySpark jobs require not only resource file, but also dependency zip packages(--py-files). spark-submt examples as follows:

# JarSpark job
$SPARK_HOME/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
fat-jar-with-dependencies.jar

# PySpark job
$SPARK_HOME/bin/spark-submit \
--master yarn \
--deploy-mode cluster \
--py-files dependency.zip \
main.py

Workaround

• For PySpark jobs, it is recommended to submit jobs by post batches existing resources instead of uploading resources.
• First upload the resource file and dependency zip packages to a location that is accessible to all nodes, such as HDFS or NFS. Then submit the job through the post-batches API. Here is an example, using HDFS.
• json parameters in pretty format

{
	"batchType": "PYSPARK",
	"resource": "hdfs:/tmp/upload/main.py",
	"conf": {
		"spark.master": "yarn",
		"spark.submit.deployMode": "cluster",
		"spark.submit.pyFiles": "hdfs:/tmp/upload/dependency.zip"
	}
}

• curl command ( it works!)

curl -H "Content-Type: application/json" \
-X POST \
-d '{"batchType":"PYSPARK","resource":"hdfs:/tmp/upload/main.py","conf":{"spark.master":"yarn","spark.submit.deployMode":"cluster","spark.submit.pyFiles":"hdfs:/tmp/upload/dependency.zip"}}' \
http://localhost:10099/api/v1/batches

WDYT？ @bowenliang123 cc @pan3793

For more PySpark job submit cases, please refer to my small project for pyspark submit sample