[Enhancement] Include plugin jars in driver and executor classspath

[chncaesar]

What changes were proposed in this pull request?

1. Add nlp_seg and nlp-lang jars in libs/ directory
2. Add plugin/*.jar to driver & executor classpath

How was this patch tested?

byzer server mode

started with byzer.sh start local mode

jars in libs

hadoop@ubuntu-dev:/tmp/byzer-lang-3.1.1-2.3.0-SNAPSHOT$ ll ./libs/
total 8496
drwxrwxr-x  2 hadoop hadoop    4096 Apr 27 15:27 ./
drwxrwxr-x 10 hadoop hadoop    4096 Apr 27 15:36 ../
-rw-rw-r--  1 hadoop hadoop 6002929 Mar 23 16:52 ansj_seg-5.1.6.jar
-rw-rw-r--  1 hadoop hadoop 2684972 Dec 29  2020 nlp-lang-1.7.8.jar

Byzer sandbox

Locally built, putting bootstrap.sh and byzer-lang main jar in sandbox. sandbox is started

Byzer on Yarn

Hadoop 3.2.0;
conf/byzer.properties.override

spark.master=yarn
streaming.name=byzer-lang
streaming.rest=true
streaming.platform=spark
streaming.spark.service=true
streaming.plugin.clzznames=tech.mlsql.plugins.ds.MLSQLExcelApp,tech.mlsql.plugins.assert.app.MLSQLAssert,tech.mlsql.plugins.shell.app.MLSQLShell,tech.mlsql.plugins.ext.ets.app.MLSQLETApp,tech.mlsql.plugins.mllib.app.MLSQLMllib

# Spark config
spark.driver.memory=2g
spark.executor.memory=1g
spark.driver.cores=1
spark.executor.cores=1
spark.executor.instances=1

Environment variables:

SPARK_HOME /work/server/spark
HADOOP_CONF_DIR /work/server/hadoop/etc/hadoop

plugin jars are manually copied to $BYZER_HOME/plugin

byzer-lang started with byzer.sh start.
list yarn app yarn app -list

2022-04-27 15:59:59,530 INFO client.RMProxy: Connecting to ResourceManager at ubuntu-dev/192.168.50.254:8032
Total number of applications (application-types: [], states: [SUBMITTED, ACCEPTED, RUNNING] and tags: []):1
                Application-Id	    Application-Name	    Application-Type	      User	     Queue	             State	       Final-State	       Progress	                       Tracking-URL
application_1651045935004_0001	          byzer-lang	               SPARK	    hadoop	root.hadoop	           RUNNING	         UNDEFINED	            10%	             http://ubuntu-dev:4040

!show version; result

!show et; result

and stopped using byzer.sh stop

hadoop@ubuntu-dev:/work/server/byzer-lang-3.1.1-2.3.0-SNAPSHOT/bin$ ./byzer.sh stop
2022-04-27 16:17:57 Stopping Byzer engine...
2022-04-27 16:17:57 Stopping Byzer-lang: 434188

Are there and DOC need to update?

• Doc is finished

Spark Core Compatibility

[hellozepp]

这里可以使用正则匹配的方式吗

[ZhengshuaiPENG]

这里是不是还缺了 license，readme 和 release note 那三个文件以及去掉 data 和 sample 两个目录

[ZhengshuaiPENG]

LGTM