[D 2017-07-12 19:03:35.633 KernelGatewayApp] Searching [u'/home/elyra', '/home/elyra/.jupyter', '/opt/anaconda2/etc/jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files [D 2017-07-12 19:03:35.633 KernelGatewayApp] Looking for jupyter_config in /etc/jupyter [D 2017-07-12 19:03:35.634 KernelGatewayApp] Looking for jupyter_config in /usr/local/etc/jupyter [D 2017-07-12 19:03:35.634 KernelGatewayApp] Looking for jupyter_config in /opt/anaconda2/etc/jupyter [D 2017-07-12 19:03:35.634 KernelGatewayApp] Looking for jupyter_config in /home/elyra/.jupyter [D 2017-07-12 19:03:35.634 KernelGatewayApp] Looking for jupyter_config in /home/elyra [D 2017-07-12 19:03:35.635 KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /etc/jupyter [D 2017-07-12 19:03:35.635 KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /usr/local/etc/jupyter [D 2017-07-12 19:03:35.635 KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /opt/anaconda2/etc/jupyter [D 2017-07-12 19:03:35.635 KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /home/elyra/.jupyter [D 2017-07-12 19:03:35.636 KernelGatewayApp] Looking for jupyter_kernel_gateway_config in /home/elyra [D 2017-07-12 19:03:35.639 KernelGatewayApp] Loading saved sessions from /var/lib/elyra/sessions/kernels.json [D 2017-07-12 19:03:35.648 KernelGatewayApp] RemoteKernelManager: Writing connection file with ip=169.45.103.145, control=34350, hb=39460, iopub=34591, stdin=42329, shell=34625 [D 2017-07-12 19:03:35.649 KernelGatewayApp] Found kernel python2 in /opt/anaconda2/share/jupyter/kernels [D 2017-07-12 19:03:35.649 KernelGatewayApp] Found kernel spark_2.1_scala_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:03:35.649 KernelGatewayApp] Found kernel spark_2.1_python_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:03:35.673 KernelGatewayApp] BaseProcessProxy env: {'KERNEL_ID': u'a85c4854-872f-4e0a-ab14-66077953eb2c'} [I 2017-07-12 19:03:35.687 KernelGatewayApp] Jupyter Kernel Gateway at http://0.0.0.0:8888 [D 2017-07-12 19:04:20.082 KernelGatewayApp] Found kernel python2 in /opt/anaconda2/share/jupyter/kernels [D 2017-07-12 19:04:20.082 KernelGatewayApp] Found kernel spark_2.1_scala_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:20.082 KernelGatewayApp] Found kernel spark_2.1_python_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:20.082 KernelGatewayApp] Found kernel python2 in /opt/anaconda2/share/jupyter/kernels [D 2017-07-12 19:04:20.083 KernelGatewayApp] Found kernel spark_2.1_scala_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:20.083 KernelGatewayApp] Found kernel spark_2.1_python_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:20.084 KernelGatewayApp] Found kernel python2 in /opt/anaconda2/share/jupyter/kernels [D 2017-07-12 19:04:20.084 KernelGatewayApp] Found kernel spark_2.1_scala_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:20.084 KernelGatewayApp] Found kernel spark_2.1_python_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:20.085 KernelGatewayApp] Found kernel python2 in /opt/anaconda2/share/jupyter/kernels [D 2017-07-12 19:04:20.085 KernelGatewayApp] Found kernel spark_2.1_scala_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:20.085 KernelGatewayApp] Found kernel spark_2.1_python_yarn_cluster in /usr/local/share/jupyter/kernels [I 170712 19:04:20 web:2063] 200 GET /api/kernelspecs (73.223.84.247) 5.34ms [D 2017-07-12 19:04:40.384 KernelGatewayApp] Found kernel python2 in /opt/anaconda2/share/jupyter/kernels [D 2017-07-12 19:04:40.385 KernelGatewayApp] Found kernel spark_2.1_scala_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:40.385 KernelGatewayApp] Found kernel spark_2.1_python_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:40.385 KernelGatewayApp] Found kernel python2 in /opt/anaconda2/share/jupyter/kernels [D 2017-07-12 19:04:40.385 KernelGatewayApp] Found kernel spark_2.1_scala_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:40.385 KernelGatewayApp] Found kernel spark_2.1_python_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:40.386 KernelGatewayApp] Found kernel python2 in /opt/anaconda2/share/jupyter/kernels [D 2017-07-12 19:04:40.386 KernelGatewayApp] Found kernel spark_2.1_scala_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:40.386 KernelGatewayApp] Found kernel spark_2.1_python_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:40.387 KernelGatewayApp] Found kernel python2 in /opt/anaconda2/share/jupyter/kernels [D 2017-07-12 19:04:40.387 KernelGatewayApp] Found kernel spark_2.1_scala_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:40.387 KernelGatewayApp] Found kernel spark_2.1_python_yarn_cluster in /usr/local/share/jupyter/kernels [I 170712 19:04:40 web:2063] 200 GET /api/kernelspecs (73.223.84.247) 3.68ms [D 2017-07-12 19:04:40.486 KernelGatewayApp] RemoteMappingKernelManager.start_kernel: spark_2.1_scala_yarn_cluster [D 2017-07-12 19:04:40.488 KernelGatewayApp] Found kernel python2 in /opt/anaconda2/share/jupyter/kernels [D 2017-07-12 19:04:40.488 KernelGatewayApp] Found kernel spark_2.1_scala_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:40.488 KernelGatewayApp] Found kernel spark_2.1_python_yarn_cluster in /usr/local/share/jupyter/kernels [D 2017-07-12 19:04:40.489 KernelGatewayApp] Instantiating kernel 'Spark 2.1 - Scala (YARN Cluster Mode)' with process proxy: kernel_gateway.services.kernels.processproxy.YarnProcessProxy [I 170712 19:04:40 base:37] Request http://spark-02.softlayer.com:8088//ws/v1/cluster/nodes [D 2017-07-12 19:04:40.491 KernelGatewayApp] BaseProcessProxy env: {'KERNEL_ID': u'c1081493-a48b-43df-ad65-c7adeb835b7e', 'LESSOPEN': '||/usr/bin/lesspipe.sh %s', 'ELYRA_REMOTE_USER': 'elyra', 'ELYRA_REMOTE_HOSTS': 'spark-02.softlayer.com,spark-03.softlayer.com,spark-04.softlayer.com,spark-05.softlayer.com', 'SELINUX_USE_CURRENT_RANGE': '', 'LOGNAME': 'elyra', 'USER': 'elyra', 'QTDIR': '/usr/lib64/qt-3.3', 'PATH': '/opt/anaconda2/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin', 'LANG': 'en_US.UTF-8', 'QTLIB': '/usr/lib64/qt-3.3/lib', 'TERM': 'xterm-256color', 'SHELL': '/bin/bash', 'ELYRA_PROXY_LAUNCH_LOG': '/var/log/elyra/proxy_launch_2017-07-12.log', 'QTINC': '/usr/lib64/qt-3.3/include', 'HISTSIZE': '1000', 'ELYRA_YARN_ENDPOINT': 'http://spark-02.softlayer.com:8088/ws/v1/cluster', 'HOME': '/home/elyra', 'XDG_RUNTIME_DIR': '/run/user/0', 'JUPYTER_DATA_DIR': '/var/lib/elyra', u'KERNEL_USERNAME': u'', 'SELINUX_ROLE_REQUESTED': '', 'ELYRA_KERNEL_LAUNCH_TIMEOUT': '120', 'MAIL': '/var/spool/mail/root', 'QT_GRAPHICSSYSTEM_CHECKED': '1', 'XDG_SESSION_ID': '131', '_': '/opt/anaconda2/bin/jupyter', 'SSH_TTY': '/dev/pts/0', 'HOSTNAME': 'spark-02.softlayer.com', 'SSH_CLIENT': '73.223.84.247 53893 22', 'SELINUX_LEVEL_REQUESTED': '', 'HISTCONTROL': 'ignoredups', 'SHLVL': '3', 'PWD': '/home/elyra', 'JUPYTER_RUNTIME_DIR': '/var/run/elyra/runtime', 'SSH_CONNECTION': '73.223.84.247 53893 169.45.103.134 22'} [D 2017-07-12 19:04:40.492 KernelGatewayApp] RemoteKernelManager: Writing connection file with ip=0.0.0.0, control=0, hb=0, iopub=0, stdin=0, shell=0 [D 2017-07-12 19:04:40.494 KernelGatewayApp] Starting kernel: [u'/usr/local/share/jupyter/kernels/spark_2.1_scala_yarn_cluster/bin/run.sh', u'--profile', u'/var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json'] [D 2017-07-12 19:04:40.494 KernelGatewayApp] Launching kernel: Spark 2.1 - Scala (YARN Cluster Mode) [D 2017-07-12 19:04:40.499 KernelGatewayApp] YarnProcessProxy - confirm startup, Kernel ID: c1081493-a48b-43df-ad65-c7adeb835b7e, YARN endpoint: http://spark-02.softlayer.com:8088/ws/v1/cluster, spark-submit pid 44269, cmd: '[u'/usr/local/share/jupyter/kernels/spark_2.1_scala_yarn_cluster/bin/run.sh', u'--profile', u'/var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json']' Starting Scala kernel for Spark 2.1 in Yarn Cluster mode as user UNSPECIFIED + eval exec /usr/hdp/current/spark2-client/bin/spark-submit --name c1081493-a48b-43df-ad65-c7adeb835b7e '--master yarn --deploy-mode cluster --name ${KERNEL_ID:-ERROR__NO__KERNEL_ID} --conf spark.yarn.submit.waitAppCompletion=false' --jars /usr/local/share/jupyter/kernels/spark_2.1_scala_yarn_cluster/lib/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar --class launcher.ToreeLauncher /usr/local/share/jupyter/kernels/spark_2.1_scala_yarn_cluster/lib/toree-launcher_2.11-1.jar '' '--response-address ${KERNEL_RESPONSE_ADDRESS:-ERROR__NO__KERNEL_RESPONSE_ADDRESS}' --profile /var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json ++ exec /usr/hdp/current/spark2-client/bin/spark-submit --name c1081493-a48b-43df-ad65-c7adeb835b7e --master yarn --deploy-mode cluster --name c1081493-a48b-43df-ad65-c7adeb835b7e --conf spark.yarn.submit.waitAppCompletion=false --jars /usr/local/share/jupyter/kernels/spark_2.1_scala_yarn_cluster/lib/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar --class launcher.ToreeLauncher /usr/local/share/jupyter/kernels/spark_2.1_scala_yarn_cluster/lib/toree-launcher_2.11-1.jar --response-address ERROR__NO__KERNEL_RESPONSE_ADDRESS --profile /var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json [I 170712 19:04:41 base:37] Request http://spark-02.softlayer.com:8088//ws/v1/cluster/apps [I 2017-07-12 19:04:41.505 KernelGatewayApp] Application ID not yet assigned for kernel: c1081493-a48b-43df-ad65-c7adeb835b7e, will retry later. 17/07/12 19:04:41 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/07/12 19:04:42 WARN DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded. 17/07/12 19:04:42 INFO RMProxy: Connecting to ResourceManager at spark-02.softlayer.com/169.45.103.134:8050 [I 170712 19:04:42 base:37] Request http://spark-02.softlayer.com:8088//ws/v1/cluster/apps [I 2017-07-12 19:04:42.510 KernelGatewayApp] Application ID not yet assigned for kernel: c1081493-a48b-43df-ad65-c7adeb835b7e, will retry later. 17/07/12 19:04:42 INFO Client: Requesting a new application from cluster with 4 NodeManagers 17/07/12 19:04:42 INFO Client: Verifying our application has not requested more than the maximum memory capability of the cluster (5120 MB per container) 17/07/12 19:04:42 INFO Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead 17/07/12 19:04:42 INFO Client: Setting up container launch context for our AM 17/07/12 19:04:42 INFO Client: Setting up the launch environment for our AM container 17/07/12 19:04:42 INFO Client: Preparing resources for our AM container [I 170712 19:04:43 base:37] Request http://spark-02.softlayer.com:8088//ws/v1/cluster/apps [I 2017-07-12 19:04:43.514 KernelGatewayApp] Application ID not yet assigned for kernel: c1081493-a48b-43df-ad65-c7adeb835b7e, will retry later. 17/07/12 19:04:43 INFO Client: Use hdfs cache file as spark.yarn.archive for HDP, hdfsCacheFile:hdfs://spark-02.softlayer.com:8020/hdp/apps/2.6.1.0-129/spark2/spark2-hdp-yarn-archive.tar.gz 17/07/12 19:04:43 INFO Client: Source and destination file systems are the same. Not copying hdfs://spark-02.softlayer.com:8020/hdp/apps/2.6.1.0-129/spark2/spark2-hdp-yarn-archive.tar.gz 17/07/12 19:04:43 INFO Client: Uploading resource file:/usr/local/share/jupyter/kernels/spark_2.1_scala_yarn_cluster/lib/toree-launcher_2.11-1.jar -> hdfs://spark-02.softlayer.com:8020/user/elyra/.sparkStaging/application_1499804611229_0007/toree-launcher_2.11-1.jar 17/07/12 19:04:43 INFO Client: Uploading resource file:/usr/local/share/jupyter/kernels/spark_2.1_scala_yarn_cluster/lib/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar -> hdfs://spark-02.softlayer.com:8020/user/elyra/.sparkStaging/application_1499804611229_0007/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar 17/07/12 19:04:43 INFO Client: Uploading resource file:/tmp/spark-5637989c-11be-4694-baa7-97c2bb94a4d6/__spark_conf__302281831433104706.zip -> hdfs://spark-02.softlayer.com:8020/user/elyra/.sparkStaging/application_1499804611229_0007/__spark_conf__.zip 17/07/12 19:04:43 INFO SecurityManager: Changing view acls to: elyra 17/07/12 19:04:43 INFO SecurityManager: Changing modify acls to: elyra 17/07/12 19:04:43 INFO SecurityManager: Changing view acls groups to: 17/07/12 19:04:43 INFO SecurityManager: Changing modify acls groups to: 17/07/12 19:04:43 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(elyra); groups with view permissions: Set(); users with modify permissions: Set(elyra); groups with modify permissions: Set() 17/07/12 19:04:44 INFO Client: Submitting application application_1499804611229_0007 to ResourceManager 17/07/12 19:04:44 INFO YarnClientImpl: Submitted application application_1499804611229_0007 17/07/12 19:04:44 INFO Client: Application report for application_1499804611229_0007 (state: ACCEPTED) 17/07/12 19:04:44 INFO Client: client token: N/A diagnostics: [Wed Jul 12 19:04:44 -0500 2017] Application is Activated, waiting for resources to be assigned for AM. Details : AM Partition = ; Partition Resource = ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 5.0 % ; Queue's Absolute max capacity = 100.0 % ; ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: default start time: 1499904284023 final status: UNDEFINED tracking URL: http://spark-02.softlayer.com:8088/proxy/application_1499804611229_0007/ user: elyra 17/07/12 19:04:44 INFO ShutdownHookManager: Shutdown hook called 17/07/12 19:04:44 INFO ShutdownHookManager: Deleting directory /tmp/spark-5637989c-11be-4694-baa7-97c2bb94a4d6 [I 170712 19:04:44 base:37] Request http://spark-02.softlayer.com:8088//ws/v1/cluster/apps [I 2017-07-12 19:04:44.519 KernelGatewayApp] Application ID: application_1499804611229_0007 assigned for kernel: c1081493-a48b-43df-ad65-c7adeb835b7e, state: ACCEPTED, 4.2 seconds after starting. [I 170712 19:04:44 base:37] Request http://spark-02.softlayer.com:8088//ws/v1/cluster/apps/application_1499804611229_0007 [D 2017-07-12 19:04:44.531 KernelGatewayApp] Kernel 'c1081493-a48b-43df-ad65-c7adeb835b7e' with app ID application_1499804611229_0007 has been assigned to host spark-04.softlayer.com. CurrentState=ACCEPTED, Attempt=4 [D 2017-07-12 19:04:44.531 KernelGatewayApp] Waiting for application to enter 'RUNNING' state. KernelID=c1081493-a48b-43df-ad65-c7adeb835b7e, ApplicationID=application_1499804611229_0007, AssignedHost=spark-04.softlayer.com, CurrentState=ACCEPTED, Attempt=4 [D 2017-07-12 19:04:45.541 KernelGatewayApp] Waiting for application to enter 'RUNNING' state. KernelID=c1081493-a48b-43df-ad65-c7adeb835b7e, ApplicationID=application_1499804611229_0007, AssignedHost=spark-04.softlayer.com, CurrentState=ACCEPTED, Attempt=5 [D 2017-07-12 19:04:46.550 KernelGatewayApp] Waiting for application to enter 'RUNNING' state. KernelID=c1081493-a48b-43df-ad65-c7adeb835b7e, ApplicationID=application_1499804611229_0007, AssignedHost=spark-04.softlayer.com, CurrentState=ACCEPTED, Attempt=6 [D 2017-07-12 19:04:47.558 KernelGatewayApp] Waiting for application to enter 'RUNNING' state. KernelID=c1081493-a48b-43df-ad65-c7adeb835b7e, ApplicationID=application_1499804611229_0007, AssignedHost=spark-04.softlayer.com, CurrentState=ACCEPTED, Attempt=7 [D 2017-07-12 19:04:48.567 KernelGatewayApp] KernelID=c1081493-a48b-43df-ad65-c7adeb835b7e, ApplicationID=application_1499804611229_0007, AssignedHost=spark-04.softlayer.com, CurrentState=RUNNING, Attempt=8 [D 2017-07-12 19:04:48.666 KernelGatewayApp] Copying file '/var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json' to file '/var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json' from host 'spark-04.softlayer.com' ... [D 2017-07-12 19:04:49.021 KernelGatewayApp] No such file '/var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json' for app 'application_1499804611229_0007', retrying... [D 2017-07-12 19:04:50.030 KernelGatewayApp] KernelID=c1081493-a48b-43df-ad65-c7adeb835b7e, ApplicationID=application_1499804611229_0007, AssignedHost=spark-04.softlayer.com, CurrentState=RUNNING, Attempt=9 [D 2017-07-12 19:04:50.127 KernelGatewayApp] Copying file '/var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json' to file '/var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json' from host 'spark-04.softlayer.com' ... [D 2017-07-12 19:04:50.482 KernelGatewayApp] No such file '/var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json' for app 'application_1499804611229_0007', retrying... [D 2017-07-12 19:04:51.494 KernelGatewayApp] KernelID=c1081493-a48b-43df-ad65-c7adeb835b7e, ApplicationID=application_1499804611229_0007, AssignedHost=spark-04.softlayer.com, CurrentState=RUNNING, Attempt=10 [D 2017-07-12 19:04:51.592 KernelGatewayApp] Copying file '/var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json' to file '/var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json' from host 'spark-04.softlayer.com' ... [D 2017-07-12 19:04:51.956 KernelGatewayApp] Pulled connection file /var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json on host spark-04.softlayer.com to local, ApplicationID=application_1499804611229_0007, Attempt=10 [D 2017-07-12 19:04:51.956 KernelGatewayApp] Loading connection file /var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json [D 2017-07-12 19:04:51.958 KernelGatewayApp] Connecting to: tcp://169.45.103.145:41509 [D 2017-07-12 19:04:51.959 KernelGatewayApp] Connecting to: tcp://169.45.103.145:39681 [I 2017-07-12 19:04:51.961 KernelGatewayApp] Kernel started: c1081493-a48b-43df-ad65-c7adeb835b7e [D 2017-07-12 19:04:51.961 KernelGatewayApp] Kernel args: {'kernel_name': u'spark_2.1_scala_yarn_cluster', 'env': {'KERNEL_ID': u'c1081493-a48b-43df-ad65-c7adeb835b7e', 'LESSOPEN': '||/usr/bin/lesspipe.sh %s', 'ELYRA_REMOTE_USER': 'elyra', 'ELYRA_REMOTE_HOSTS': 'spark-02.softlayer.com,spark-03.softlayer.com,spark-04.softlayer.com,spark-05.softlayer.com', 'SELINUX_USE_CURRENT_RANGE': '', 'LOGNAME': 'elyra', 'USER': 'elyra', 'QTDIR': '/usr/lib64/qt-3.3', 'PATH': '/opt/anaconda2/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin', 'LANG': 'en_US.UTF-8', 'QTLIB': '/usr/lib64/qt-3.3/lib', 'TERM': 'xterm-256color', 'SHELL': '/bin/bash', 'ELYRA_PROXY_LAUNCH_LOG': '/var/log/elyra/proxy_launch_2017-07-12.log', 'QTINC': '/usr/lib64/qt-3.3/include', 'HISTSIZE': '1000', 'ELYRA_YARN_ENDPOINT': 'http://spark-02.softlayer.com:8088/ws/v1/cluster', 'HOME': '/home/elyra', 'XDG_RUNTIME_DIR': '/run/user/0', 'JUPYTER_DATA_DIR': '/var/lib/elyra', u'KERNEL_USERNAME': u'', 'SELINUX_ROLE_REQUESTED': '', 'ELYRA_KERNEL_LAUNCH_TIMEOUT': '120', 'MAIL': '/var/spool/mail/root', 'QT_GRAPHICSSYSTEM_CHECKED': '1', 'XDG_SESSION_ID': '131', '_': '/opt/anaconda2/bin/jupyter', 'SSH_TTY': '/dev/pts/0', 'HOSTNAME': 'spark-02.softlayer.com', 'SSH_CLIENT': '73.223.84.247 53893 22', 'SELINUX_LEVEL_REQUESTED': '', 'HISTCONTROL': 'ignoredups', 'SHLVL': '3', 'PWD': '/home/elyra', 'JUPYTER_RUNTIME_DIR': '/var/run/elyra/runtime', 'SSH_CONNECTION': '73.223.84.247 53893 169.45.103.134 22'}} [I 2017-07-12 19:04:51.962 KernelGatewayApp] Culling kernels with idle durations > 600 seconds at 30 second intervals ... [I 2017-07-12 19:04:51.962 KernelGatewayApp] Culling kernels even with connected clients [I 170712 19:04:51 web:2063] 201 POST /api/kernels (73.223.84.247) 11494.19ms [I 170712 19:04:52 web:2063] 200 GET /api/kernels/c1081493-a48b-43df-ad65-c7adeb835b7e (73.223.84.247) 0.68ms [D 2017-07-12 19:04:55.284 KernelGatewayApp] Initializing websocket connection /api/kernels/c1081493-a48b-43df-ad65-c7adeb835b7e/channels [W 2017-07-12 19:04:55.286 KernelGatewayApp] No session ID specified [D 2017-07-12 19:04:55.287 KernelGatewayApp] Requesting kernel info from c1081493-a48b-43df-ad65-c7adeb835b7e [D 2017-07-12 19:04:55.287 KernelGatewayApp] Connecting to: tcp://169.45.103.145:39864 [D 2017-07-12 19:04:58.179 KernelGatewayApp] activity on c1081493-a48b-43df-ad65-c7adeb835b7e: status [D 2017-07-12 19:04:58.187 KernelGatewayApp] Received kernel info: {u'implementation': u'spark', u'protocol_version': u'5.0', u'language_info': {u'mimetype': u'text/x-scala', u'name': u'scala', u'pygments_lexer': u'scala', u'version': u'2.11.8', u'file_extension': u'.scala', u'codemirror_mode': u'text/x-scala'}, u'implementation_version': u'0.2.0.dev1-incubating-SNAPSHOT', u'banner': u'Apache Toree'} [I 2017-07-12 19:04:58.187 KernelGatewayApp] Adapting to protocol v5.0 for kernel c1081493-a48b-43df-ad65-c7adeb835b7e [D 2017-07-12 19:04:58.188 KernelGatewayApp] activity on c1081493-a48b-43df-ad65-c7adeb835b7e: status [I 170712 19:04:58 web:2063] 101 GET /api/kernels/c1081493-a48b-43df-ad65-c7adeb835b7e/channels (73.223.84.247) 2904.45ms [D 2017-07-12 19:04:58.189 KernelGatewayApp] Opening websocket /api/kernels/c1081493-a48b-43df-ad65-c7adeb835b7e/channels [D 2017-07-12 19:04:58.189 KernelGatewayApp] Connecting to: tcp://169.45.103.145:39864 [D 2017-07-12 19:04:58.190 KernelGatewayApp] Connecting to: tcp://169.45.103.145:39681 [D 2017-07-12 19:04:58.190 KernelGatewayApp] Connecting to: tcp://169.45.103.145:38724 [D 2017-07-12 19:04:58.211 KernelGatewayApp] activity on c1081493-a48b-43df-ad65-c7adeb835b7e: status [D 2017-07-12 19:04:58.213 KernelGatewayApp] activity on c1081493-a48b-43df-ad65-c7adeb835b7e: status ======================== [elyra@spark-02 root]$ yarn logs --applicationId application_1499804611229_0007 17/07/12 19:16:57 INFO client.RMProxy: Connecting to ResourceManager at spark-02.softlayer.com/169.45.103.134:8050 17/07/12 19:16:57 INFO client.AHSProxy: Connecting to Application History server at spark-02.softlayer.com/169.45.103.134:10200 17/07/12 19:16:58 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 17/07/12 19:16:58 INFO compress.CodecPool: Got brand-new decompressor [.deflate] Container: container_1499804611229_0007_01_000002 on spark-02.softlayer.com_45454_1499904434725 LogAggregationType: AGGREGATED =============================================================================================== LogType:directory.info Log Upload Time:Wed Jul 12 19:07:14 -0500 2017 LogLength:32197 Log Contents: ls -l: total 20 lrwxrwxrwx. 1 yarn hadoop 73 Jul 12 19:04 __app__.jar -> /hadoop/yarn/local/usercache/elyra/filecache/14/toree-launcher_2.11-1.jar -rw-r--r--. 1 yarn hadoop 93 Jul 12 19:04 container_tokens -rwx------. 1 yarn hadoop 626 Jul 12 19:04 default_container_executor_session.sh -rwx------. 1 yarn hadoop 680 Jul 12 19:04 default_container_executor.sh -rwx------. 1 yarn hadoop 5830 Jul 12 19:04 launch_container.sh lrwxrwxrwx. 1 yarn hadoop 66 Jul 12 19:04 __spark_conf__ -> /hadoop/yarn/local/usercache/elyra/filecache/13/__spark_conf__.zip lrwxrwxrwx. 1 yarn hadoop 62 Jul 12 19:04 __spark_libs__ -> /hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz drwx--x---. 2 yarn hadoop 6 Jul 12 19:04 tmp lrwxrwxrwx. 1 yarn hadoop 97 Jul 12 19:04 toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar -> /hadoop/yarn/local/usercache/elyra/filecache/15/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar find -L . -maxdepth 5 -ls: 1078043287 4 drwx--x--- 3 yarn hadoop 4096 Jul 12 19:04 . 3223941401 0 drwx--x--- 2 yarn hadoop 6 Jul 12 19:04 ./tmp 1078043288 4 -rw-r--r-- 1 yarn hadoop 93 Jul 12 19:04 ./container_tokens 1078043289 4 -rw-r--r-- 1 yarn hadoop 12 Jul 12 19:04 ./.container_tokens.crc 1078043290 8 -rwx------ 1 yarn hadoop 5830 Jul 12 19:04 ./launch_container.sh 1078043291 4 -rw-r--r-- 1 yarn hadoop 56 Jul 12 19:04 ./.launch_container.sh.crc 1078043292 4 -rwx------ 1 yarn hadoop 626 Jul 12 19:04 ./default_container_executor_session.sh 1078043293 4 -rw-r--r-- 1 yarn hadoop 16 Jul 12 19:04 ./.default_container_executor_session.sh.crc 1078043294 4 -rwx------ 1 yarn hadoop 680 Jul 12 19:04 ./default_container_executor.sh 1078043295 4 -rw-r--r-- 1 yarn hadoop 16 Jul 12 19:04 ./.default_container_executor.sh.crc 2153013132 26948 -r-x------ 1 yarn hadoop 27591359 Jul 12 19:04 ./toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar 4458444 16 -r-x------ 1 yarn hadoop 14747 Jul 12 19:04 ./__app__.jar 2152901536 4 drwx------ 2 yarn hadoop 4096 Jul 12 19:04 ./__spark_conf__ 2152901537 8 -r-x------ 1 yarn hadoop 6846 Jul 12 19:04 ./__spark_conf__/mapred-site.xml 2152901538 8 -r-x------ 1 yarn hadoop 6369 Jul 12 19:04 ./__spark_conf__/hadoop-env.sh 2152901539 4 -r-x------ 1 yarn hadoop 620 Jul 12 19:04 ./__spark_conf__/log4j.properties 2152901540 4 -r-x------ 1 yarn hadoop 900 Jul 12 19:04 ./__spark_conf__/hadoop-metrics2.properties 2152901541 20 -r-x------ 1 yarn hadoop 17527 Jul 12 19:04 ./__spark_conf__/yarn-site.xml 2152901542 4 -r-x------ 1 yarn hadoop 3979 Jul 12 19:04 ./__spark_conf__/hadoop-env.cmd 2152901543 0 -r-x------ 1 yarn hadoop 0 Jul 12 19:04 ./__spark_conf__/yarn.exclude 2152901544 4 -r-x------ 1 yarn hadoop 1 Jul 12 19:04 ./__spark_conf__/dfs.exclude 2152901545 4 -r-x------ 1 yarn hadoop 2614 Jul 12 19:04 ./__spark_conf__/core-site.xml 2152901546 4 -r-x------ 1 yarn hadoop 1631 Jul 12 19:04 ./__spark_conf__/kms-log4j.properties 2152901547 4 -r-x------ 1 yarn hadoop 2250 Jul 12 19:04 ./__spark_conf__/yarn-env.cmd 2152901548 4 -r-x------ 1 yarn hadoop 884 Jul 12 19:04 ./__spark_conf__/ssl-client.xml 2152901549 4 -r-x------ 1 yarn hadoop 2035 Jul 12 19:04 ./__spark_conf__/capacity-scheduler.xml 2152901550 4 -r-x------ 1 yarn hadoop 3518 Jul 12 19:04 ./__spark_conf__/kms-acls.xml 2152901551 4 -r-x------ 1 yarn hadoop 2358 Jul 12 19:04 ./__spark_conf__/topology_script.py 2152901552 4 -r-x------ 1 yarn hadoop 758 Jul 12 19:04 ./__spark_conf__/mapred-site.xml.template 2152901553 4 -r-x------ 1 yarn hadoop 1335 Jul 12 19:04 ./__spark_conf__/configuration.xsl 2152901554 8 -r-x------ 1 yarn hadoop 5310 Jul 12 19:04 ./__spark_conf__/yarn-env.sh 2152901555 8 -r-x------ 1 yarn hadoop 6748 Jul 12 19:04 ./__spark_conf__/hdfs-site.xml 2152901556 4 -r-x------ 1 yarn hadoop 1020 Jul 12 19:04 ./__spark_conf__/commons-logging.properties 2152901557 4 -r-x------ 1 yarn hadoop 1019 Jul 12 19:04 ./__spark_conf__/container-executor.cfg 2152901558 8 -r-x------ 1 yarn hadoop 4221 Jul 12 19:04 ./__spark_conf__/task-log4j.properties 2152901559 4 -r-x------ 1 yarn hadoop 2490 Jul 12 19:04 ./__spark_conf__/hadoop-metrics.properties 2152901560 4 -r-x------ 1 yarn hadoop 856 Jul 12 19:04 ./__spark_conf__/mapred-env.sh 2152901561 4 -r-x------ 1 yarn hadoop 1602 Jul 12 19:04 ./__spark_conf__/health_check 2152901562 4 -r-x------ 1 yarn hadoop 2316 Jul 12 19:04 ./__spark_conf__/ssl-client.xml.example 2152901563 4 -r-x------ 1 yarn hadoop 1527 Jul 12 19:04 ./__spark_conf__/kms-env.sh 2152901564 4 -r-x------ 1 yarn hadoop 1308 Jul 12 19:04 ./__spark_conf__/hadoop-policy.xml 2152901565 4 -r-x------ 1 yarn hadoop 93 Jul 12 19:04 ./__spark_conf__/slaves 2152901566 4 -r-x------ 1 yarn hadoop 268 Jul 12 19:04 ./__spark_conf__/topology_mappings.data 2152901567 8 -r-x------ 1 yarn hadoop 4956 Jul 12 19:04 ./__spark_conf__/metrics.properties 2153001743 4 -r-x------ 1 yarn hadoop 951 Jul 12 19:04 ./__spark_conf__/mapred-env.cmd 2153001744 4 -r-x------ 1 yarn hadoop 1000 Jul 12 19:04 ./__spark_conf__/ssl-server.xml 2153013123 4 -r-x------ 1 yarn hadoop 2697 Jul 12 19:04 ./__spark_conf__/ssl-server.xml.example 2153013127 4 -r-x------ 1 yarn hadoop 945 Jul 12 19:04 ./__spark_conf__/taskcontroller.cfg 2153013128 8 -r-x------ 1 yarn hadoop 5511 Jul 12 19:04 ./__spark_conf__/kms-site.xml 2153013129 8 -r-x------ 1 yarn hadoop 4113 Jul 12 19:04 ./__spark_conf__/mapred-queues.xml.template 2153013130 4 -r-x------ 1 yarn hadoop 1884 Jul 12 19:04 ./__spark_conf__/__spark_conf__.properties 1085120575 16 drwxr-xr-x 2 yarn hadoop 12288 Jul 11 15:45 ./__spark_libs__ 1085120576 176 -r-xr-xr-x 1 yarn hadoop 178947 May 30 22:46 ./__spark_libs__/hk2-api-2.4.0-b34.jar 1085120577 20 -r-xr-xr-x 1 yarn hadoop 16993 May 30 22:46 ./__spark_libs__/JavaEWAH-0.3.2.jar 1085120578 332 -r-xr-xr-x 1 yarn hadoop 339666 May 30 22:46 ./__spark_libs__/datanucleus-api-jdo-3.2.6.jar 1085120579 200 -r-xr-xr-x 1 yarn hadoop 201928 May 30 22:46 ./__spark_libs__/RoaringBitmap-0.5.11.jar 1085120580 180 -r-xr-xr-x 1 yarn hadoop 181271 May 30 22:46 ./__spark_libs__/hk2-locator-2.4.0-b34.jar 1085120584 232 -r-xr-xr-x 1 yarn hadoop 236660 May 30 22:46 ./__spark_libs__/ST4-4.0.4.jar 1085120596 120 -r-xr-xr-x 1 yarn hadoop 118973 May 30 22:46 ./__spark_libs__/hk2-utils-2.4.0-b34.jar 1085120597 68 -r-xr-xr-x 1 yarn hadoop 69409 May 30 22:46 ./__spark_libs__/activation-1.1.1.jar 1085120598 320 -r-xr-xr-x 1 yarn hadoop 326724 May 30 22:46 ./__spark_libs__/httpcore-4.4.4.jar 1085120599 436 -r-xr-xr-x 1 yarn hadoop 445288 May 30 22:46 ./__spark_libs__/antlr-2.7.7.jar 1085120600 1848 -r-xr-xr-x 1 yarn hadoop 1890075 May 30 22:46 ./__spark_libs__/datanucleus-core-3.2.10.jar 1085120601 164 -r-xr-xr-x 1 yarn hadoop 164368 May 30 22:46 ./__spark_libs__/antlr-runtime-3.4.jar 1085120602 1768 -r-xr-xr-x 1 yarn hadoop 1809447 May 30 22:46 ./__spark_libs__/datanucleus-rdbms-3.2.9.jar 1085120603 296 -r-xr-xr-x 1 yarn hadoop 302248 May 30 22:46 ./__spark_libs__/antlr4-runtime-4.5.3.jar 1085120604 720 -r-xr-xr-x 1 yarn hadoop 736658 May 30 22:46 ./__spark_libs__/httpclient-4.5.2.jar 1085120605 8 -r-xr-xr-x 1 yarn hadoop 4467 May 30 22:46 ./__spark_libs__/aopalliance-1.0.jar 1085120606 132 -r-xr-xr-x 1 yarn hadoop 134595 May 30 22:46 ./__spark_libs__/commons-crypto-1.0.0.jar 1085120607 16 -r-xr-xr-x 1 yarn hadoop 14766 May 30 22:46 ./__spark_libs__/aopalliance-repackaged-2.4.0-b34.jar 1084669148 160 -r-xr-xr-x 1 yarn hadoop 160519 May 30 22:46 ./__spark_libs__/commons-dbcp-1.4.jar 1084669149 12 -r-xr-xr-x 1 yarn hadoop 12131 May 30 22:46 ./__spark_libs__/jpam-1.1.jar 1084669042 440 -r-xr-xr-x 1 yarn hadoop 448794 May 30 22:46 ./__spark_libs__/apache-log4j-extras-1.2.17.jar 1084669043 3152 -r-xr-xr-x 1 yarn hadoop 3224708 May 30 22:46 ./__spark_libs__/derby-10.12.1.1.jar 1084669044 44 -r-xr-xr-x 1 yarn hadoop 44925 May 30 22:46 ./__spark_libs__/apacheds-i18n-2.0.0-M15.jar 1084669045 508 -r-xr-xr-x 1 yarn hadoop 516634 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-app-2.7.3.2.6.1.0-129.jar 1084669046 676 -r-xr-xr-x 1 yarn hadoop 691479 May 30 22:46 ./__spark_libs__/apacheds-kerberos-codec-2.0.0-M15.jar 1084669047 20 -r-xr-xr-x 1 yarn hadoop 18482 May 30 22:46 ./__spark_libs__/eigenbase-properties-1.1.5.jar 1084669048 20 -r-xr-xr-x 1 yarn hadoop 16560 May 30 22:46 ./__spark_libs__/api-asn1-api-1.0.0-M20.jar 1084669049 188 -r-xr-xr-x 1 yarn hadoop 190432 May 30 22:46 ./__spark_libs__/gson-2.2.4.jar 1084669050 80 -r-xr-xr-x 1 yarn hadoop 79912 May 30 22:46 ./__spark_libs__/api-util-1.0.0-M20.jar 1084669051 2140 -r-xr-xr-x 1 yarn hadoop 2189117 May 30 22:46 ./__spark_libs__/guava-14.0.1.jar 1084668914 1168 -r-xr-xr-x 1 yarn hadoop 1194003 May 30 22:46 ./__spark_libs__/arpack_combined_all-0.1.jar 1084668915 1256 -r-xr-xr-x 1 yarn hadoop 1282424 May 30 22:46 ./__spark_libs__/ivy-2.4.0.jar 1084668916 428 -r-xr-xr-x 1 yarn hadoop 436303 May 30 22:46 ./__spark_libs__/avro-1.7.7.jar 1084668926 256 -r-xr-xr-x 1 yarn hadoop 258876 May 30 22:46 ./__spark_libs__/jackson-core-2.6.5.jar 1084668927 192 -r-xr-xr-x 1 yarn hadoop 192993 May 30 22:46 ./__spark_libs__/avro-ipc-1.7.7.jar 1084668928 144 -r-xr-xr-x 1 yarn hadoop 143602 May 30 22:46 ./__spark_libs__/commons-digester-1.8.jar 1084668929 180 -r-xr-xr-x 1 yarn hadoop 180736 May 30 22:46 ./__spark_libs__/avro-mapred-1.7.7-hadoop2.jar 1084668930 696 -r-xr-xr-x 1 yarn hadoop 710492 May 30 22:46 ./__spark_libs__/guice-3.0.jar 1084668931 16 -r-xr-xr-x 1 yarn hadoop 15071 May 30 22:46 ./__spark_libs__/jta-1.1.jar 1084668932 504 -r-xr-xr-x 1 yarn hadoop 516062 May 30 22:46 ./__spark_libs__/aws-java-sdk-core-1.10.6.jar 1084668933 64 -r-xr-xr-x 1 yarn hadoop 65012 May 30 22:46 ./__spark_libs__/guice-servlet-3.0.jar 1084668934 256 -r-xr-xr-x 1 yarn hadoop 258578 May 30 22:46 ./__spark_libs__/aws-java-sdk-kms-1.10.6.jar 1084668935 80 -r-xr-xr-x 1 yarn hadoop 79175 May 30 22:46 ./__spark_libs__/hadoop-auth-2.7.3.2.6.1.0-129.jar 1084668936 560 -r-xr-xr-x 1 yarn hadoop 570101 May 30 22:46 ./__spark_libs__/aws-java-sdk-s3-1.10.6.jar 1084668937 1988 -r-xr-xr-x 1 yarn hadoop 2035066 May 30 22:46 ./__spark_libs__/commons-math3-3.4.1.jar 1084668938 80 -r-xr-xr-x 1 yarn hadoop 78849 May 30 22:46 ./__spark_libs__/azure-data-lake-store-sdk-2.1.4.jar 1084668939 208 -r-xr-xr-x 1 yarn hadoop 212453 May 30 22:46 ./__spark_libs__/commons-net-2.2.jar 1084668940 232 -r-xr-xr-x 1 yarn hadoop 236880 May 30 22:46 ./__spark_libs__/lz4-1.3.0.jar 1084668941 12 -r-xr-xr-x 1 yarn hadoop 10092 May 30 22:46 ./__spark_libs__/azure-keyvault-core-0.8.0.jar 1084668942 220 -r-xr-xr-x 1 yarn hadoop 222041 May 30 22:46 ./__spark_libs__/hadoop-aws-2.7.3.2.6.1.0-129.jar 1084668943 728 -r-xr-xr-x 1 yarn hadoop 745325 May 30 22:46 ./__spark_libs__/azure-storage-4.2.0.jar 1084668944 48 -r-xr-xr-x 1 yarn hadoop 46983 May 30 22:46 ./__spark_libs__/jackson-annotations-2.6.5.jar 1084668945 20 -r-xr-xr-x 1 yarn hadoop 17008 May 30 22:46 ./__spark_libs__/base64-2.3.8.jar 1084668946 212 -r-xr-xr-x 1 yarn hadoop 213891 May 30 22:46 ./__spark_libs__/hadoop-azure-2.7.3.2.6.1.0-129.jar 1084668947 2780 -r-xr-xr-x 1 yarn hadoop 2842667 May 30 22:46 ./__spark_libs__/bcprov-jdk15on-1.51.jar 1084668948 4 -r-xr-xr-x 1 yarn hadoop 2567 May 30 22:46 ./__spark_libs__/hadoop-client-2.7.3.2.6.1.0-129.jar 1084668949 112 -r-xr-xr-x 1 yarn hadoop 110600 May 30 22:46 ./__spark_libs__/bonecp-0.8.0.RELEASE.jar 1084668950 3564 -r-xr-xr-x 1 yarn hadoop 3647240 May 30 22:46 ./__spark_libs__/hadoop-common-2.7.3.2.6.1.0-129.jar 1084668951 180 -r-xr-xr-x 1 yarn hadoop 181926 May 30 22:46 ./__spark_libs__/breeze-macros_2.11-0.12.jar 1084668768 228 -r-xr-xr-x 1 yarn hadoop 232248 May 30 22:46 ./__spark_libs__/jackson-core-asl-1.9.13.jar 1084668769 14080 -r-xr-xr-x 1 yarn hadoop 14413834 May 30 22:46 ./__spark_libs__/breeze_2.11-0.12.jar 1084668770 96 -r-xr-xr-x 1 yarn hadoop 96221 May 30 22:46 ./__spark_libs__/commons-pool-1.5.4.jar 1084668771 256 -r-xr-xr-x 1 yarn hadoop 258370 May 30 22:46 ./__spark_libs__/calcite-avatica-1.2.0-incubating.jar 1084668772 80 -r-xr-xr-x 1 yarn hadoop 79845 May 30 22:46 ./__spark_libs__/compress-lzf-1.0.3.jar 1084668773 3440 -r-xr-xr-x 1 yarn hadoop 3519262 May 30 22:46 ./__spark_libs__/calcite-core-1.2.0-incubating.jar 1084668774 164 -r-xr-xr-x 1 yarn hadoop 164422 May 30 22:46 ./__spark_libs__/core-1.1.2.jar 1084668775 64 -r-xr-xr-x 1 yarn hadoop 65261 May 30 22:46 ./__spark_libs__/oro-2.0.8.jar 1084668776 436 -r-xr-xr-x 1 yarn hadoop 442406 May 30 22:46 ./__spark_libs__/calcite-linq4j-1.2.0-incubating.jar 1084668777 1144 -r-xr-xr-x 1 yarn hadoop 1171380 May 30 22:46 ./__spark_libs__/jackson-databind-2.6.5.jar 1084668778 52 -r-xr-xr-x 1 yarn hadoop 50619 May 30 22:46 ./__spark_libs__/chill-java-0.8.0.jar 1084668779 20 -r-xr-xr-x 1 yarn hadoop 18336 May 30 22:46 ./__spark_libs__/jackson-jaxrs-1.9.13.jar 1084668780 220 -r-xr-xr-x 1 yarn hadoop 223573 May 30 22:46 ./__spark_libs__/chill_2.11-0.8.0.jar 1084668781 8508 -r-xr-xr-x 1 yarn hadoop 8711541 May 30 22:46 ./__spark_libs__/hadoop-hdfs-2.7.3.2.6.1.0-129.jar 1084668785 188 -r-xr-xr-x 1 yarn hadoop 188671 May 30 22:46 ./__spark_libs__/commons-beanutils-1.7.0.jar 1084668654 68 -r-xr-xr-x 1 yarn hadoop 68866 May 30 22:46 ./__spark_libs__/curator-client-2.6.0.jar 1084668655 204 -r-xr-xr-x 1 yarn hadoop 206035 May 30 22:46 ./__spark_libs__/commons-beanutils-core-1.8.0.jar 1084668656 764 -r-xr-xr-x 1 yarn hadoop 780664 May 30 22:46 ./__spark_libs__/jackson-mapper-asl-1.9.13.jar 1084668657 44 -r-xr-xr-x 1 yarn hadoop 41123 May 30 22:46 ./__spark_libs__/commons-cli-1.2.jar 1084668658 2068 -r-xr-xr-x 1 yarn hadoop 2117555 May 30 22:46 ./__spark_libs__/hadoop-yarn-api-2.7.3.2.6.1.0-129.jar 1084668659 280 -r-xr-xr-x 1 yarn hadoop 284184 May 30 22:46 ./__spark_libs__/commons-codec-1.10.jar 1084668660 184 -r-xr-xr-x 1 yarn hadoop 185245 May 30 22:46 ./__spark_libs__/curator-framework-2.6.0.jar 1084668661 576 -r-xr-xr-x 1 yarn hadoop 588337 May 30 22:46 ./__spark_libs__/commons-collections-3.2.2.jar 1084668662 136 -r-xr-xr-x 1 yarn hadoop 138453 May 30 22:46 ./__spark_libs__/hive-beeline-1.2.1.spark2.hdp.1.jar 1084668663 36 -r-xr-xr-x 1 yarn hadoop 36627 May 30 22:46 ./__spark_libs__/commons-compiler-3.0.0.jar 1084668664 40 -r-xr-xr-x 1 yarn hadoop 40864 May 30 22:46 ./__spark_libs__/hive-cli-1.2.1.spark2.hdp.1.jar 1084668665 236 -r-xr-xr-x 1 yarn hadoop 241367 May 30 22:46 ./__spark_libs__/commons-compress-1.4.1.jar 1084668666 244 -r-xr-xr-x 1 yarn hadoop 248171 May 30 22:46 ./__spark_libs__/curator-recipes-2.6.0.jar 1084668667 292 -r-xr-xr-x 1 yarn hadoop 298829 May 30 22:46 ./__spark_libs__/commons-configuration-1.6.jar 1084668668 300 -r-xr-xr-x 1 yarn hadoop 305001 May 30 22:46 ./__spark_libs__/commons-httpclient-3.1.jar 1084668669 28 -r-xr-xr-x 1 yarn hadoop 27084 May 30 22:46 ./__spark_libs__/jackson-xc-1.9.13.jar 1084668670 184 -r-xr-xr-x 1 yarn hadoop 185140 May 30 22:46 ./__spark_libs__/commons-io-2.4.jar 1084668671 12 -r-xr-xr-x 1 yarn hadoop 10023 May 30 22:46 ./__spark_libs__/java-xmlbuilder-1.0.jar 1084668682 280 -r-xr-xr-x 1 yarn hadoop 284220 May 30 22:46 ./__spark_libs__/commons-lang-2.6.jar 1084668683 11232 -r-xr-xr-x 1 yarn hadoop 11499183 May 30 22:46 ./__spark_libs__/hive-exec-1.2.1.spark2.hdp.1.jar 1084668701 472 -r-xr-xr-x 1 yarn hadoop 479881 May 30 22:46 ./__spark_libs__/commons-lang3-3.5.jar 1084668702 100 -r-xr-xr-x 1 yarn hadoop 101402 May 30 22:46 ./__spark_libs__/hive-jdbc-1.2.1.spark2.hdp.1.jar 1084668703 64 -r-xr-xr-x 1 yarn hadoop 62050 May 30 22:46 ./__spark_libs__/commons-logging-1.1.3.jar 1084668585 720 -r-xr-xr-x 1 yarn hadoop 735574 May 30 22:46 ./__spark_libs__/janino-3.0.0.jar 1084668586 20 -r-xr-xr-x 1 yarn hadoop 17369 May 30 22:46 ./__spark_libs__/hadoop-annotations-2.7.3.2.6.1.0-129.jar 1084668594 24 -r-xr-xr-x 1 yarn hadoop 23519 May 30 22:46 ./__spark_libs__/hadoop-azure-datalake-2.7.3.2.6.1.0-129.jar 1084668597 52 -r-xr-xr-x 1 yarn hadoop 51484 May 30 22:46 ./__spark_libs__/json-smart-1.1.1.jar 1084668598 280 -r-xr-xr-x 1 yarn hadoop 285479 May 30 22:46 ./__spark_libs__/parquet-encoding-1.8.1.jar 1084668599 736 -r-xr-xr-x 1 yarn hadoop 752754 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-common-2.7.3.2.6.1.0-129.jar 1084667363 96 -r-xr-xr-x 1 yarn hadoop 94672 May 30 22:46 ./__spark_libs__/xz-1.0.jar 1084668600 84 -r-xr-xr-x 1 yarn hadoop 82421 May 30 22:46 ./__spark_libs__/json4s-ast_2.11-3.2.11.jar 1084668601 788 -r-xr-xr-x 1 yarn hadoop 802818 May 30 22:46 ./__spark_libs__/scalap-2.11.8.jar 1084668602 1500 -r-xr-xr-x 1 yarn hadoop 1535771 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-core-2.7.3.2.6.1.0-129.jar 1084668603 96 -r-xr-xr-x 1 yarn hadoop 94796 May 30 22:46 ./__spark_libs__/pyrolite-4.13.jar 1084668034 116 -r-xr-xr-x 1 yarn hadoop 115472 May 30 22:46 ./__spark_libs__/py4j-0.10.4.jar 1084668044 40 -r-xr-xr-x 1 yarn hadoop 39398 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-jobclient-2.7.3.2.6.1.0-129.jar 1084668083 420 -r-xr-xr-x 1 yarn hadoop 427780 May 30 22:46 ./__spark_libs__/jodd-core-3.5.2.jar 1084667990 40 -r-xr-xr-x 1 yarn hadoop 39280 May 30 22:46 ./__spark_libs__/metrics-jvm-3.1.2.jar 1084667993 56 -r-xr-xr-x 1 yarn hadoop 54074 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-shuffle-2.7.3.2.6.1.0-129.jar 1084667997 36 -r-xr-xr-x 1 yarn hadoop 33015 May 30 22:46 ./__spark_libs__/jsr305-1.3.9.jar 1084668001 40 -r-xr-xr-x 1 yarn hadoop 40509 May 30 22:46 ./__spark_libs__/slf4j-api-1.7.16.jar 1084668010 116 -r-xr-xr-x 1 yarn hadoop 114748 May 30 22:46 ./__spark_libs__/hadoop-openstack-2.7.3.2.6.1.0-129.jar 1084668018 748 -r-xr-xr-x 1 yarn hadoop 764569 May 30 22:46 ./__spark_libs__/jtransforms-2.4.0.jar 1084667360 48 -r-xr-xr-x 1 yarn hadoop 48720 May 30 22:46 ./__spark_libs__/snappy-0.2.jar 1084667361 192 -r-xr-xr-x 1 yarn hadoop 193240 May 30 22:46 ./__spark_libs__/hadoop-yarn-client-2.7.3.2.6.1.0-129.jar 1084667362 8 -r-xr-xr-x 1 yarn hadoop 4596 May 30 22:46 ./__spark_libs__/jul-to-slf4j-1.7.16.jar 1084667364 1712 -r-xr-xr-x 1 yarn hadoop 1750536 May 30 22:46 ./__spark_libs__/hadoop-yarn-common-2.7.3.2.6.1.0-129.jar 1084667365 352 -r-xr-xr-x 1 yarn hadoop 358390 May 30 22:46 ./__spark_libs__/kryo-shaded-3.0.3.jar 1084667366 7108 -r-xr-xr-x 1 yarn hadoop 7276083 May 30 22:46 ./__spark_libs__/spire_2.11-0.7.4.jar 1084667367 100 -r-xr-xr-x 1 yarn hadoop 101126 May 30 22:46 ./__spark_libs__/hadoop-yarn-registry-2.7.3.2.6.1.0-129.jar 1084667368 576 -r-xr-xr-x 1 yarn hadoop 589462 May 30 22:46 ./__spark_libs__/json4s-core_2.11-3.2.11.jar 1084667369 24 -r-xr-xr-x 1 yarn hadoop 23346 May 30 22:46 ./__spark_libs__/stax-api-1.0-2.jar 1084667370 412 -r-xr-xr-x 1 yarn hadoop 420788 May 30 22:46 ./__spark_libs__/hadoop-yarn-server-common-2.7.3.2.6.1.0-129.jar 1084667372 40 -r-xr-xr-x 1 yarn hadoop 40341 May 30 22:46 ./__spark_libs__/json4s-jackson_2.11-3.2.11.jar 1084667374 36 -r-xr-xr-x 1 yarn hadoop 36249 May 30 22:46 ./__spark_libs__/hadoop-yarn-server-web-proxy-2.7.3.2.6.1.0-129.jar 1084667375 1024 -r-xr-xr-x 1 yarn hadoop 1045744 May 30 22:46 ./__spark_libs__/leveldbjni-all-1.8.jar 1084667376 5380 -r-xr-xr-x 1 yarn hadoop 5505070 May 30 22:46 ./__spark_libs__/hive-metastore-1.2.1.spark2.hdp.1.jar 1084667377 232 -r-xr-xr-x 1 yarn hadoop 234201 May 30 22:46 ./__spark_libs__/libthrift-0.9.3.jar 1084667393 1444 -r-xr-xr-x 1 yarn hadoop 1475955 May 30 22:46 ./__spark_libs__/htrace-core-3.1.0-incubating.jar 1084667394 480 -r-xr-xr-x 1 yarn hadoop 489884 May 30 22:46 ./__spark_libs__/log4j-1.2.17.jar 1084667395 48 -r-xr-xr-x 1 yarn hadoop 48418 May 30 22:46 ./__spark_libs__/jackson-dataformat-cbor-2.6.5.jar 1084667396 512 -r-xr-xr-x 1 yarn hadoop 521157 May 30 22:46 ./__spark_libs__/mail-1.4.7.jar 1084515136 44 -r-xr-xr-x 1 yarn hadoop 41263 May 30 22:46 ./__spark_libs__/jackson-module-paranamer-2.6.5.jar 1084515148 112 -r-xr-xr-x 1 yarn hadoop 112558 May 30 22:46 ./__spark_libs__/metrics-core-3.1.2.jar 1084515153 504 -r-xr-xr-x 1 yarn hadoop 515604 May 30 22:46 ./__spark_libs__/jackson-module-scala_2.11-2.6.5.jar 1084515154 8 -r-xr-xr-x 1 yarn hadoop 5711 May 30 22:46 ./__spark_libs__/minlog-1.3.0.jar 1084515155 700 -r-xr-xr-x 1 yarn hadoop 714194 May 30 22:46 ./__spark_libs__/javassist-3.18.1-GA.jar 1084515156 400 -r-xr-xr-x 1 yarn hadoop 409467 May 30 22:46 ./__spark_libs__/mx4j-3.0.2.jar 1084515157 28 -r-xr-xr-x 1 yarn hadoop 26366 May 30 22:46 ./__spark_libs__/javax.annotation-api-1.2.jar 1084515163 216 -r-xr-xr-x 1 yarn hadoop 218076 May 30 22:46 ./__spark_libs__/parquet-hadoop-1.8.1.jar 1084515164 4 -r-xr-xr-x 1 yarn hadoop 2497 May 30 22:46 ./__spark_libs__/javax.inject-1.jar 1084515165 1204 -r-xr-xr-x 1 yarn hadoop 1230201 May 30 22:46 ./__spark_libs__/netty-3.8.0.Final.jar 1084515166 8 -r-xr-xr-x 1 yarn hadoop 5950 May 30 22:46 ./__spark_libs__/javax.inject-2.4.0-b34.jar 1084515167 2264 -r-xr-xr-x 1 yarn hadoop 2314839 May 30 22:46 ./__spark_libs__/netty-all-4.0.42.Final.jar 1084515169 96 -r-xr-xr-x 1 yarn hadoop 95806 May 30 22:46 ./__spark_libs__/javax.servlet-api-3.1.0.jar 1084515170 176 -r-xr-xr-x 1 yarn hadoop 178648 May 30 22:46 ./__spark_libs__/nimbus-jose-jwt-3.9.jar 1084515171 116 -r-xr-xr-x 1 yarn hadoop 115534 May 30 22:46 ./__spark_libs__/javax.ws.rs-api-2.0.1.jar 1084515172 1024 -r-xr-xr-x 1 yarn hadoop 1048117 May 30 22:46 ./__spark_libs__/parquet-jackson-1.8.1.jar 1084515173 388 -r-xr-xr-x 1 yarn hadoop 395195 May 30 22:46 ./__spark_libs__/javolution-5.5.1.jar 1084515176 640 -r-xr-xr-x 1 yarn hadoop 654216 May 30 22:46 ./__spark_libs__/pmml-model-1.2.15.jar 1084515177 104 -r-xr-xr-x 1 yarn hadoop 105134 May 30 22:46 ./__spark_libs__/jaxb-api-2.2.2.jar 1084515178 44 -r-xr-xr-x 1 yarn hadoop 41755 May 30 22:46 ./__spark_libs__/objenesis-2.1.jar 1084515179 4 -r-xr-xr-x 1 yarn hadoop 2254 May 30 22:46 ./__spark_libs__/jcip-annotations-1.0.jar 1084515180 312 -r-xr-xr-x 1 yarn hadoop 319099 May 30 22:46 ./__spark_libs__/okhttp-2.4.0.jar 1084515181 20 -r-xr-xr-x 1 yarn hadoop 16430 May 30 22:46 ./__spark_libs__/jcl-over-slf4j-1.7.16.jar 1084515182 8 -r-xr-xr-x 1 yarn hadoop 5310 May 30 22:46 ./__spark_libs__/pmml-schema-1.2.15.jar 1084515183 200 -r-xr-xr-x 1 yarn hadoop 201124 May 30 22:46 ./__spark_libs__/jdo-api-3.0.1.jar 1084515184 64 -r-xr-xr-x 1 yarn hadoop 64661 May 30 22:46 ./__spark_libs__/okio-1.4.0.jar 1084515185 164 -r-xr-xr-x 1 yarn hadoop 167421 May 30 22:46 ./__spark_libs__/jersey-client-2.22.2.jar 1084515186 20 -r-xr-xr-x 1 yarn hadoop 19827 May 30 22:46 ./__spark_libs__/opencsv-2.3.jar 1084515187 684 -r-xr-xr-x 1 yarn hadoop 698375 May 30 22:46 ./__spark_libs__/jersey-common-2.22.2.jar 1084515188 24 -r-xr-xr-x 1 yarn hadoop 20852 May 30 22:46 ./__spark_libs__/metrics-graphite-3.1.2.jar 1084515189 20 -r-xr-xr-x 1 yarn hadoop 18098 May 30 22:46 ./__spark_libs__/jersey-container-servlet-2.22.2.jar 1084515190 308 -r-xr-xr-x 1 yarn hadoop 313702 May 30 22:46 ./__spark_libs__/libfb303-0.9.3.jar 1084515191 28 -r-xr-xr-x 1 yarn hadoop 26514 May 30 22:46 ./__spark_libs__/stax-api-1.0.1.jar 1084515192 68 -r-xr-xr-x 1 yarn hadoop 66270 May 30 22:46 ./__spark_libs__/jersey-container-servlet-core-2.22.2.jar 1084515193 20 -r-xr-xr-x 1 yarn hadoop 20235 May 30 22:46 ./__spark_libs__/osgi-resource-locator-1.0.1.jar 1084515194 952 -r-xr-xr-x 1 yarn hadoop 971310 May 30 22:46 ./__spark_libs__/jersey-guava-2.22.2.jar 1084515196 32 -r-xr-xr-x 1 yarn hadoop 29555 May 30 22:46 ./__spark_libs__/paranamer-2.3.jar 1084515197 72 -r-xr-xr-x 1 yarn hadoop 72733 May 30 22:46 ./__spark_libs__/jersey-media-jaxb-2.22.2.jar 1084515198 924 -r-xr-xr-x 1 yarn hadoop 945914 May 30 22:46 ./__spark_libs__/parquet-column-1.8.1.jar 1084515199 932 -r-xr-xr-x 1 yarn hadoop 951701 May 30 22:46 ./__spark_libs__/jersey-server-2.22.2.jar 1084515008 524 -r-xr-xr-x 1 yarn hadoop 533455 May 30 22:46 ./__spark_libs__/protobuf-java-2.5.0.jar 1084515009 1996 -r-xr-xr-x 1 yarn hadoop 2041628 May 30 22:46 ./__spark_libs__/jets3t-0.9.3.jar 1084515010 5612 -r-xr-xr-x 1 yarn hadoop 5744974 May 30 22:46 ./__spark_libs__/scala-library-2.11.8.jar 1084515011 528 -r-xr-xr-x 1 yarn hadoop 540298 May 30 22:46 ./__spark_libs__/jetty-6.1.26.hwx.jar 1084515012 16 -r-xr-xr-x 1 yarn hadoop 15827 May 30 22:46 ./__spark_libs__/metrics-json-3.1.2.jar 1084515013 20 -r-xr-xr-x 1 yarn hadoop 18899 May 30 22:46 ./__spark_libs__/jetty-sslengine-6.1.26.hwx.jar 1084515014 40 -r-xr-xr-x 1 yarn hadoop 38604 May 30 22:46 ./__spark_libs__/parquet-common-1.8.1.jar 1084515015 176 -r-xr-xr-x 1 yarn hadoop 177214 May 30 22:46 ./__spark_libs__/jetty-util-6.1.26.hwx.jar 1084515016 15128 -r-xr-xr-x 1 yarn hadoop 15487351 May 30 22:46 ./__spark_libs__/scala-compiler-2.11.8.jar 1084515017 212 -r-xr-xr-x 1 yarn hadoop 213911 May 30 22:46 ./__spark_libs__/jline-2.12.1.jar 1084515018 4468 -r-xr-xr-x 1 yarn hadoop 4573750 May 30 22:46 ./__spark_libs__/scala-reflect-2.11.8.jar 1084515019 616 -r-xr-xr-x 1 yarn hadoop 627814 May 30 22:46 ./__spark_libs__/joda-time-2.9.3.jar 1084515020 100 -r-xr-xr-x 1 yarn hadoop 100636 May 30 22:46 ./__spark_libs__/jsp-api-2.1.jar 1084515021 636 -r-xr-xr-x 1 yarn hadoop 648678 May 30 22:46 ./__spark_libs__/scala-xml_2.11-1.0.2.jar 1084515022 380 -r-xr-xr-x 1 yarn hadoop 387188 May 30 22:46 ./__spark_libs__/parquet-format-2.3.0-incubating.jar 1084515023 2732 -r-xr-xr-x 1 yarn hadoop 2796935 May 30 22:46 ./__spark_libs__/parquet-hadoop-bundle-1.6.0.jar 1084515024 416 -r-xr-xr-x 1 yarn hadoop 423753 May 30 22:46 ./__spark_libs__/scala-parser-combinators_2.11-1.0.4.jar 1084515025 2004 -r-xr-xr-x 1 yarn hadoop 2049451 May 30 22:46 ./__spark_libs__/shapeless_2.11-2.0.0.jar 1084515026 12 -r-xr-xr-x 1 yarn hadoop 9939 May 30 22:46 ./__spark_libs__/slf4j-log4j12-1.7.16.jar 1084515027 1032 -r-xr-xr-x 1 yarn hadoop 1056168 May 30 22:46 ./__spark_libs__/snappy-java-1.1.2.6.jar 1084515028 7336 -r-xr-xr-x 1 yarn hadoop 7508336 May 30 22:46 ./__spark_libs__/spark-catalyst_2.11-2.1.1.2.6.1.0-129.jar 1084515029 84 -r-xr-xr-x 1 yarn hadoop 82916 May 30 22:46 ./__spark_libs__/spark-cloud_2.11-2.1.1.2.6.1.0-129.jar 1084515030 11668 -r-xr-xr-x 1 yarn hadoop 11946206 May 30 22:46 ./__spark_libs__/spark-core_2.11-2.1.1.2.6.1.0-129.jar 1084515031 684 -r-xr-xr-x 1 yarn hadoop 699290 May 30 22:46 ./__spark_libs__/spark-graphx_2.11-2.1.1.2.6.1.0-129.jar 1084515032 1776 -r-xr-xr-x 1 yarn hadoop 1815017 May 30 22:46 ./__spark_libs__/spark-hive-thriftserver_2.11-2.1.1.2.6.1.0-129.jar 1084515033 1208 -r-xr-xr-x 1 yarn hadoop 1234838 May 30 22:46 ./__spark_libs__/spark-hive_2.11-2.1.1.2.6.1.0-129.jar 1084515054 88 -r-xr-xr-x 1 yarn hadoop 86811 May 30 22:46 ./__spark_libs__/spire-macros_2.11-0.7.4.jar 1084515034 72 -r-xr-xr-x 1 yarn hadoop 70595 May 30 22:46 ./__spark_libs__/spark-launcher_2.11-2.1.1.2.6.1.0-129.jar 1084515035 176 -r-xr-xr-x 1 yarn hadoop 178967 May 30 22:46 ./__spark_libs__/spark-mllib-local_2.11-2.1.1.2.6.1.0-129.jar 1084515036 6048 -r-xr-xr-x 1 yarn hadoop 6190594 May 30 22:46 ./__spark_libs__/spark-mllib_2.11-2.1.1.2.6.1.0-129.jar 1084515040 2304 -r-xr-xr-x 1 yarn hadoop 2356803 May 30 22:46 ./__spark_libs__/spark-network-common_2.11-2.1.1.2.6.1.0-129.jar 1084515046 64 -r-xr-xr-x 1 yarn hadoop 61843 May 30 22:46 ./__spark_libs__/spark-network-shuffle_2.11-2.1.1.2.6.1.0-129.jar 1084515047 68 -r-xr-xr-x 1 yarn hadoop 65842 May 30 22:46 ./__spark_libs__/spark-repl_2.11-2.1.1.2.6.1.0-129.jar 1084515048 32 -r-xr-xr-x 1 yarn hadoop 29765 May 30 22:46 ./__spark_libs__/spark-sketch_2.11-2.1.1.2.6.1.0-129.jar 1084515049 6504 -r-xr-xr-x 1 yarn hadoop 6657160 May 30 22:46 ./__spark_libs__/spark-sql_2.11-2.1.1.2.6.1.0-129.jar 1084515050 2052 -r-xr-xr-x 1 yarn hadoop 2101086 May 30 22:46 ./__spark_libs__/spark-streaming_2.11-2.1.1.2.6.1.0-129.jar 1084515051 16 -r-xr-xr-x 1 yarn hadoop 15527 May 30 22:46 ./__spark_libs__/spark-tags_2.11-2.1.1.2.6.1.0-129.jar 1084515052 48 -r-xr-xr-x 1 yarn hadoop 45278 May 30 22:46 ./__spark_libs__/spark-unsafe_2.11-2.1.1.2.6.1.0-129.jar 1084515053 728 -r-xr-xr-x 1 yarn hadoop 742074 May 30 22:46 ./__spark_libs__/spark-yarn_2.11-2.1.1.2.6.1.0-129.jar 1084515055 172 -r-xr-xr-x 1 yarn hadoop 174351 May 30 22:46 ./__spark_libs__/stream-2.7.0.jar 1084515061 148 -r-xr-xr-x 1 yarn hadoop 148627 May 30 22:46 ./__spark_libs__/stringtemplate-3.2.1.jar 1084515062 92 -r-xr-xr-x 1 yarn hadoop 93210 May 30 22:46 ./__spark_libs__/super-csv-2.2.0.jar 1084515063 320 -r-xr-xr-x 1 yarn hadoop 325553 May 30 22:46 ./__spark_libs__/univocity-parsers-2.2.1.jar 1084515064 64 -r-xr-xr-x 1 yarn hadoop 63777 May 30 22:46 ./__spark_libs__/validation-api-1.1.0.Final.jar 1084515065 144 -r-xr-xr-x 1 yarn hadoop 144660 May 30 22:46 ./__spark_libs__/xbean-asm5-shaded-4.4.jar 1084515066 1204 -r-xr-xr-x 1 yarn hadoop 1229125 May 30 22:46 ./__spark_libs__/xercesImpl-2.9.1.jar 1084515067 16 -r-xr-xr-x 1 yarn hadoop 15010 May 30 22:46 ./__spark_libs__/xmlenc-0.52.jar 1084515069 780 -r-xr-xr-x 1 yarn hadoop 797314 May 30 22:46 ./__spark_libs__/zookeeper-3.4.6.2.6.1.0-129.jar broken symlinks(find -L . -maxdepth 5 -type l -ls): End of LogType:directory.info LogType:launch_container.sh Log Upload Time:Wed Jul 12 19:07:14 -0500 2017 LogLength:5830 Log Contents: #!/bin/bash export SPARK_YARN_STAGING_DIR="hdfs://spark-02.softlayer.com:8020/user/elyra/.sparkStaging/application_1499804611229_0007" export PATH="/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent" export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/current/hadoop-client/conf"} export JAVA_HOME=${JAVA_HOME:-"/usr/jdk64/jdk1.8.0_112"} export LANG="en_US.UTF-8" export SPARK_LOG_URL_STDOUT="http://spark-02.softlayer.com:8042/node/containerlogs/container_1499804611229_0007_01_000002/elyra/stdout?start=-4096" export NM_HOST="spark-02.softlayer.com" export LOGNAME="elyra" export JVM_PID="$$" export PWD="/hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007/container_1499804611229_0007_01_000002" export LOCAL_DIRS="/hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007" export NM_HTTP_PORT="8042" export LOG_DIRS="/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002" export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= " export NM_PORT="45454" export USER="elyra" export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/current/hadoop-yarn-nodemanager"} export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/etc/hadoop/conf:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.1.0-129/hadoop/lib/hadoop-lzo-0.6.0.2.6.1.0-129.jar:/etc/hadoop/conf/secure" export SPARK_YARN_MODE="true" export HADOOP_TOKEN_FILE_LOCATION="/hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007/container_1499804611229_0007_01_000002/container_tokens" export NM_AUX_SERVICE_spark_shuffle="" export SPARK_USER="elyra" export LOCAL_USER_DIRS="/hadoop/yarn/local/usercache/elyra/" export HADOOP_HOME="/usr/hdp/2.6.1.0-129/hadoop" export SPARK_LOG_URL_STDERR="http://spark-02.softlayer.com:8042/node/containerlogs/container_1499804611229_0007_01_000002/elyra/stderr?start=-4096" export HOME="/home/" export NM_AUX_SERVICE_spark2_shuffle="" export CONTAINER_ID="container_1499804611229_0007_01_000002" export MALLOC_ARENA_MAX="4" ln -sf "/hadoop/yarn/local/usercache/elyra/filecache/15/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar" "toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/hadoop/yarn/local/usercache/elyra/filecache/14/toree-launcher_2.11-1.jar" "__app__.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/hadoop/yarn/local/usercache/elyra/filecache/13/__spark_conf__.zip" "__spark_conf__" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz" "__spark_libs__" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi # Creating copy of launch script cp "launch_container.sh" "/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002/launch_container.sh" chmod 640 "/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002/launch_container.sh" # Determining directory contents echo "ls -l:" 1>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002/directory.info" ls -l 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002/directory.info" find -L . -maxdepth 5 -ls 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002/directory.info" exec /bin/bash -c "LD_LIBRARY_PATH="/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp '-Dspark.history.ui.port=18081' '-Dspark.ui.port=0' '-Dspark.driver.port=44474' -Dspark.yarn.app.container.log.dir=/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@169.45.103.145:44474 --executor-id 1 --hostname spark-02.softlayer.com --cores 1 --app-id application_1499804611229_0007 --user-class-path file:$PWD/__app__.jar --user-class-path file:$PWD/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar 1>/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002/stdout 2>/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000002/stderr" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi End of LogType:launch_container.sh LogType:stderr Log Upload Time:Wed Jul 12 19:07:14 -0500 2017 LogLength:3795 Log Contents: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.1.0-129/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 17/07/12 19:04:48 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 44503@spark-02.softlayer.com 17/07/12 19:04:48 INFO SignalUtils: Registered signal handler for TERM 17/07/12 19:04:48 INFO SignalUtils: Registered signal handler for HUP 17/07/12 19:04:48 INFO SignalUtils: Registered signal handler for INT 17/07/12 19:04:49 INFO SecurityManager: Changing view acls to: yarn,elyra 17/07/12 19:04:49 INFO SecurityManager: Changing modify acls to: yarn,elyra 17/07/12 19:04:49 INFO SecurityManager: Changing view acls groups to: 17/07/12 19:04:49 INFO SecurityManager: Changing modify acls groups to: 17/07/12 19:04:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, elyra); groups with view permissions: Set(); users with modify permissions: Set(yarn, elyra); groups with modify permissions: Set() 17/07/12 19:04:49 INFO TransportClientFactory: Successfully created connection to /169.45.103.145:44474 after 61 ms (0 ms spent in bootstraps) 17/07/12 19:04:49 INFO SecurityManager: Changing view acls to: yarn,elyra 17/07/12 19:04:49 INFO SecurityManager: Changing modify acls to: yarn,elyra 17/07/12 19:04:49 INFO SecurityManager: Changing view acls groups to: 17/07/12 19:04:49 INFO SecurityManager: Changing modify acls groups to: 17/07/12 19:04:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, elyra); groups with view permissions: Set(); users with modify permissions: Set(yarn, elyra); groups with modify permissions: Set() 17/07/12 19:04:49 INFO TransportClientFactory: Successfully created connection to /169.45.103.145:44474 after 1 ms (0 ms spent in bootstraps) 17/07/12 19:04:49 INFO DiskBlockManager: Created local directory at /hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007/blockmgr-07a89e64-c16d-4f3a-ae8a-3c78e983b479 17/07/12 19:04:49 INFO MemoryStore: MemoryStore started with capacity 366.3 MB 17/07/12 19:04:50 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@169.45.103.145:44474 17/07/12 19:04:50 INFO CoarseGrainedExecutorBackend: Successfully registered with driver 17/07/12 19:04:50 INFO Executor: Starting executor ID 1 on host spark-02.softlayer.com 17/07/12 19:04:50 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 53007. 17/07/12 19:04:50 INFO NettyBlockTransferService: Server created on spark-02.softlayer.com:53007 17/07/12 19:04:50 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 17/07/12 19:04:50 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, spark-02.softlayer.com, 53007, None) 17/07/12 19:04:50 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, spark-02.softlayer.com, 53007, None) 17/07/12 19:04:50 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, spark-02.softlayer.com, 53007, None) 17/07/12 19:07:13 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown 17/07/12 19:07:13 INFO MemoryStore: MemoryStore cleared 17/07/12 19:07:13 INFO BlockManager: BlockManager stopped 17/07/12 19:07:13 INFO ShutdownHookManager: Shutdown hook called End of LogType:stderr LogType:stdout Log Upload Time:Wed Jul 12 19:07:14 -0500 2017 LogLength:0 Log Contents: End of LogType:stdout Container: container_1499804611229_0007_01_000003 on spark-04.softlayer.com_45454_1499904435190 LogAggregationType: AGGREGATED =============================================================================================== LogType:directory.info Log Upload Time:Wed Jul 12 19:07:15 -0500 2017 LogLength:31315 Log Contents: ls -l: total 20 lrwxrwxrwx. 1 yarn hadoop 73 Jul 12 19:04 __app__.jar -> /hadoop/yarn/local/usercache/elyra/filecache/14/toree-launcher_2.11-1.jar -rw-r--r--. 1 yarn hadoop 93 Jul 12 19:04 container_tokens -rwx------. 1 yarn hadoop 626 Jul 12 19:04 default_container_executor_session.sh -rwx------. 1 yarn hadoop 680 Jul 12 19:04 default_container_executor.sh -rwx------. 1 yarn hadoop 5830 Jul 12 19:04 launch_container.sh lrwxrwxrwx. 1 yarn hadoop 66 Jul 12 19:04 __spark_conf__ -> /hadoop/yarn/local/usercache/elyra/filecache/13/__spark_conf__.zip lrwxrwxrwx. 1 yarn hadoop 62 Jul 12 19:04 __spark_libs__ -> /hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz drwx--x---. 2 yarn hadoop 6 Jul 12 19:04 tmp lrwxrwxrwx. 1 yarn hadoop 97 Jul 12 19:04 toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar -> /hadoop/yarn/local/usercache/elyra/filecache/15/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar find -L . -maxdepth 5 -ls: 1075194431 4 drwx--x--- 3 yarn hadoop 4096 Jul 12 19:04 . 3223210141 0 drwx--x--- 2 yarn hadoop 6 Jul 12 19:04 ./tmp 1075194432 4 -rw-r--r-- 1 yarn hadoop 93 Jul 12 19:04 ./container_tokens 1075194433 4 -rw-r--r-- 1 yarn hadoop 12 Jul 12 19:04 ./.container_tokens.crc 1075194434 8 -rwx------ 1 yarn hadoop 5830 Jul 12 19:04 ./launch_container.sh 1075194435 4 -rw-r--r-- 1 yarn hadoop 56 Jul 12 19:04 ./.launch_container.sh.crc 1075194436 4 -rwx------ 1 yarn hadoop 626 Jul 12 19:04 ./default_container_executor_session.sh 1075194437 4 -rw-r--r-- 1 yarn hadoop 16 Jul 12 19:04 ./.default_container_executor_session.sh.crc 1075194438 4 -rwx------ 1 yarn hadoop 680 Jul 12 19:04 ./default_container_executor.sh 1075194439 4 -rw-r--r-- 1 yarn hadoop 16 Jul 12 19:04 ./.default_container_executor.sh.crc 3223210133 26948 -r-x------ 1 yarn hadoop 27591359 Jul 12 19:04 ./toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar 1075194423 16 -r-x------ 1 yarn hadoop 14747 Jul 12 19:04 ./__app__.jar 3226980184 4 drwx------ 2 yarn hadoop 4096 Jul 12 19:04 ./__spark_conf__ 3226980185 8 -r-x------ 1 yarn hadoop 6846 Jul 12 19:04 ./__spark_conf__/mapred-site.xml 3226980186 8 -r-x------ 1 yarn hadoop 6369 Jul 12 19:04 ./__spark_conf__/hadoop-env.sh 3226980187 4 -r-x------ 1 yarn hadoop 620 Jul 12 19:04 ./__spark_conf__/log4j.properties 3226980188 4 -r-x------ 1 yarn hadoop 900 Jul 12 19:04 ./__spark_conf__/hadoop-metrics2.properties 3226980189 20 -r-x------ 1 yarn hadoop 17527 Jul 12 19:04 ./__spark_conf__/yarn-site.xml 3226980190 4 -r-x------ 1 yarn hadoop 3979 Jul 12 19:04 ./__spark_conf__/hadoop-env.cmd 3226980191 0 -r-x------ 1 yarn hadoop 0 Jul 12 19:04 ./__spark_conf__/yarn.exclude 3224697025 4 -r-x------ 1 yarn hadoop 1 Jul 12 19:04 ./__spark_conf__/dfs.exclude 3221682549 4 -r-x------ 1 yarn hadoop 2614 Jul 12 19:04 ./__spark_conf__/core-site.xml 3221682551 4 -r-x------ 1 yarn hadoop 1631 Jul 12 19:04 ./__spark_conf__/kms-log4j.properties 3221682554 4 -r-x------ 1 yarn hadoop 2250 Jul 12 19:04 ./__spark_conf__/yarn-env.cmd 3221930525 4 -r-x------ 1 yarn hadoop 884 Jul 12 19:04 ./__spark_conf__/ssl-client.xml 3221930795 4 -r-x------ 1 yarn hadoop 2035 Jul 12 19:04 ./__spark_conf__/capacity-scheduler.xml 3221965927 4 -r-x------ 1 yarn hadoop 3518 Jul 12 19:04 ./__spark_conf__/kms-acls.xml 3221991375 4 -r-x------ 1 yarn hadoop 2358 Jul 12 19:04 ./__spark_conf__/topology_script.py 3221991379 4 -r-x------ 1 yarn hadoop 758 Jul 12 19:04 ./__spark_conf__/mapred-site.xml.template 3221991382 4 -r-x------ 1 yarn hadoop 1335 Jul 12 19:04 ./__spark_conf__/configuration.xsl 3221991384 8 -r-x------ 1 yarn hadoop 5310 Jul 12 19:04 ./__spark_conf__/yarn-env.sh 3223210112 8 -r-x------ 1 yarn hadoop 6748 Jul 12 19:04 ./__spark_conf__/hdfs-site.xml 3223210113 4 -r-x------ 1 yarn hadoop 1020 Jul 12 19:04 ./__spark_conf__/commons-logging.properties 3223210114 4 -r-x------ 1 yarn hadoop 1019 Jul 12 19:04 ./__spark_conf__/container-executor.cfg 3223210115 8 -r-x------ 1 yarn hadoop 4221 Jul 12 19:04 ./__spark_conf__/task-log4j.properties 3223210116 4 -r-x------ 1 yarn hadoop 2490 Jul 12 19:04 ./__spark_conf__/hadoop-metrics.properties 3223210117 4 -r-x------ 1 yarn hadoop 856 Jul 12 19:04 ./__spark_conf__/mapred-env.sh 3223210118 4 -r-x------ 1 yarn hadoop 1602 Jul 12 19:04 ./__spark_conf__/health_check 3223210119 4 -r-x------ 1 yarn hadoop 2316 Jul 12 19:04 ./__spark_conf__/ssl-client.xml.example 3223210120 4 -r-x------ 1 yarn hadoop 1527 Jul 12 19:04 ./__spark_conf__/kms-env.sh 3223210121 4 -r-x------ 1 yarn hadoop 1308 Jul 12 19:04 ./__spark_conf__/hadoop-policy.xml 3223210122 4 -r-x------ 1 yarn hadoop 93 Jul 12 19:04 ./__spark_conf__/slaves 3223210123 4 -r-x------ 1 yarn hadoop 268 Jul 12 19:04 ./__spark_conf__/topology_mappings.data 3223210124 8 -r-x------ 1 yarn hadoop 4956 Jul 12 19:04 ./__spark_conf__/metrics.properties 3223210125 4 -r-x------ 1 yarn hadoop 951 Jul 12 19:04 ./__spark_conf__/mapred-env.cmd 3223210126 4 -r-x------ 1 yarn hadoop 1000 Jul 12 19:04 ./__spark_conf__/ssl-server.xml 3223210127 4 -r-x------ 1 yarn hadoop 2697 Jul 12 19:04 ./__spark_conf__/ssl-server.xml.example 3223210128 4 -r-x------ 1 yarn hadoop 945 Jul 12 19:04 ./__spark_conf__/taskcontroller.cfg 3223210129 8 -r-x------ 1 yarn hadoop 5511 Jul 12 19:04 ./__spark_conf__/kms-site.xml 3223210130 8 -r-x------ 1 yarn hadoop 4113 Jul 12 19:04 ./__spark_conf__/mapred-queues.xml.template 3223210131 4 -r-x------ 1 yarn hadoop 1884 Jul 12 19:04 ./__spark_conf__/__spark_conf__.properties 3702028 16 drwxr-xr-x 2 yarn hadoop 12288 Jul 11 15:24 ./__spark_libs__ 3702029 176 -r-xr-xr-x 1 yarn hadoop 178947 May 30 22:46 ./__spark_libs__/hk2-api-2.4.0-b34.jar 3702031 20 -r-xr-xr-x 1 yarn hadoop 16993 May 30 22:46 ./__spark_libs__/JavaEWAH-0.3.2.jar 3702032 332 -r-xr-xr-x 1 yarn hadoop 339666 May 30 22:46 ./__spark_libs__/datanucleus-api-jdo-3.2.6.jar 3702033 200 -r-xr-xr-x 1 yarn hadoop 201928 May 30 22:46 ./__spark_libs__/RoaringBitmap-0.5.11.jar 3702034 180 -r-xr-xr-x 1 yarn hadoop 181271 May 30 22:46 ./__spark_libs__/hk2-locator-2.4.0-b34.jar 3702035 232 -r-xr-xr-x 1 yarn hadoop 236660 May 30 22:46 ./__spark_libs__/ST4-4.0.4.jar 3702036 120 -r-xr-xr-x 1 yarn hadoop 118973 May 30 22:46 ./__spark_libs__/hk2-utils-2.4.0-b34.jar 3702037 68 -r-xr-xr-x 1 yarn hadoop 69409 May 30 22:46 ./__spark_libs__/activation-1.1.1.jar 3702038 320 -r-xr-xr-x 1 yarn hadoop 326724 May 30 22:46 ./__spark_libs__/httpcore-4.4.4.jar 3702039 436 -r-xr-xr-x 1 yarn hadoop 445288 May 30 22:46 ./__spark_libs__/antlr-2.7.7.jar 3702040 1848 -r-xr-xr-x 1 yarn hadoop 1890075 May 30 22:46 ./__spark_libs__/datanucleus-core-3.2.10.jar 3702041 164 -r-xr-xr-x 1 yarn hadoop 164368 May 30 22:46 ./__spark_libs__/antlr-runtime-3.4.jar 3702042 1768 -r-xr-xr-x 1 yarn hadoop 1809447 May 30 22:46 ./__spark_libs__/datanucleus-rdbms-3.2.9.jar 3702043 296 -r-xr-xr-x 1 yarn hadoop 302248 May 30 22:46 ./__spark_libs__/antlr4-runtime-4.5.3.jar 609856 720 -r-xr-xr-x 1 yarn hadoop 736658 May 30 22:46 ./__spark_libs__/httpclient-4.5.2.jar 609857 8 -r-xr-xr-x 1 yarn hadoop 4467 May 30 22:46 ./__spark_libs__/aopalliance-1.0.jar 609858 132 -r-xr-xr-x 1 yarn hadoop 134595 May 30 22:46 ./__spark_libs__/commons-crypto-1.0.0.jar 609859 16 -r-xr-xr-x 1 yarn hadoop 14766 May 30 22:46 ./__spark_libs__/aopalliance-repackaged-2.4.0-b34.jar 609860 160 -r-xr-xr-x 1 yarn hadoop 160519 May 30 22:46 ./__spark_libs__/commons-dbcp-1.4.jar 609861 12 -r-xr-xr-x 1 yarn hadoop 12131 May 30 22:46 ./__spark_libs__/jpam-1.1.jar 609862 440 -r-xr-xr-x 1 yarn hadoop 448794 May 30 22:46 ./__spark_libs__/apache-log4j-extras-1.2.17.jar 609863 3152 -r-xr-xr-x 1 yarn hadoop 3224708 May 30 22:46 ./__spark_libs__/derby-10.12.1.1.jar 609864 44 -r-xr-xr-x 1 yarn hadoop 44925 May 30 22:46 ./__spark_libs__/apacheds-i18n-2.0.0-M15.jar 609865 508 -r-xr-xr-x 1 yarn hadoop 516634 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-app-2.7.3.2.6.1.0-129.jar 609866 676 -r-xr-xr-x 1 yarn hadoop 691479 May 30 22:46 ./__spark_libs__/apacheds-kerberos-codec-2.0.0-M15.jar 609867 20 -r-xr-xr-x 1 yarn hadoop 18482 May 30 22:46 ./__spark_libs__/eigenbase-properties-1.1.5.jar 609868 20 -r-xr-xr-x 1 yarn hadoop 16560 May 30 22:46 ./__spark_libs__/api-asn1-api-1.0.0-M20.jar 609869 188 -r-xr-xr-x 1 yarn hadoop 190432 May 30 22:46 ./__spark_libs__/gson-2.2.4.jar 609870 80 -r-xr-xr-x 1 yarn hadoop 79912 May 30 22:46 ./__spark_libs__/api-util-1.0.0-M20.jar 609871 2140 -r-xr-xr-x 1 yarn hadoop 2189117 May 30 22:46 ./__spark_libs__/guava-14.0.1.jar 609872 1168 -r-xr-xr-x 1 yarn hadoop 1194003 May 30 22:46 ./__spark_libs__/arpack_combined_all-0.1.jar 609873 1256 -r-xr-xr-x 1 yarn hadoop 1282424 May 30 22:46 ./__spark_libs__/ivy-2.4.0.jar 609874 428 -r-xr-xr-x 1 yarn hadoop 436303 May 30 22:46 ./__spark_libs__/avro-1.7.7.jar 609875 256 -r-xr-xr-x 1 yarn hadoop 258876 May 30 22:46 ./__spark_libs__/jackson-core-2.6.5.jar 609876 192 -r-xr-xr-x 1 yarn hadoop 192993 May 30 22:46 ./__spark_libs__/avro-ipc-1.7.7.jar 609877 144 -r-xr-xr-x 1 yarn hadoop 143602 May 30 22:46 ./__spark_libs__/commons-digester-1.8.jar 609878 180 -r-xr-xr-x 1 yarn hadoop 180736 May 30 22:46 ./__spark_libs__/avro-mapred-1.7.7-hadoop2.jar 609879 696 -r-xr-xr-x 1 yarn hadoop 710492 May 30 22:46 ./__spark_libs__/guice-3.0.jar 609880 16 -r-xr-xr-x 1 yarn hadoop 15071 May 30 22:46 ./__spark_libs__/jta-1.1.jar 609881 504 -r-xr-xr-x 1 yarn hadoop 516062 May 30 22:46 ./__spark_libs__/aws-java-sdk-core-1.10.6.jar 609882 64 -r-xr-xr-x 1 yarn hadoop 65012 May 30 22:46 ./__spark_libs__/guice-servlet-3.0.jar 609883 256 -r-xr-xr-x 1 yarn hadoop 258578 May 30 22:46 ./__spark_libs__/aws-java-sdk-kms-1.10.6.jar 609884 80 -r-xr-xr-x 1 yarn hadoop 79175 May 30 22:46 ./__spark_libs__/hadoop-auth-2.7.3.2.6.1.0-129.jar 609885 560 -r-xr-xr-x 1 yarn hadoop 570101 May 30 22:46 ./__spark_libs__/aws-java-sdk-s3-1.10.6.jar 609886 1988 -r-xr-xr-x 1 yarn hadoop 2035066 May 30 22:46 ./__spark_libs__/commons-math3-3.4.1.jar 609887 80 -r-xr-xr-x 1 yarn hadoop 78849 May 30 22:46 ./__spark_libs__/azure-data-lake-store-sdk-2.1.4.jar 609888 208 -r-xr-xr-x 1 yarn hadoop 212453 May 30 22:46 ./__spark_libs__/commons-net-2.2.jar 609889 232 -r-xr-xr-x 1 yarn hadoop 236880 May 30 22:46 ./__spark_libs__/lz4-1.3.0.jar 609890 12 -r-xr-xr-x 1 yarn hadoop 10092 May 30 22:46 ./__spark_libs__/azure-keyvault-core-0.8.0.jar 609891 220 -r-xr-xr-x 1 yarn hadoop 222041 May 30 22:46 ./__spark_libs__/hadoop-aws-2.7.3.2.6.1.0-129.jar 609892 728 -r-xr-xr-x 1 yarn hadoop 745325 May 30 22:46 ./__spark_libs__/azure-storage-4.2.0.jar 609893 48 -r-xr-xr-x 1 yarn hadoop 46983 May 30 22:46 ./__spark_libs__/jackson-annotations-2.6.5.jar 609894 20 -r-xr-xr-x 1 yarn hadoop 17008 May 30 22:46 ./__spark_libs__/base64-2.3.8.jar 609895 212 -r-xr-xr-x 1 yarn hadoop 213891 May 30 22:46 ./__spark_libs__/hadoop-azure-2.7.3.2.6.1.0-129.jar 609896 2780 -r-xr-xr-x 1 yarn hadoop 2842667 May 30 22:46 ./__spark_libs__/bcprov-jdk15on-1.51.jar 609897 4 -r-xr-xr-x 1 yarn hadoop 2567 May 30 22:46 ./__spark_libs__/hadoop-client-2.7.3.2.6.1.0-129.jar 609898 112 -r-xr-xr-x 1 yarn hadoop 110600 May 30 22:46 ./__spark_libs__/bonecp-0.8.0.RELEASE.jar 609899 3564 -r-xr-xr-x 1 yarn hadoop 3647240 May 30 22:46 ./__spark_libs__/hadoop-common-2.7.3.2.6.1.0-129.jar 609900 180 -r-xr-xr-x 1 yarn hadoop 181926 May 30 22:46 ./__spark_libs__/breeze-macros_2.11-0.12.jar 609901 228 -r-xr-xr-x 1 yarn hadoop 232248 May 30 22:46 ./__spark_libs__/jackson-core-asl-1.9.13.jar 609902 14080 -r-xr-xr-x 1 yarn hadoop 14413834 May 30 22:46 ./__spark_libs__/breeze_2.11-0.12.jar 609903 96 -r-xr-xr-x 1 yarn hadoop 96221 May 30 22:46 ./__spark_libs__/commons-pool-1.5.4.jar 609904 256 -r-xr-xr-x 1 yarn hadoop 258370 May 30 22:46 ./__spark_libs__/calcite-avatica-1.2.0-incubating.jar 609905 80 -r-xr-xr-x 1 yarn hadoop 79845 May 30 22:46 ./__spark_libs__/compress-lzf-1.0.3.jar 609906 3440 -r-xr-xr-x 1 yarn hadoop 3519262 May 30 22:46 ./__spark_libs__/calcite-core-1.2.0-incubating.jar 609907 164 -r-xr-xr-x 1 yarn hadoop 164422 May 30 22:46 ./__spark_libs__/core-1.1.2.jar 609908 64 -r-xr-xr-x 1 yarn hadoop 65261 May 30 22:46 ./__spark_libs__/oro-2.0.8.jar 609909 436 -r-xr-xr-x 1 yarn hadoop 442406 May 30 22:46 ./__spark_libs__/calcite-linq4j-1.2.0-incubating.jar 609910 1144 -r-xr-xr-x 1 yarn hadoop 1171380 May 30 22:46 ./__spark_libs__/jackson-databind-2.6.5.jar 609911 52 -r-xr-xr-x 1 yarn hadoop 50619 May 30 22:46 ./__spark_libs__/chill-java-0.8.0.jar 609912 20 -r-xr-xr-x 1 yarn hadoop 18336 May 30 22:46 ./__spark_libs__/jackson-jaxrs-1.9.13.jar 609913 220 -r-xr-xr-x 1 yarn hadoop 223573 May 30 22:46 ./__spark_libs__/chill_2.11-0.8.0.jar 609914 8508 -r-xr-xr-x 1 yarn hadoop 8711541 May 30 22:46 ./__spark_libs__/hadoop-hdfs-2.7.3.2.6.1.0-129.jar 609915 188 -r-xr-xr-x 1 yarn hadoop 188671 May 30 22:46 ./__spark_libs__/commons-beanutils-1.7.0.jar 609916 68 -r-xr-xr-x 1 yarn hadoop 68866 May 30 22:46 ./__spark_libs__/curator-client-2.6.0.jar 609917 204 -r-xr-xr-x 1 yarn hadoop 206035 May 30 22:46 ./__spark_libs__/commons-beanutils-core-1.8.0.jar 609918 764 -r-xr-xr-x 1 yarn hadoop 780664 May 30 22:46 ./__spark_libs__/jackson-mapper-asl-1.9.13.jar 609919 44 -r-xr-xr-x 1 yarn hadoop 41123 May 30 22:46 ./__spark_libs__/commons-cli-1.2.jar 609920 2068 -r-xr-xr-x 1 yarn hadoop 2117555 May 30 22:46 ./__spark_libs__/hadoop-yarn-api-2.7.3.2.6.1.0-129.jar 609921 280 -r-xr-xr-x 1 yarn hadoop 284184 May 30 22:46 ./__spark_libs__/commons-codec-1.10.jar 609922 184 -r-xr-xr-x 1 yarn hadoop 185245 May 30 22:46 ./__spark_libs__/curator-framework-2.6.0.jar 609923 576 -r-xr-xr-x 1 yarn hadoop 588337 May 30 22:46 ./__spark_libs__/commons-collections-3.2.2.jar 609924 136 -r-xr-xr-x 1 yarn hadoop 138453 May 30 22:46 ./__spark_libs__/hive-beeline-1.2.1.spark2.hdp.1.jar 609925 36 -r-xr-xr-x 1 yarn hadoop 36627 May 30 22:46 ./__spark_libs__/commons-compiler-3.0.0.jar 609926 40 -r-xr-xr-x 1 yarn hadoop 40864 May 30 22:46 ./__spark_libs__/hive-cli-1.2.1.spark2.hdp.1.jar 609927 236 -r-xr-xr-x 1 yarn hadoop 241367 May 30 22:46 ./__spark_libs__/commons-compress-1.4.1.jar 609928 244 -r-xr-xr-x 1 yarn hadoop 248171 May 30 22:46 ./__spark_libs__/curator-recipes-2.6.0.jar 609929 292 -r-xr-xr-x 1 yarn hadoop 298829 May 30 22:46 ./__spark_libs__/commons-configuration-1.6.jar 609930 300 -r-xr-xr-x 1 yarn hadoop 305001 May 30 22:46 ./__spark_libs__/commons-httpclient-3.1.jar 609931 28 -r-xr-xr-x 1 yarn hadoop 27084 May 30 22:46 ./__spark_libs__/jackson-xc-1.9.13.jar 609932 184 -r-xr-xr-x 1 yarn hadoop 185140 May 30 22:46 ./__spark_libs__/commons-io-2.4.jar 609933 12 -r-xr-xr-x 1 yarn hadoop 10023 May 30 22:46 ./__spark_libs__/java-xmlbuilder-1.0.jar 609934 280 -r-xr-xr-x 1 yarn hadoop 284220 May 30 22:46 ./__spark_libs__/commons-lang-2.6.jar 609935 11232 -r-xr-xr-x 1 yarn hadoop 11499183 May 30 22:46 ./__spark_libs__/hive-exec-1.2.1.spark2.hdp.1.jar 609936 472 -r-xr-xr-x 1 yarn hadoop 479881 May 30 22:46 ./__spark_libs__/commons-lang3-3.5.jar 609937 100 -r-xr-xr-x 1 yarn hadoop 101402 May 30 22:46 ./__spark_libs__/hive-jdbc-1.2.1.spark2.hdp.1.jar 609938 64 -r-xr-xr-x 1 yarn hadoop 62050 May 30 22:46 ./__spark_libs__/commons-logging-1.1.3.jar 609939 720 -r-xr-xr-x 1 yarn hadoop 735574 May 30 22:46 ./__spark_libs__/janino-3.0.0.jar 609940 20 -r-xr-xr-x 1 yarn hadoop 17369 May 30 22:46 ./__spark_libs__/hadoop-annotations-2.7.3.2.6.1.0-129.jar 609941 24 -r-xr-xr-x 1 yarn hadoop 23519 May 30 22:46 ./__spark_libs__/hadoop-azure-datalake-2.7.3.2.6.1.0-129.jar 609942 52 -r-xr-xr-x 1 yarn hadoop 51484 May 30 22:46 ./__spark_libs__/json-smart-1.1.1.jar 609943 280 -r-xr-xr-x 1 yarn hadoop 285479 May 30 22:46 ./__spark_libs__/parquet-encoding-1.8.1.jar 609944 736 -r-xr-xr-x 1 yarn hadoop 752754 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-common-2.7.3.2.6.1.0-129.jar 609961 96 -r-xr-xr-x 1 yarn hadoop 94672 May 30 22:46 ./__spark_libs__/xz-1.0.jar 609945 84 -r-xr-xr-x 1 yarn hadoop 82421 May 30 22:46 ./__spark_libs__/json4s-ast_2.11-3.2.11.jar 609946 788 -r-xr-xr-x 1 yarn hadoop 802818 May 30 22:46 ./__spark_libs__/scalap-2.11.8.jar 609947 1500 -r-xr-xr-x 1 yarn hadoop 1535771 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-core-2.7.3.2.6.1.0-129.jar 609948 96 -r-xr-xr-x 1 yarn hadoop 94796 May 30 22:46 ./__spark_libs__/pyrolite-4.13.jar 609949 116 -r-xr-xr-x 1 yarn hadoop 115472 May 30 22:46 ./__spark_libs__/py4j-0.10.4.jar 609950 40 -r-xr-xr-x 1 yarn hadoop 39398 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-jobclient-2.7.3.2.6.1.0-129.jar 609951 420 -r-xr-xr-x 1 yarn hadoop 427780 May 30 22:46 ./__spark_libs__/jodd-core-3.5.2.jar 609952 40 -r-xr-xr-x 1 yarn hadoop 39280 May 30 22:46 ./__spark_libs__/metrics-jvm-3.1.2.jar 609953 56 -r-xr-xr-x 1 yarn hadoop 54074 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-shuffle-2.7.3.2.6.1.0-129.jar 609954 36 -r-xr-xr-x 1 yarn hadoop 33015 May 30 22:46 ./__spark_libs__/jsr305-1.3.9.jar 609955 40 -r-xr-xr-x 1 yarn hadoop 40509 May 30 22:46 ./__spark_libs__/slf4j-api-1.7.16.jar 609956 116 -r-xr-xr-x 1 yarn hadoop 114748 May 30 22:46 ./__spark_libs__/hadoop-openstack-2.7.3.2.6.1.0-129.jar 609957 748 -r-xr-xr-x 1 yarn hadoop 764569 May 30 22:46 ./__spark_libs__/jtransforms-2.4.0.jar 609958 48 -r-xr-xr-x 1 yarn hadoop 48720 May 30 22:46 ./__spark_libs__/snappy-0.2.jar 609959 192 -r-xr-xr-x 1 yarn hadoop 193240 May 30 22:46 ./__spark_libs__/hadoop-yarn-client-2.7.3.2.6.1.0-129.jar 609960 8 -r-xr-xr-x 1 yarn hadoop 4596 May 30 22:46 ./__spark_libs__/jul-to-slf4j-1.7.16.jar 609962 1712 -r-xr-xr-x 1 yarn hadoop 1750536 May 30 22:46 ./__spark_libs__/hadoop-yarn-common-2.7.3.2.6.1.0-129.jar 609963 352 -r-xr-xr-x 1 yarn hadoop 358390 May 30 22:46 ./__spark_libs__/kryo-shaded-3.0.3.jar 609964 7108 -r-xr-xr-x 1 yarn hadoop 7276083 May 30 22:46 ./__spark_libs__/spire_2.11-0.7.4.jar 609965 100 -r-xr-xr-x 1 yarn hadoop 101126 May 30 22:46 ./__spark_libs__/hadoop-yarn-registry-2.7.3.2.6.1.0-129.jar 609966 576 -r-xr-xr-x 1 yarn hadoop 589462 May 30 22:46 ./__spark_libs__/json4s-core_2.11-3.2.11.jar 609967 24 -r-xr-xr-x 1 yarn hadoop 23346 May 30 22:46 ./__spark_libs__/stax-api-1.0-2.jar 609968 412 -r-xr-xr-x 1 yarn hadoop 420788 May 30 22:46 ./__spark_libs__/hadoop-yarn-server-common-2.7.3.2.6.1.0-129.jar 609969 40 -r-xr-xr-x 1 yarn hadoop 40341 May 30 22:46 ./__spark_libs__/json4s-jackson_2.11-3.2.11.jar 609970 36 -r-xr-xr-x 1 yarn hadoop 36249 May 30 22:46 ./__spark_libs__/hadoop-yarn-server-web-proxy-2.7.3.2.6.1.0-129.jar 609971 1024 -r-xr-xr-x 1 yarn hadoop 1045744 May 30 22:46 ./__spark_libs__/leveldbjni-all-1.8.jar 609972 5380 -r-xr-xr-x 1 yarn hadoop 5505070 May 30 22:46 ./__spark_libs__/hive-metastore-1.2.1.spark2.hdp.1.jar 609973 232 -r-xr-xr-x 1 yarn hadoop 234201 May 30 22:46 ./__spark_libs__/libthrift-0.9.3.jar 609974 1444 -r-xr-xr-x 1 yarn hadoop 1475955 May 30 22:46 ./__spark_libs__/htrace-core-3.1.0-incubating.jar 609975 480 -r-xr-xr-x 1 yarn hadoop 489884 May 30 22:46 ./__spark_libs__/log4j-1.2.17.jar 609976 48 -r-xr-xr-x 1 yarn hadoop 48418 May 30 22:46 ./__spark_libs__/jackson-dataformat-cbor-2.6.5.jar 609977 512 -r-xr-xr-x 1 yarn hadoop 521157 May 30 22:46 ./__spark_libs__/mail-1.4.7.jar 609978 44 -r-xr-xr-x 1 yarn hadoop 41263 May 30 22:46 ./__spark_libs__/jackson-module-paranamer-2.6.5.jar 609979 112 -r-xr-xr-x 1 yarn hadoop 112558 May 30 22:46 ./__spark_libs__/metrics-core-3.1.2.jar 609980 504 -r-xr-xr-x 1 yarn hadoop 515604 May 30 22:46 ./__spark_libs__/jackson-module-scala_2.11-2.6.5.jar 609981 8 -r-xr-xr-x 1 yarn hadoop 5711 May 30 22:46 ./__spark_libs__/minlog-1.3.0.jar 609982 700 -r-xr-xr-x 1 yarn hadoop 714194 May 30 22:46 ./__spark_libs__/javassist-3.18.1-GA.jar 609983 400 -r-xr-xr-x 1 yarn hadoop 409467 May 30 22:46 ./__spark_libs__/mx4j-3.0.2.jar 773376 28 -r-xr-xr-x 1 yarn hadoop 26366 May 30 22:46 ./__spark_libs__/javax.annotation-api-1.2.jar 773377 216 -r-xr-xr-x 1 yarn hadoop 218076 May 30 22:46 ./__spark_libs__/parquet-hadoop-1.8.1.jar 773378 4 -r-xr-xr-x 1 yarn hadoop 2497 May 30 22:46 ./__spark_libs__/javax.inject-1.jar 773379 1204 -r-xr-xr-x 1 yarn hadoop 1230201 May 30 22:46 ./__spark_libs__/netty-3.8.0.Final.jar 773380 8 -r-xr-xr-x 1 yarn hadoop 5950 May 30 22:46 ./__spark_libs__/javax.inject-2.4.0-b34.jar 773381 2264 -r-xr-xr-x 1 yarn hadoop 2314839 May 30 22:46 ./__spark_libs__/netty-all-4.0.42.Final.jar 773382 96 -r-xr-xr-x 1 yarn hadoop 95806 May 30 22:46 ./__spark_libs__/javax.servlet-api-3.1.0.jar 773383 176 -r-xr-xr-x 1 yarn hadoop 178648 May 30 22:46 ./__spark_libs__/nimbus-jose-jwt-3.9.jar 773384 116 -r-xr-xr-x 1 yarn hadoop 115534 May 30 22:46 ./__spark_libs__/javax.ws.rs-api-2.0.1.jar 773385 1024 -r-xr-xr-x 1 yarn hadoop 1048117 May 30 22:46 ./__spark_libs__/parquet-jackson-1.8.1.jar 773386 388 -r-xr-xr-x 1 yarn hadoop 395195 May 30 22:46 ./__spark_libs__/javolution-5.5.1.jar 773387 640 -r-xr-xr-x 1 yarn hadoop 654216 May 30 22:46 ./__spark_libs__/pmml-model-1.2.15.jar 773388 104 -r-xr-xr-x 1 yarn hadoop 105134 May 30 22:46 ./__spark_libs__/jaxb-api-2.2.2.jar 773389 44 -r-xr-xr-x 1 yarn hadoop 41755 May 30 22:46 ./__spark_libs__/objenesis-2.1.jar 773390 4 -r-xr-xr-x 1 yarn hadoop 2254 May 30 22:46 ./__spark_libs__/jcip-annotations-1.0.jar 773391 312 -r-xr-xr-x 1 yarn hadoop 319099 May 30 22:46 ./__spark_libs__/okhttp-2.4.0.jar 773392 20 -r-xr-xr-x 1 yarn hadoop 16430 May 30 22:46 ./__spark_libs__/jcl-over-slf4j-1.7.16.jar 773393 8 -r-xr-xr-x 1 yarn hadoop 5310 May 30 22:46 ./__spark_libs__/pmml-schema-1.2.15.jar 773394 200 -r-xr-xr-x 1 yarn hadoop 201124 May 30 22:46 ./__spark_libs__/jdo-api-3.0.1.jar 773395 64 -r-xr-xr-x 1 yarn hadoop 64661 May 30 22:46 ./__spark_libs__/okio-1.4.0.jar 773396 164 -r-xr-xr-x 1 yarn hadoop 167421 May 30 22:46 ./__spark_libs__/jersey-client-2.22.2.jar 773397 20 -r-xr-xr-x 1 yarn hadoop 19827 May 30 22:46 ./__spark_libs__/opencsv-2.3.jar 773398 684 -r-xr-xr-x 1 yarn hadoop 698375 May 30 22:46 ./__spark_libs__/jersey-common-2.22.2.jar 773399 24 -r-xr-xr-x 1 yarn hadoop 20852 May 30 22:46 ./__spark_libs__/metrics-graphite-3.1.2.jar 773400 20 -r-xr-xr-x 1 yarn hadoop 18098 May 30 22:46 ./__spark_libs__/jersey-container-servlet-2.22.2.jar 773401 308 -r-xr-xr-x 1 yarn hadoop 313702 May 30 22:46 ./__spark_libs__/libfb303-0.9.3.jar 773402 28 -r-xr-xr-x 1 yarn hadoop 26514 May 30 22:46 ./__spark_libs__/stax-api-1.0.1.jar 773403 68 -r-xr-xr-x 1 yarn hadoop 66270 May 30 22:46 ./__spark_libs__/jersey-container-servlet-core-2.22.2.jar 773404 20 -r-xr-xr-x 1 yarn hadoop 20235 May 30 22:46 ./__spark_libs__/osgi-resource-locator-1.0.1.jar 773405 952 -r-xr-xr-x 1 yarn hadoop 971310 May 30 22:46 ./__spark_libs__/jersey-guava-2.22.2.jar 773406 32 -r-xr-xr-x 1 yarn hadoop 29555 May 30 22:46 ./__spark_libs__/paranamer-2.3.jar 773407 72 -r-xr-xr-x 1 yarn hadoop 72733 May 30 22:46 ./__spark_libs__/jersey-media-jaxb-2.22.2.jar 773408 924 -r-xr-xr-x 1 yarn hadoop 945914 May 30 22:46 ./__spark_libs__/parquet-column-1.8.1.jar 773409 932 -r-xr-xr-x 1 yarn hadoop 951701 May 30 22:46 ./__spark_libs__/jersey-server-2.22.2.jar 773410 524 -r-xr-xr-x 1 yarn hadoop 533455 May 30 22:46 ./__spark_libs__/protobuf-java-2.5.0.jar 773411 1996 -r-xr-xr-x 1 yarn hadoop 2041628 May 30 22:46 ./__spark_libs__/jets3t-0.9.3.jar 773412 5612 -r-xr-xr-x 1 yarn hadoop 5744974 May 30 22:46 ./__spark_libs__/scala-library-2.11.8.jar 773413 528 -r-xr-xr-x 1 yarn hadoop 540298 May 30 22:46 ./__spark_libs__/jetty-6.1.26.hwx.jar 773414 16 -r-xr-xr-x 1 yarn hadoop 15827 May 30 22:46 ./__spark_libs__/metrics-json-3.1.2.jar 773415 20 -r-xr-xr-x 1 yarn hadoop 18899 May 30 22:46 ./__spark_libs__/jetty-sslengine-6.1.26.hwx.jar 773416 40 -r-xr-xr-x 1 yarn hadoop 38604 May 30 22:46 ./__spark_libs__/parquet-common-1.8.1.jar 773417 176 -r-xr-xr-x 1 yarn hadoop 177214 May 30 22:46 ./__spark_libs__/jetty-util-6.1.26.hwx.jar 773418 15128 -r-xr-xr-x 1 yarn hadoop 15487351 May 30 22:46 ./__spark_libs__/scala-compiler-2.11.8.jar 773419 212 -r-xr-xr-x 1 yarn hadoop 213911 May 30 22:46 ./__spark_libs__/jline-2.12.1.jar 773420 4468 -r-xr-xr-x 1 yarn hadoop 4573750 May 30 22:46 ./__spark_libs__/scala-reflect-2.11.8.jar 773421 616 -r-xr-xr-x 1 yarn hadoop 627814 May 30 22:46 ./__spark_libs__/joda-time-2.9.3.jar 773422 100 -r-xr-xr-x 1 yarn hadoop 100636 May 30 22:46 ./__spark_libs__/jsp-api-2.1.jar 773423 636 -r-xr-xr-x 1 yarn hadoop 648678 May 30 22:46 ./__spark_libs__/scala-xml_2.11-1.0.2.jar 773424 380 -r-xr-xr-x 1 yarn hadoop 387188 May 30 22:46 ./__spark_libs__/parquet-format-2.3.0-incubating.jar 773425 2732 -r-xr-xr-x 1 yarn hadoop 2796935 May 30 22:46 ./__spark_libs__/parquet-hadoop-bundle-1.6.0.jar 773426 416 -r-xr-xr-x 1 yarn hadoop 423753 May 30 22:46 ./__spark_libs__/scala-parser-combinators_2.11-1.0.4.jar 773427 2004 -r-xr-xr-x 1 yarn hadoop 2049451 May 30 22:46 ./__spark_libs__/shapeless_2.11-2.0.0.jar 773428 12 -r-xr-xr-x 1 yarn hadoop 9939 May 30 22:46 ./__spark_libs__/slf4j-log4j12-1.7.16.jar 773429 1032 -r-xr-xr-x 1 yarn hadoop 1056168 May 30 22:46 ./__spark_libs__/snappy-java-1.1.2.6.jar 773430 7336 -r-xr-xr-x 1 yarn hadoop 7508336 May 30 22:46 ./__spark_libs__/spark-catalyst_2.11-2.1.1.2.6.1.0-129.jar 773431 84 -r-xr-xr-x 1 yarn hadoop 82916 May 30 22:46 ./__spark_libs__/spark-cloud_2.11-2.1.1.2.6.1.0-129.jar 773432 11668 -r-xr-xr-x 1 yarn hadoop 11946206 May 30 22:46 ./__spark_libs__/spark-core_2.11-2.1.1.2.6.1.0-129.jar 773433 684 -r-xr-xr-x 1 yarn hadoop 699290 May 30 22:46 ./__spark_libs__/spark-graphx_2.11-2.1.1.2.6.1.0-129.jar 773434 1776 -r-xr-xr-x 1 yarn hadoop 1815017 May 30 22:46 ./__spark_libs__/spark-hive-thriftserver_2.11-2.1.1.2.6.1.0-129.jar 773435 1208 -r-xr-xr-x 1 yarn hadoop 1234838 May 30 22:46 ./__spark_libs__/spark-hive_2.11-2.1.1.2.6.1.0-129.jar 874216 88 -r-xr-xr-x 1 yarn hadoop 86811 May 30 22:46 ./__spark_libs__/spire-macros_2.11-0.7.4.jar 773436 72 -r-xr-xr-x 1 yarn hadoop 70595 May 30 22:46 ./__spark_libs__/spark-launcher_2.11-2.1.1.2.6.1.0-129.jar 773437 176 -r-xr-xr-x 1 yarn hadoop 178967 May 30 22:46 ./__spark_libs__/spark-mllib-local_2.11-2.1.1.2.6.1.0-129.jar 773438 6048 -r-xr-xr-x 1 yarn hadoop 6190594 May 30 22:46 ./__spark_libs__/spark-mllib_2.11-2.1.1.2.6.1.0-129.jar 773439 2304 -r-xr-xr-x 1 yarn hadoop 2356803 May 30 22:46 ./__spark_libs__/spark-network-common_2.11-2.1.1.2.6.1.0-129.jar 874208 64 -r-xr-xr-x 1 yarn hadoop 61843 May 30 22:46 ./__spark_libs__/spark-network-shuffle_2.11-2.1.1.2.6.1.0-129.jar 874209 68 -r-xr-xr-x 1 yarn hadoop 65842 May 30 22:46 ./__spark_libs__/spark-repl_2.11-2.1.1.2.6.1.0-129.jar 874210 32 -r-xr-xr-x 1 yarn hadoop 29765 May 30 22:46 ./__spark_libs__/spark-sketch_2.11-2.1.1.2.6.1.0-129.jar 874211 6504 -r-xr-xr-x 1 yarn hadoop 6657160 May 30 22:46 ./__spark_libs__/spark-sql_2.11-2.1.1.2.6.1.0-129.jar 874212 2052 -r-xr-xr-x 1 yarn hadoop 2101086 May 30 22:46 ./__spark_libs__/spark-streaming_2.11-2.1.1.2.6.1.0-129.jar 874213 16 -r-xr-xr-x 1 yarn hadoop 15527 May 30 22:46 ./__spark_libs__/spark-tags_2.11-2.1.1.2.6.1.0-129.jar 874214 48 -r-xr-xr-x 1 yarn hadoop 45278 May 30 22:46 ./__spark_libs__/spark-unsafe_2.11-2.1.1.2.6.1.0-129.jar 874215 728 -r-xr-xr-x 1 yarn hadoop 742074 May 30 22:46 ./__spark_libs__/spark-yarn_2.11-2.1.1.2.6.1.0-129.jar 874217 172 -r-xr-xr-x 1 yarn hadoop 174351 May 30 22:46 ./__spark_libs__/stream-2.7.0.jar 874218 148 -r-xr-xr-x 1 yarn hadoop 148627 May 30 22:46 ./__spark_libs__/stringtemplate-3.2.1.jar 874219 92 -r-xr-xr-x 1 yarn hadoop 93210 May 30 22:46 ./__spark_libs__/super-csv-2.2.0.jar 874220 320 -r-xr-xr-x 1 yarn hadoop 325553 May 30 22:46 ./__spark_libs__/univocity-parsers-2.2.1.jar 874221 64 -r-xr-xr-x 1 yarn hadoop 63777 May 30 22:46 ./__spark_libs__/validation-api-1.1.0.Final.jar 874222 144 -r-xr-xr-x 1 yarn hadoop 144660 May 30 22:46 ./__spark_libs__/xbean-asm5-shaded-4.4.jar 874223 1204 -r-xr-xr-x 1 yarn hadoop 1229125 May 30 22:46 ./__spark_libs__/xercesImpl-2.9.1.jar 874224 16 -r-xr-xr-x 1 yarn hadoop 15010 May 30 22:46 ./__spark_libs__/xmlenc-0.52.jar 874225 780 -r-xr-xr-x 1 yarn hadoop 797314 May 30 22:46 ./__spark_libs__/zookeeper-3.4.6.2.6.1.0-129.jar broken symlinks(find -L . -maxdepth 5 -type l -ls): End of LogType:directory.info LogType:launch_container.sh Log Upload Time:Wed Jul 12 19:07:15 -0500 2017 LogLength:5830 Log Contents: #!/bin/bash export SPARK_YARN_STAGING_DIR="hdfs://spark-02.softlayer.com:8020/user/elyra/.sparkStaging/application_1499804611229_0007" export PATH="/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent" export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/current/hadoop-client/conf"} export JAVA_HOME=${JAVA_HOME:-"/usr/jdk64/jdk1.8.0_112"} export LANG="en_US.UTF-8" export SPARK_LOG_URL_STDOUT="http://spark-04.softlayer.com:8042/node/containerlogs/container_1499804611229_0007_01_000003/elyra/stdout?start=-4096" export NM_HOST="spark-04.softlayer.com" export LOGNAME="elyra" export JVM_PID="$$" export PWD="/hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007/container_1499804611229_0007_01_000003" export LOCAL_DIRS="/hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007" export NM_HTTP_PORT="8042" export LOG_DIRS="/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003" export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= " export NM_PORT="45454" export USER="elyra" export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/current/hadoop-yarn-nodemanager"} export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/etc/hadoop/conf:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.1.0-129/hadoop/lib/hadoop-lzo-0.6.0.2.6.1.0-129.jar:/etc/hadoop/conf/secure" export SPARK_YARN_MODE="true" export HADOOP_TOKEN_FILE_LOCATION="/hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007/container_1499804611229_0007_01_000003/container_tokens" export NM_AUX_SERVICE_spark_shuffle="" export SPARK_USER="elyra" export LOCAL_USER_DIRS="/hadoop/yarn/local/usercache/elyra/" export HADOOP_HOME="/usr/hdp/2.6.1.0-129/hadoop" export SPARK_LOG_URL_STDERR="http://spark-04.softlayer.com:8042/node/containerlogs/container_1499804611229_0007_01_000003/elyra/stderr?start=-4096" export HOME="/home/" export NM_AUX_SERVICE_spark2_shuffle="" export CONTAINER_ID="container_1499804611229_0007_01_000003" export MALLOC_ARENA_MAX="4" ln -sf "/hadoop/yarn/local/usercache/elyra/filecache/15/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar" "toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/hadoop/yarn/local/usercache/elyra/filecache/14/toree-launcher_2.11-1.jar" "__app__.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/hadoop/yarn/local/usercache/elyra/filecache/13/__spark_conf__.zip" "__spark_conf__" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz" "__spark_libs__" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi # Creating copy of launch script cp "launch_container.sh" "/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003/launch_container.sh" chmod 640 "/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003/launch_container.sh" # Determining directory contents echo "ls -l:" 1>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003/directory.info" ls -l 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003/directory.info" find -L . -maxdepth 5 -ls 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003/directory.info" exec /bin/bash -c "LD_LIBRARY_PATH="/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp '-Dspark.history.ui.port=18081' '-Dspark.ui.port=0' '-Dspark.driver.port=44474' -Dspark.yarn.app.container.log.dir=/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@169.45.103.145:44474 --executor-id 2 --hostname spark-04.softlayer.com --cores 1 --app-id application_1499804611229_0007 --user-class-path file:$PWD/__app__.jar --user-class-path file:$PWD/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar 1>/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003/stdout 2>/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000003/stderr" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi End of LogType:launch_container.sh LogType:stderr Log Upload Time:Wed Jul 12 19:07:15 -0500 2017 LogLength:3795 Log Contents: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.1.0-129/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 17/07/12 19:04:49 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 15422@spark-04.softlayer.com 17/07/12 19:04:49 INFO SignalUtils: Registered signal handler for TERM 17/07/12 19:04:49 INFO SignalUtils: Registered signal handler for HUP 17/07/12 19:04:49 INFO SignalUtils: Registered signal handler for INT 17/07/12 19:04:49 INFO SecurityManager: Changing view acls to: yarn,elyra 17/07/12 19:04:49 INFO SecurityManager: Changing modify acls to: yarn,elyra 17/07/12 19:04:49 INFO SecurityManager: Changing view acls groups to: 17/07/12 19:04:49 INFO SecurityManager: Changing modify acls groups to: 17/07/12 19:04:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, elyra); groups with view permissions: Set(); users with modify permissions: Set(yarn, elyra); groups with modify permissions: Set() 17/07/12 19:04:50 INFO TransportClientFactory: Successfully created connection to /169.45.103.145:44474 after 55 ms (0 ms spent in bootstraps) 17/07/12 19:04:50 INFO SecurityManager: Changing view acls to: yarn,elyra 17/07/12 19:04:50 INFO SecurityManager: Changing modify acls to: yarn,elyra 17/07/12 19:04:50 INFO SecurityManager: Changing view acls groups to: 17/07/12 19:04:50 INFO SecurityManager: Changing modify acls groups to: 17/07/12 19:04:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, elyra); groups with view permissions: Set(); users with modify permissions: Set(yarn, elyra); groups with modify permissions: Set() 17/07/12 19:04:50 INFO TransportClientFactory: Successfully created connection to /169.45.103.145:44474 after 1 ms (0 ms spent in bootstraps) 17/07/12 19:04:50 INFO DiskBlockManager: Created local directory at /hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007/blockmgr-40782af2-dd62-4841-9c58-901d438a6234 17/07/12 19:04:50 INFO MemoryStore: MemoryStore started with capacity 366.3 MB 17/07/12 19:04:50 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@169.45.103.145:44474 17/07/12 19:04:50 INFO CoarseGrainedExecutorBackend: Successfully registered with driver 17/07/12 19:04:50 INFO Executor: Starting executor ID 2 on host spark-04.softlayer.com 17/07/12 19:04:50 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40705. 17/07/12 19:04:50 INFO NettyBlockTransferService: Server created on spark-04.softlayer.com:40705 17/07/12 19:04:50 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 17/07/12 19:04:50 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(2, spark-04.softlayer.com, 40705, None) 17/07/12 19:04:50 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(2, spark-04.softlayer.com, 40705, None) 17/07/12 19:04:50 INFO BlockManager: Initialized BlockManager: BlockManagerId(2, spark-04.softlayer.com, 40705, None) 17/07/12 19:07:13 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown 17/07/12 19:07:13 INFO MemoryStore: MemoryStore cleared 17/07/12 19:07:13 INFO BlockManager: BlockManager stopped 17/07/12 19:07:13 INFO ShutdownHookManager: Shutdown hook called End of LogType:stderr LogType:stdout Log Upload Time:Wed Jul 12 19:07:15 -0500 2017 LogLength:0 Log Contents: End of LogType:stdout Container: container_1499804611229_0007_01_000001 on spark-04.softlayer.com_45454_1499904435190 LogAggregationType: AGGREGATED =============================================================================================== LogType:directory.info Log Upload Time:Wed Jul 12 19:07:15 -0500 2017 LogLength:31311 Log Contents: ls -l: total 20 lrwxrwxrwx. 1 yarn hadoop 73 Jul 12 19:04 __app__.jar -> /hadoop/yarn/local/usercache/elyra/filecache/14/toree-launcher_2.11-1.jar -rw-r--r--. 1 yarn hadoop 74 Jul 12 19:04 container_tokens -rwx------. 1 yarn hadoop 626 Jul 12 19:04 default_container_executor_session.sh -rwx------. 1 yarn hadoop 680 Jul 12 19:04 default_container_executor.sh -rwx------. 1 yarn hadoop 5671 Jul 12 19:04 launch_container.sh lrwxrwxrwx. 1 yarn hadoop 66 Jul 12 19:04 __spark_conf__ -> /hadoop/yarn/local/usercache/elyra/filecache/13/__spark_conf__.zip lrwxrwxrwx. 1 yarn hadoop 62 Jul 12 19:04 __spark_libs__ -> /hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz drwx--x---. 2 yarn hadoop 6 Jul 12 19:04 tmp lrwxrwxrwx. 1 yarn hadoop 97 Jul 12 19:04 toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar -> /hadoop/yarn/local/usercache/elyra/filecache/15/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar find -L . -maxdepth 5 -ls: 2150991806 4 drwx--x--- 3 yarn hadoop 4096 Jul 12 19:04 . 250242 0 drwx--x--- 2 yarn hadoop 6 Jul 12 19:04 ./tmp 2150991807 4 -rw-r--r-- 1 yarn hadoop 74 Jul 12 19:04 ./container_tokens 2150991808 4 -rw-r--r-- 1 yarn hadoop 12 Jul 12 19:04 ./.container_tokens.crc 2150991809 8 -rwx------ 1 yarn hadoop 5671 Jul 12 19:04 ./launch_container.sh 2150991810 4 -rw-r--r-- 1 yarn hadoop 56 Jul 12 19:04 ./.launch_container.sh.crc 2150991811 4 -rwx------ 1 yarn hadoop 626 Jul 12 19:04 ./default_container_executor_session.sh 2150991812 4 -rw-r--r-- 1 yarn hadoop 16 Jul 12 19:04 ./.default_container_executor_session.sh.crc 2150991813 4 -rwx------ 1 yarn hadoop 680 Jul 12 19:04 ./default_container_executor.sh 2150991814 4 -rw-r--r-- 1 yarn hadoop 16 Jul 12 19:04 ./.default_container_executor.sh.crc 3223210133 26948 -r-x------ 1 yarn hadoop 27591359 Jul 12 19:04 ./toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar 1075194423 16 -r-x------ 1 yarn hadoop 14747 Jul 12 19:04 ./__app__.jar 3226980184 4 drwx------ 2 yarn hadoop 4096 Jul 12 19:04 ./__spark_conf__ 3226980185 8 -r-x------ 1 yarn hadoop 6846 Jul 12 19:04 ./__spark_conf__/mapred-site.xml 3226980186 8 -r-x------ 1 yarn hadoop 6369 Jul 12 19:04 ./__spark_conf__/hadoop-env.sh 3226980187 4 -r-x------ 1 yarn hadoop 620 Jul 12 19:04 ./__spark_conf__/log4j.properties 3226980188 4 -r-x------ 1 yarn hadoop 900 Jul 12 19:04 ./__spark_conf__/hadoop-metrics2.properties 3226980189 20 -r-x------ 1 yarn hadoop 17527 Jul 12 19:04 ./__spark_conf__/yarn-site.xml 3226980190 4 -r-x------ 1 yarn hadoop 3979 Jul 12 19:04 ./__spark_conf__/hadoop-env.cmd 3226980191 0 -r-x------ 1 yarn hadoop 0 Jul 12 19:04 ./__spark_conf__/yarn.exclude 3224697025 4 -r-x------ 1 yarn hadoop 1 Jul 12 19:04 ./__spark_conf__/dfs.exclude 3221682549 4 -r-x------ 1 yarn hadoop 2614 Jul 12 19:04 ./__spark_conf__/core-site.xml 3221682551 4 -r-x------ 1 yarn hadoop 1631 Jul 12 19:04 ./__spark_conf__/kms-log4j.properties 3221682554 4 -r-x------ 1 yarn hadoop 2250 Jul 12 19:04 ./__spark_conf__/yarn-env.cmd 3221930525 4 -r-x------ 1 yarn hadoop 884 Jul 12 19:04 ./__spark_conf__/ssl-client.xml 3221930795 4 -r-x------ 1 yarn hadoop 2035 Jul 12 19:04 ./__spark_conf__/capacity-scheduler.xml 3221965927 4 -r-x------ 1 yarn hadoop 3518 Jul 12 19:04 ./__spark_conf__/kms-acls.xml 3221991375 4 -r-x------ 1 yarn hadoop 2358 Jul 12 19:04 ./__spark_conf__/topology_script.py 3221991379 4 -r-x------ 1 yarn hadoop 758 Jul 12 19:04 ./__spark_conf__/mapred-site.xml.template 3221991382 4 -r-x------ 1 yarn hadoop 1335 Jul 12 19:04 ./__spark_conf__/configuration.xsl 3221991384 8 -r-x------ 1 yarn hadoop 5310 Jul 12 19:04 ./__spark_conf__/yarn-env.sh 3223210112 8 -r-x------ 1 yarn hadoop 6748 Jul 12 19:04 ./__spark_conf__/hdfs-site.xml 3223210113 4 -r-x------ 1 yarn hadoop 1020 Jul 12 19:04 ./__spark_conf__/commons-logging.properties 3223210114 4 -r-x------ 1 yarn hadoop 1019 Jul 12 19:04 ./__spark_conf__/container-executor.cfg 3223210115 8 -r-x------ 1 yarn hadoop 4221 Jul 12 19:04 ./__spark_conf__/task-log4j.properties 3223210116 4 -r-x------ 1 yarn hadoop 2490 Jul 12 19:04 ./__spark_conf__/hadoop-metrics.properties 3223210117 4 -r-x------ 1 yarn hadoop 856 Jul 12 19:04 ./__spark_conf__/mapred-env.sh 3223210118 4 -r-x------ 1 yarn hadoop 1602 Jul 12 19:04 ./__spark_conf__/health_check 3223210119 4 -r-x------ 1 yarn hadoop 2316 Jul 12 19:04 ./__spark_conf__/ssl-client.xml.example 3223210120 4 -r-x------ 1 yarn hadoop 1527 Jul 12 19:04 ./__spark_conf__/kms-env.sh 3223210121 4 -r-x------ 1 yarn hadoop 1308 Jul 12 19:04 ./__spark_conf__/hadoop-policy.xml 3223210122 4 -r-x------ 1 yarn hadoop 93 Jul 12 19:04 ./__spark_conf__/slaves 3223210123 4 -r-x------ 1 yarn hadoop 268 Jul 12 19:04 ./__spark_conf__/topology_mappings.data 3223210124 8 -r-x------ 1 yarn hadoop 4956 Jul 12 19:04 ./__spark_conf__/metrics.properties 3223210125 4 -r-x------ 1 yarn hadoop 951 Jul 12 19:04 ./__spark_conf__/mapred-env.cmd 3223210126 4 -r-x------ 1 yarn hadoop 1000 Jul 12 19:04 ./__spark_conf__/ssl-server.xml 3223210127 4 -r-x------ 1 yarn hadoop 2697 Jul 12 19:04 ./__spark_conf__/ssl-server.xml.example 3223210128 4 -r-x------ 1 yarn hadoop 945 Jul 12 19:04 ./__spark_conf__/taskcontroller.cfg 3223210129 8 -r-x------ 1 yarn hadoop 5511 Jul 12 19:04 ./__spark_conf__/kms-site.xml 3223210130 8 -r-x------ 1 yarn hadoop 4113 Jul 12 19:04 ./__spark_conf__/mapred-queues.xml.template 3223210131 4 -r-x------ 1 yarn hadoop 1884 Jul 12 19:04 ./__spark_conf__/__spark_conf__.properties 3702028 16 drwxr-xr-x 2 yarn hadoop 12288 Jul 11 15:24 ./__spark_libs__ 3702029 176 -r-xr-xr-x 1 yarn hadoop 178947 May 30 22:46 ./__spark_libs__/hk2-api-2.4.0-b34.jar 3702031 20 -r-xr-xr-x 1 yarn hadoop 16993 May 30 22:46 ./__spark_libs__/JavaEWAH-0.3.2.jar 3702032 332 -r-xr-xr-x 1 yarn hadoop 339666 May 30 22:46 ./__spark_libs__/datanucleus-api-jdo-3.2.6.jar 3702033 200 -r-xr-xr-x 1 yarn hadoop 201928 May 30 22:46 ./__spark_libs__/RoaringBitmap-0.5.11.jar 3702034 180 -r-xr-xr-x 1 yarn hadoop 181271 May 30 22:46 ./__spark_libs__/hk2-locator-2.4.0-b34.jar 3702035 232 -r-xr-xr-x 1 yarn hadoop 236660 May 30 22:46 ./__spark_libs__/ST4-4.0.4.jar 3702036 120 -r-xr-xr-x 1 yarn hadoop 118973 May 30 22:46 ./__spark_libs__/hk2-utils-2.4.0-b34.jar 3702037 68 -r-xr-xr-x 1 yarn hadoop 69409 May 30 22:46 ./__spark_libs__/activation-1.1.1.jar 3702038 320 -r-xr-xr-x 1 yarn hadoop 326724 May 30 22:46 ./__spark_libs__/httpcore-4.4.4.jar 3702039 436 -r-xr-xr-x 1 yarn hadoop 445288 May 30 22:46 ./__spark_libs__/antlr-2.7.7.jar 3702040 1848 -r-xr-xr-x 1 yarn hadoop 1890075 May 30 22:46 ./__spark_libs__/datanucleus-core-3.2.10.jar 3702041 164 -r-xr-xr-x 1 yarn hadoop 164368 May 30 22:46 ./__spark_libs__/antlr-runtime-3.4.jar 3702042 1768 -r-xr-xr-x 1 yarn hadoop 1809447 May 30 22:46 ./__spark_libs__/datanucleus-rdbms-3.2.9.jar 3702043 296 -r-xr-xr-x 1 yarn hadoop 302248 May 30 22:46 ./__spark_libs__/antlr4-runtime-4.5.3.jar 609856 720 -r-xr-xr-x 1 yarn hadoop 736658 May 30 22:46 ./__spark_libs__/httpclient-4.5.2.jar 609857 8 -r-xr-xr-x 1 yarn hadoop 4467 May 30 22:46 ./__spark_libs__/aopalliance-1.0.jar 609858 132 -r-xr-xr-x 1 yarn hadoop 134595 May 30 22:46 ./__spark_libs__/commons-crypto-1.0.0.jar 609859 16 -r-xr-xr-x 1 yarn hadoop 14766 May 30 22:46 ./__spark_libs__/aopalliance-repackaged-2.4.0-b34.jar 609860 160 -r-xr-xr-x 1 yarn hadoop 160519 May 30 22:46 ./__spark_libs__/commons-dbcp-1.4.jar 609861 12 -r-xr-xr-x 1 yarn hadoop 12131 May 30 22:46 ./__spark_libs__/jpam-1.1.jar 609862 440 -r-xr-xr-x 1 yarn hadoop 448794 May 30 22:46 ./__spark_libs__/apache-log4j-extras-1.2.17.jar 609863 3152 -r-xr-xr-x 1 yarn hadoop 3224708 May 30 22:46 ./__spark_libs__/derby-10.12.1.1.jar 609864 44 -r-xr-xr-x 1 yarn hadoop 44925 May 30 22:46 ./__spark_libs__/apacheds-i18n-2.0.0-M15.jar 609865 508 -r-xr-xr-x 1 yarn hadoop 516634 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-app-2.7.3.2.6.1.0-129.jar 609866 676 -r-xr-xr-x 1 yarn hadoop 691479 May 30 22:46 ./__spark_libs__/apacheds-kerberos-codec-2.0.0-M15.jar 609867 20 -r-xr-xr-x 1 yarn hadoop 18482 May 30 22:46 ./__spark_libs__/eigenbase-properties-1.1.5.jar 609868 20 -r-xr-xr-x 1 yarn hadoop 16560 May 30 22:46 ./__spark_libs__/api-asn1-api-1.0.0-M20.jar 609869 188 -r-xr-xr-x 1 yarn hadoop 190432 May 30 22:46 ./__spark_libs__/gson-2.2.4.jar 609870 80 -r-xr-xr-x 1 yarn hadoop 79912 May 30 22:46 ./__spark_libs__/api-util-1.0.0-M20.jar 609871 2140 -r-xr-xr-x 1 yarn hadoop 2189117 May 30 22:46 ./__spark_libs__/guava-14.0.1.jar 609872 1168 -r-xr-xr-x 1 yarn hadoop 1194003 May 30 22:46 ./__spark_libs__/arpack_combined_all-0.1.jar 609873 1256 -r-xr-xr-x 1 yarn hadoop 1282424 May 30 22:46 ./__spark_libs__/ivy-2.4.0.jar 609874 428 -r-xr-xr-x 1 yarn hadoop 436303 May 30 22:46 ./__spark_libs__/avro-1.7.7.jar 609875 256 -r-xr-xr-x 1 yarn hadoop 258876 May 30 22:46 ./__spark_libs__/jackson-core-2.6.5.jar 609876 192 -r-xr-xr-x 1 yarn hadoop 192993 May 30 22:46 ./__spark_libs__/avro-ipc-1.7.7.jar 609877 144 -r-xr-xr-x 1 yarn hadoop 143602 May 30 22:46 ./__spark_libs__/commons-digester-1.8.jar 609878 180 -r-xr-xr-x 1 yarn hadoop 180736 May 30 22:46 ./__spark_libs__/avro-mapred-1.7.7-hadoop2.jar 609879 696 -r-xr-xr-x 1 yarn hadoop 710492 May 30 22:46 ./__spark_libs__/guice-3.0.jar 609880 16 -r-xr-xr-x 1 yarn hadoop 15071 May 30 22:46 ./__spark_libs__/jta-1.1.jar 609881 504 -r-xr-xr-x 1 yarn hadoop 516062 May 30 22:46 ./__spark_libs__/aws-java-sdk-core-1.10.6.jar 609882 64 -r-xr-xr-x 1 yarn hadoop 65012 May 30 22:46 ./__spark_libs__/guice-servlet-3.0.jar 609883 256 -r-xr-xr-x 1 yarn hadoop 258578 May 30 22:46 ./__spark_libs__/aws-java-sdk-kms-1.10.6.jar 609884 80 -r-xr-xr-x 1 yarn hadoop 79175 May 30 22:46 ./__spark_libs__/hadoop-auth-2.7.3.2.6.1.0-129.jar 609885 560 -r-xr-xr-x 1 yarn hadoop 570101 May 30 22:46 ./__spark_libs__/aws-java-sdk-s3-1.10.6.jar 609886 1988 -r-xr-xr-x 1 yarn hadoop 2035066 May 30 22:46 ./__spark_libs__/commons-math3-3.4.1.jar 609887 80 -r-xr-xr-x 1 yarn hadoop 78849 May 30 22:46 ./__spark_libs__/azure-data-lake-store-sdk-2.1.4.jar 609888 208 -r-xr-xr-x 1 yarn hadoop 212453 May 30 22:46 ./__spark_libs__/commons-net-2.2.jar 609889 232 -r-xr-xr-x 1 yarn hadoop 236880 May 30 22:46 ./__spark_libs__/lz4-1.3.0.jar 609890 12 -r-xr-xr-x 1 yarn hadoop 10092 May 30 22:46 ./__spark_libs__/azure-keyvault-core-0.8.0.jar 609891 220 -r-xr-xr-x 1 yarn hadoop 222041 May 30 22:46 ./__spark_libs__/hadoop-aws-2.7.3.2.6.1.0-129.jar 609892 728 -r-xr-xr-x 1 yarn hadoop 745325 May 30 22:46 ./__spark_libs__/azure-storage-4.2.0.jar 609893 48 -r-xr-xr-x 1 yarn hadoop 46983 May 30 22:46 ./__spark_libs__/jackson-annotations-2.6.5.jar 609894 20 -r-xr-xr-x 1 yarn hadoop 17008 May 30 22:46 ./__spark_libs__/base64-2.3.8.jar 609895 212 -r-xr-xr-x 1 yarn hadoop 213891 May 30 22:46 ./__spark_libs__/hadoop-azure-2.7.3.2.6.1.0-129.jar 609896 2780 -r-xr-xr-x 1 yarn hadoop 2842667 May 30 22:46 ./__spark_libs__/bcprov-jdk15on-1.51.jar 609897 4 -r-xr-xr-x 1 yarn hadoop 2567 May 30 22:46 ./__spark_libs__/hadoop-client-2.7.3.2.6.1.0-129.jar 609898 112 -r-xr-xr-x 1 yarn hadoop 110600 May 30 22:46 ./__spark_libs__/bonecp-0.8.0.RELEASE.jar 609899 3564 -r-xr-xr-x 1 yarn hadoop 3647240 May 30 22:46 ./__spark_libs__/hadoop-common-2.7.3.2.6.1.0-129.jar 609900 180 -r-xr-xr-x 1 yarn hadoop 181926 May 30 22:46 ./__spark_libs__/breeze-macros_2.11-0.12.jar 609901 228 -r-xr-xr-x 1 yarn hadoop 232248 May 30 22:46 ./__spark_libs__/jackson-core-asl-1.9.13.jar 609902 14080 -r-xr-xr-x 1 yarn hadoop 14413834 May 30 22:46 ./__spark_libs__/breeze_2.11-0.12.jar 609903 96 -r-xr-xr-x 1 yarn hadoop 96221 May 30 22:46 ./__spark_libs__/commons-pool-1.5.4.jar 609904 256 -r-xr-xr-x 1 yarn hadoop 258370 May 30 22:46 ./__spark_libs__/calcite-avatica-1.2.0-incubating.jar 609905 80 -r-xr-xr-x 1 yarn hadoop 79845 May 30 22:46 ./__spark_libs__/compress-lzf-1.0.3.jar 609906 3440 -r-xr-xr-x 1 yarn hadoop 3519262 May 30 22:46 ./__spark_libs__/calcite-core-1.2.0-incubating.jar 609907 164 -r-xr-xr-x 1 yarn hadoop 164422 May 30 22:46 ./__spark_libs__/core-1.1.2.jar 609908 64 -r-xr-xr-x 1 yarn hadoop 65261 May 30 22:46 ./__spark_libs__/oro-2.0.8.jar 609909 436 -r-xr-xr-x 1 yarn hadoop 442406 May 30 22:46 ./__spark_libs__/calcite-linq4j-1.2.0-incubating.jar 609910 1144 -r-xr-xr-x 1 yarn hadoop 1171380 May 30 22:46 ./__spark_libs__/jackson-databind-2.6.5.jar 609911 52 -r-xr-xr-x 1 yarn hadoop 50619 May 30 22:46 ./__spark_libs__/chill-java-0.8.0.jar 609912 20 -r-xr-xr-x 1 yarn hadoop 18336 May 30 22:46 ./__spark_libs__/jackson-jaxrs-1.9.13.jar 609913 220 -r-xr-xr-x 1 yarn hadoop 223573 May 30 22:46 ./__spark_libs__/chill_2.11-0.8.0.jar 609914 8508 -r-xr-xr-x 1 yarn hadoop 8711541 May 30 22:46 ./__spark_libs__/hadoop-hdfs-2.7.3.2.6.1.0-129.jar 609915 188 -r-xr-xr-x 1 yarn hadoop 188671 May 30 22:46 ./__spark_libs__/commons-beanutils-1.7.0.jar 609916 68 -r-xr-xr-x 1 yarn hadoop 68866 May 30 22:46 ./__spark_libs__/curator-client-2.6.0.jar 609917 204 -r-xr-xr-x 1 yarn hadoop 206035 May 30 22:46 ./__spark_libs__/commons-beanutils-core-1.8.0.jar 609918 764 -r-xr-xr-x 1 yarn hadoop 780664 May 30 22:46 ./__spark_libs__/jackson-mapper-asl-1.9.13.jar 609919 44 -r-xr-xr-x 1 yarn hadoop 41123 May 30 22:46 ./__spark_libs__/commons-cli-1.2.jar 609920 2068 -r-xr-xr-x 1 yarn hadoop 2117555 May 30 22:46 ./__spark_libs__/hadoop-yarn-api-2.7.3.2.6.1.0-129.jar 609921 280 -r-xr-xr-x 1 yarn hadoop 284184 May 30 22:46 ./__spark_libs__/commons-codec-1.10.jar 609922 184 -r-xr-xr-x 1 yarn hadoop 185245 May 30 22:46 ./__spark_libs__/curator-framework-2.6.0.jar 609923 576 -r-xr-xr-x 1 yarn hadoop 588337 May 30 22:46 ./__spark_libs__/commons-collections-3.2.2.jar 609924 136 -r-xr-xr-x 1 yarn hadoop 138453 May 30 22:46 ./__spark_libs__/hive-beeline-1.2.1.spark2.hdp.1.jar 609925 36 -r-xr-xr-x 1 yarn hadoop 36627 May 30 22:46 ./__spark_libs__/commons-compiler-3.0.0.jar 609926 40 -r-xr-xr-x 1 yarn hadoop 40864 May 30 22:46 ./__spark_libs__/hive-cli-1.2.1.spark2.hdp.1.jar 609927 236 -r-xr-xr-x 1 yarn hadoop 241367 May 30 22:46 ./__spark_libs__/commons-compress-1.4.1.jar 609928 244 -r-xr-xr-x 1 yarn hadoop 248171 May 30 22:46 ./__spark_libs__/curator-recipes-2.6.0.jar 609929 292 -r-xr-xr-x 1 yarn hadoop 298829 May 30 22:46 ./__spark_libs__/commons-configuration-1.6.jar 609930 300 -r-xr-xr-x 1 yarn hadoop 305001 May 30 22:46 ./__spark_libs__/commons-httpclient-3.1.jar 609931 28 -r-xr-xr-x 1 yarn hadoop 27084 May 30 22:46 ./__spark_libs__/jackson-xc-1.9.13.jar 609932 184 -r-xr-xr-x 1 yarn hadoop 185140 May 30 22:46 ./__spark_libs__/commons-io-2.4.jar 609933 12 -r-xr-xr-x 1 yarn hadoop 10023 May 30 22:46 ./__spark_libs__/java-xmlbuilder-1.0.jar 609934 280 -r-xr-xr-x 1 yarn hadoop 284220 May 30 22:46 ./__spark_libs__/commons-lang-2.6.jar 609935 11232 -r-xr-xr-x 1 yarn hadoop 11499183 May 30 22:46 ./__spark_libs__/hive-exec-1.2.1.spark2.hdp.1.jar 609936 472 -r-xr-xr-x 1 yarn hadoop 479881 May 30 22:46 ./__spark_libs__/commons-lang3-3.5.jar 609937 100 -r-xr-xr-x 1 yarn hadoop 101402 May 30 22:46 ./__spark_libs__/hive-jdbc-1.2.1.spark2.hdp.1.jar 609938 64 -r-xr-xr-x 1 yarn hadoop 62050 May 30 22:46 ./__spark_libs__/commons-logging-1.1.3.jar 609939 720 -r-xr-xr-x 1 yarn hadoop 735574 May 30 22:46 ./__spark_libs__/janino-3.0.0.jar 609940 20 -r-xr-xr-x 1 yarn hadoop 17369 May 30 22:46 ./__spark_libs__/hadoop-annotations-2.7.3.2.6.1.0-129.jar 609941 24 -r-xr-xr-x 1 yarn hadoop 23519 May 30 22:46 ./__spark_libs__/hadoop-azure-datalake-2.7.3.2.6.1.0-129.jar 609942 52 -r-xr-xr-x 1 yarn hadoop 51484 May 30 22:46 ./__spark_libs__/json-smart-1.1.1.jar 609943 280 -r-xr-xr-x 1 yarn hadoop 285479 May 30 22:46 ./__spark_libs__/parquet-encoding-1.8.1.jar 609944 736 -r-xr-xr-x 1 yarn hadoop 752754 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-common-2.7.3.2.6.1.0-129.jar 609961 96 -r-xr-xr-x 1 yarn hadoop 94672 May 30 22:46 ./__spark_libs__/xz-1.0.jar 609945 84 -r-xr-xr-x 1 yarn hadoop 82421 May 30 22:46 ./__spark_libs__/json4s-ast_2.11-3.2.11.jar 609946 788 -r-xr-xr-x 1 yarn hadoop 802818 May 30 22:46 ./__spark_libs__/scalap-2.11.8.jar 609947 1500 -r-xr-xr-x 1 yarn hadoop 1535771 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-core-2.7.3.2.6.1.0-129.jar 609948 96 -r-xr-xr-x 1 yarn hadoop 94796 May 30 22:46 ./__spark_libs__/pyrolite-4.13.jar 609949 116 -r-xr-xr-x 1 yarn hadoop 115472 May 30 22:46 ./__spark_libs__/py4j-0.10.4.jar 609950 40 -r-xr-xr-x 1 yarn hadoop 39398 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-jobclient-2.7.3.2.6.1.0-129.jar 609951 420 -r-xr-xr-x 1 yarn hadoop 427780 May 30 22:46 ./__spark_libs__/jodd-core-3.5.2.jar 609952 40 -r-xr-xr-x 1 yarn hadoop 39280 May 30 22:46 ./__spark_libs__/metrics-jvm-3.1.2.jar 609953 56 -r-xr-xr-x 1 yarn hadoop 54074 May 30 22:46 ./__spark_libs__/hadoop-mapreduce-client-shuffle-2.7.3.2.6.1.0-129.jar 609954 36 -r-xr-xr-x 1 yarn hadoop 33015 May 30 22:46 ./__spark_libs__/jsr305-1.3.9.jar 609955 40 -r-xr-xr-x 1 yarn hadoop 40509 May 30 22:46 ./__spark_libs__/slf4j-api-1.7.16.jar 609956 116 -r-xr-xr-x 1 yarn hadoop 114748 May 30 22:46 ./__spark_libs__/hadoop-openstack-2.7.3.2.6.1.0-129.jar 609957 748 -r-xr-xr-x 1 yarn hadoop 764569 May 30 22:46 ./__spark_libs__/jtransforms-2.4.0.jar 609958 48 -r-xr-xr-x 1 yarn hadoop 48720 May 30 22:46 ./__spark_libs__/snappy-0.2.jar 609959 192 -r-xr-xr-x 1 yarn hadoop 193240 May 30 22:46 ./__spark_libs__/hadoop-yarn-client-2.7.3.2.6.1.0-129.jar 609960 8 -r-xr-xr-x 1 yarn hadoop 4596 May 30 22:46 ./__spark_libs__/jul-to-slf4j-1.7.16.jar 609962 1712 -r-xr-xr-x 1 yarn hadoop 1750536 May 30 22:46 ./__spark_libs__/hadoop-yarn-common-2.7.3.2.6.1.0-129.jar 609963 352 -r-xr-xr-x 1 yarn hadoop 358390 May 30 22:46 ./__spark_libs__/kryo-shaded-3.0.3.jar 609964 7108 -r-xr-xr-x 1 yarn hadoop 7276083 May 30 22:46 ./__spark_libs__/spire_2.11-0.7.4.jar 609965 100 -r-xr-xr-x 1 yarn hadoop 101126 May 30 22:46 ./__spark_libs__/hadoop-yarn-registry-2.7.3.2.6.1.0-129.jar 609966 576 -r-xr-xr-x 1 yarn hadoop 589462 May 30 22:46 ./__spark_libs__/json4s-core_2.11-3.2.11.jar 609967 24 -r-xr-xr-x 1 yarn hadoop 23346 May 30 22:46 ./__spark_libs__/stax-api-1.0-2.jar 609968 412 -r-xr-xr-x 1 yarn hadoop 420788 May 30 22:46 ./__spark_libs__/hadoop-yarn-server-common-2.7.3.2.6.1.0-129.jar 609969 40 -r-xr-xr-x 1 yarn hadoop 40341 May 30 22:46 ./__spark_libs__/json4s-jackson_2.11-3.2.11.jar 609970 36 -r-xr-xr-x 1 yarn hadoop 36249 May 30 22:46 ./__spark_libs__/hadoop-yarn-server-web-proxy-2.7.3.2.6.1.0-129.jar 609971 1024 -r-xr-xr-x 1 yarn hadoop 1045744 May 30 22:46 ./__spark_libs__/leveldbjni-all-1.8.jar 609972 5380 -r-xr-xr-x 1 yarn hadoop 5505070 May 30 22:46 ./__spark_libs__/hive-metastore-1.2.1.spark2.hdp.1.jar 609973 232 -r-xr-xr-x 1 yarn hadoop 234201 May 30 22:46 ./__spark_libs__/libthrift-0.9.3.jar 609974 1444 -r-xr-xr-x 1 yarn hadoop 1475955 May 30 22:46 ./__spark_libs__/htrace-core-3.1.0-incubating.jar 609975 480 -r-xr-xr-x 1 yarn hadoop 489884 May 30 22:46 ./__spark_libs__/log4j-1.2.17.jar 609976 48 -r-xr-xr-x 1 yarn hadoop 48418 May 30 22:46 ./__spark_libs__/jackson-dataformat-cbor-2.6.5.jar 609977 512 -r-xr-xr-x 1 yarn hadoop 521157 May 30 22:46 ./__spark_libs__/mail-1.4.7.jar 609978 44 -r-xr-xr-x 1 yarn hadoop 41263 May 30 22:46 ./__spark_libs__/jackson-module-paranamer-2.6.5.jar 609979 112 -r-xr-xr-x 1 yarn hadoop 112558 May 30 22:46 ./__spark_libs__/metrics-core-3.1.2.jar 609980 504 -r-xr-xr-x 1 yarn hadoop 515604 May 30 22:46 ./__spark_libs__/jackson-module-scala_2.11-2.6.5.jar 609981 8 -r-xr-xr-x 1 yarn hadoop 5711 May 30 22:46 ./__spark_libs__/minlog-1.3.0.jar 609982 700 -r-xr-xr-x 1 yarn hadoop 714194 May 30 22:46 ./__spark_libs__/javassist-3.18.1-GA.jar 609983 400 -r-xr-xr-x 1 yarn hadoop 409467 May 30 22:46 ./__spark_libs__/mx4j-3.0.2.jar 773376 28 -r-xr-xr-x 1 yarn hadoop 26366 May 30 22:46 ./__spark_libs__/javax.annotation-api-1.2.jar 773377 216 -r-xr-xr-x 1 yarn hadoop 218076 May 30 22:46 ./__spark_libs__/parquet-hadoop-1.8.1.jar 773378 4 -r-xr-xr-x 1 yarn hadoop 2497 May 30 22:46 ./__spark_libs__/javax.inject-1.jar 773379 1204 -r-xr-xr-x 1 yarn hadoop 1230201 May 30 22:46 ./__spark_libs__/netty-3.8.0.Final.jar 773380 8 -r-xr-xr-x 1 yarn hadoop 5950 May 30 22:46 ./__spark_libs__/javax.inject-2.4.0-b34.jar 773381 2264 -r-xr-xr-x 1 yarn hadoop 2314839 May 30 22:46 ./__spark_libs__/netty-all-4.0.42.Final.jar 773382 96 -r-xr-xr-x 1 yarn hadoop 95806 May 30 22:46 ./__spark_libs__/javax.servlet-api-3.1.0.jar 773383 176 -r-xr-xr-x 1 yarn hadoop 178648 May 30 22:46 ./__spark_libs__/nimbus-jose-jwt-3.9.jar 773384 116 -r-xr-xr-x 1 yarn hadoop 115534 May 30 22:46 ./__spark_libs__/javax.ws.rs-api-2.0.1.jar 773385 1024 -r-xr-xr-x 1 yarn hadoop 1048117 May 30 22:46 ./__spark_libs__/parquet-jackson-1.8.1.jar 773386 388 -r-xr-xr-x 1 yarn hadoop 395195 May 30 22:46 ./__spark_libs__/javolution-5.5.1.jar 773387 640 -r-xr-xr-x 1 yarn hadoop 654216 May 30 22:46 ./__spark_libs__/pmml-model-1.2.15.jar 773388 104 -r-xr-xr-x 1 yarn hadoop 105134 May 30 22:46 ./__spark_libs__/jaxb-api-2.2.2.jar 773389 44 -r-xr-xr-x 1 yarn hadoop 41755 May 30 22:46 ./__spark_libs__/objenesis-2.1.jar 773390 4 -r-xr-xr-x 1 yarn hadoop 2254 May 30 22:46 ./__spark_libs__/jcip-annotations-1.0.jar 773391 312 -r-xr-xr-x 1 yarn hadoop 319099 May 30 22:46 ./__spark_libs__/okhttp-2.4.0.jar 773392 20 -r-xr-xr-x 1 yarn hadoop 16430 May 30 22:46 ./__spark_libs__/jcl-over-slf4j-1.7.16.jar 773393 8 -r-xr-xr-x 1 yarn hadoop 5310 May 30 22:46 ./__spark_libs__/pmml-schema-1.2.15.jar 773394 200 -r-xr-xr-x 1 yarn hadoop 201124 May 30 22:46 ./__spark_libs__/jdo-api-3.0.1.jar 773395 64 -r-xr-xr-x 1 yarn hadoop 64661 May 30 22:46 ./__spark_libs__/okio-1.4.0.jar 773396 164 -r-xr-xr-x 1 yarn hadoop 167421 May 30 22:46 ./__spark_libs__/jersey-client-2.22.2.jar 773397 20 -r-xr-xr-x 1 yarn hadoop 19827 May 30 22:46 ./__spark_libs__/opencsv-2.3.jar 773398 684 -r-xr-xr-x 1 yarn hadoop 698375 May 30 22:46 ./__spark_libs__/jersey-common-2.22.2.jar 773399 24 -r-xr-xr-x 1 yarn hadoop 20852 May 30 22:46 ./__spark_libs__/metrics-graphite-3.1.2.jar 773400 20 -r-xr-xr-x 1 yarn hadoop 18098 May 30 22:46 ./__spark_libs__/jersey-container-servlet-2.22.2.jar 773401 308 -r-xr-xr-x 1 yarn hadoop 313702 May 30 22:46 ./__spark_libs__/libfb303-0.9.3.jar 773402 28 -r-xr-xr-x 1 yarn hadoop 26514 May 30 22:46 ./__spark_libs__/stax-api-1.0.1.jar 773403 68 -r-xr-xr-x 1 yarn hadoop 66270 May 30 22:46 ./__spark_libs__/jersey-container-servlet-core-2.22.2.jar 773404 20 -r-xr-xr-x 1 yarn hadoop 20235 May 30 22:46 ./__spark_libs__/osgi-resource-locator-1.0.1.jar 773405 952 -r-xr-xr-x 1 yarn hadoop 971310 May 30 22:46 ./__spark_libs__/jersey-guava-2.22.2.jar 773406 32 -r-xr-xr-x 1 yarn hadoop 29555 May 30 22:46 ./__spark_libs__/paranamer-2.3.jar 773407 72 -r-xr-xr-x 1 yarn hadoop 72733 May 30 22:46 ./__spark_libs__/jersey-media-jaxb-2.22.2.jar 773408 924 -r-xr-xr-x 1 yarn hadoop 945914 May 30 22:46 ./__spark_libs__/parquet-column-1.8.1.jar 773409 932 -r-xr-xr-x 1 yarn hadoop 951701 May 30 22:46 ./__spark_libs__/jersey-server-2.22.2.jar 773410 524 -r-xr-xr-x 1 yarn hadoop 533455 May 30 22:46 ./__spark_libs__/protobuf-java-2.5.0.jar 773411 1996 -r-xr-xr-x 1 yarn hadoop 2041628 May 30 22:46 ./__spark_libs__/jets3t-0.9.3.jar 773412 5612 -r-xr-xr-x 1 yarn hadoop 5744974 May 30 22:46 ./__spark_libs__/scala-library-2.11.8.jar 773413 528 -r-xr-xr-x 1 yarn hadoop 540298 May 30 22:46 ./__spark_libs__/jetty-6.1.26.hwx.jar 773414 16 -r-xr-xr-x 1 yarn hadoop 15827 May 30 22:46 ./__spark_libs__/metrics-json-3.1.2.jar 773415 20 -r-xr-xr-x 1 yarn hadoop 18899 May 30 22:46 ./__spark_libs__/jetty-sslengine-6.1.26.hwx.jar 773416 40 -r-xr-xr-x 1 yarn hadoop 38604 May 30 22:46 ./__spark_libs__/parquet-common-1.8.1.jar 773417 176 -r-xr-xr-x 1 yarn hadoop 177214 May 30 22:46 ./__spark_libs__/jetty-util-6.1.26.hwx.jar 773418 15128 -r-xr-xr-x 1 yarn hadoop 15487351 May 30 22:46 ./__spark_libs__/scala-compiler-2.11.8.jar 773419 212 -r-xr-xr-x 1 yarn hadoop 213911 May 30 22:46 ./__spark_libs__/jline-2.12.1.jar 773420 4468 -r-xr-xr-x 1 yarn hadoop 4573750 May 30 22:46 ./__spark_libs__/scala-reflect-2.11.8.jar 773421 616 -r-xr-xr-x 1 yarn hadoop 627814 May 30 22:46 ./__spark_libs__/joda-time-2.9.3.jar 773422 100 -r-xr-xr-x 1 yarn hadoop 100636 May 30 22:46 ./__spark_libs__/jsp-api-2.1.jar 773423 636 -r-xr-xr-x 1 yarn hadoop 648678 May 30 22:46 ./__spark_libs__/scala-xml_2.11-1.0.2.jar 773424 380 -r-xr-xr-x 1 yarn hadoop 387188 May 30 22:46 ./__spark_libs__/parquet-format-2.3.0-incubating.jar 773425 2732 -r-xr-xr-x 1 yarn hadoop 2796935 May 30 22:46 ./__spark_libs__/parquet-hadoop-bundle-1.6.0.jar 773426 416 -r-xr-xr-x 1 yarn hadoop 423753 May 30 22:46 ./__spark_libs__/scala-parser-combinators_2.11-1.0.4.jar 773427 2004 -r-xr-xr-x 1 yarn hadoop 2049451 May 30 22:46 ./__spark_libs__/shapeless_2.11-2.0.0.jar 773428 12 -r-xr-xr-x 1 yarn hadoop 9939 May 30 22:46 ./__spark_libs__/slf4j-log4j12-1.7.16.jar 773429 1032 -r-xr-xr-x 1 yarn hadoop 1056168 May 30 22:46 ./__spark_libs__/snappy-java-1.1.2.6.jar 773430 7336 -r-xr-xr-x 1 yarn hadoop 7508336 May 30 22:46 ./__spark_libs__/spark-catalyst_2.11-2.1.1.2.6.1.0-129.jar 773431 84 -r-xr-xr-x 1 yarn hadoop 82916 May 30 22:46 ./__spark_libs__/spark-cloud_2.11-2.1.1.2.6.1.0-129.jar 773432 11668 -r-xr-xr-x 1 yarn hadoop 11946206 May 30 22:46 ./__spark_libs__/spark-core_2.11-2.1.1.2.6.1.0-129.jar 773433 684 -r-xr-xr-x 1 yarn hadoop 699290 May 30 22:46 ./__spark_libs__/spark-graphx_2.11-2.1.1.2.6.1.0-129.jar 773434 1776 -r-xr-xr-x 1 yarn hadoop 1815017 May 30 22:46 ./__spark_libs__/spark-hive-thriftserver_2.11-2.1.1.2.6.1.0-129.jar 773435 1208 -r-xr-xr-x 1 yarn hadoop 1234838 May 30 22:46 ./__spark_libs__/spark-hive_2.11-2.1.1.2.6.1.0-129.jar 874216 88 -r-xr-xr-x 1 yarn hadoop 86811 May 30 22:46 ./__spark_libs__/spire-macros_2.11-0.7.4.jar 773436 72 -r-xr-xr-x 1 yarn hadoop 70595 May 30 22:46 ./__spark_libs__/spark-launcher_2.11-2.1.1.2.6.1.0-129.jar 773437 176 -r-xr-xr-x 1 yarn hadoop 178967 May 30 22:46 ./__spark_libs__/spark-mllib-local_2.11-2.1.1.2.6.1.0-129.jar 773438 6048 -r-xr-xr-x 1 yarn hadoop 6190594 May 30 22:46 ./__spark_libs__/spark-mllib_2.11-2.1.1.2.6.1.0-129.jar 773439 2304 -r-xr-xr-x 1 yarn hadoop 2356803 May 30 22:46 ./__spark_libs__/spark-network-common_2.11-2.1.1.2.6.1.0-129.jar 874208 64 -r-xr-xr-x 1 yarn hadoop 61843 May 30 22:46 ./__spark_libs__/spark-network-shuffle_2.11-2.1.1.2.6.1.0-129.jar 874209 68 -r-xr-xr-x 1 yarn hadoop 65842 May 30 22:46 ./__spark_libs__/spark-repl_2.11-2.1.1.2.6.1.0-129.jar 874210 32 -r-xr-xr-x 1 yarn hadoop 29765 May 30 22:46 ./__spark_libs__/spark-sketch_2.11-2.1.1.2.6.1.0-129.jar 874211 6504 -r-xr-xr-x 1 yarn hadoop 6657160 May 30 22:46 ./__spark_libs__/spark-sql_2.11-2.1.1.2.6.1.0-129.jar 874212 2052 -r-xr-xr-x 1 yarn hadoop 2101086 May 30 22:46 ./__spark_libs__/spark-streaming_2.11-2.1.1.2.6.1.0-129.jar 874213 16 -r-xr-xr-x 1 yarn hadoop 15527 May 30 22:46 ./__spark_libs__/spark-tags_2.11-2.1.1.2.6.1.0-129.jar 874214 48 -r-xr-xr-x 1 yarn hadoop 45278 May 30 22:46 ./__spark_libs__/spark-unsafe_2.11-2.1.1.2.6.1.0-129.jar 874215 728 -r-xr-xr-x 1 yarn hadoop 742074 May 30 22:46 ./__spark_libs__/spark-yarn_2.11-2.1.1.2.6.1.0-129.jar 874217 172 -r-xr-xr-x 1 yarn hadoop 174351 May 30 22:46 ./__spark_libs__/stream-2.7.0.jar 874218 148 -r-xr-xr-x 1 yarn hadoop 148627 May 30 22:46 ./__spark_libs__/stringtemplate-3.2.1.jar 874219 92 -r-xr-xr-x 1 yarn hadoop 93210 May 30 22:46 ./__spark_libs__/super-csv-2.2.0.jar 874220 320 -r-xr-xr-x 1 yarn hadoop 325553 May 30 22:46 ./__spark_libs__/univocity-parsers-2.2.1.jar 874221 64 -r-xr-xr-x 1 yarn hadoop 63777 May 30 22:46 ./__spark_libs__/validation-api-1.1.0.Final.jar 874222 144 -r-xr-xr-x 1 yarn hadoop 144660 May 30 22:46 ./__spark_libs__/xbean-asm5-shaded-4.4.jar 874223 1204 -r-xr-xr-x 1 yarn hadoop 1229125 May 30 22:46 ./__spark_libs__/xercesImpl-2.9.1.jar 874224 16 -r-xr-xr-x 1 yarn hadoop 15010 May 30 22:46 ./__spark_libs__/xmlenc-0.52.jar 874225 780 -r-xr-xr-x 1 yarn hadoop 797314 May 30 22:46 ./__spark_libs__/zookeeper-3.4.6.2.6.1.0-129.jar broken symlinks(find -L . -maxdepth 5 -type l -ls): End of LogType:directory.info LogType:launch_container.sh Log Upload Time:Wed Jul 12 19:07:15 -0500 2017 LogLength:5671 Log Contents: #!/bin/bash export SPARK_YARN_STAGING_DIR="hdfs://spark-02.softlayer.com:8020/user/elyra/.sparkStaging/application_1499804611229_0007" export PATH="/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent" export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/usr/hdp/current/hadoop-client/conf"} export MAX_APP_ATTEMPTS="2" export JAVA_HOME=${JAVA_HOME:-"/usr/jdk64/jdk1.8.0_112"} export LANG="en_US.UTF-8" export APP_SUBMIT_TIME_ENV="1499904284023" export NM_HOST="spark-04.softlayer.com" export LOGNAME="elyra" export JVM_PID="$$" export PWD="/hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007/container_1499804611229_0007_01_000001" export LOCAL_DIRS="/hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007" export APPLICATION_WEB_PROXY_BASE="/proxy/application_1499804611229_0007" export NM_HTTP_PORT="8042" export LOG_DIRS="/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001" export NM_AUX_SERVICE_mapreduce_shuffle="AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= " export NM_PORT="45454" export USER="elyra" export HADOOP_YARN_HOME=${HADOOP_YARN_HOME:-"/usr/hdp/current/hadoop-yarn-nodemanager"} export CLASSPATH="$PWD:$PWD/__spark_conf__:$PWD/__spark_libs__/*:/etc/hadoop/conf:/usr/hdp/current/hadoop-client/*:/usr/hdp/current/hadoop-client/lib/*:/usr/hdp/current/hadoop-hdfs-client/*:/usr/hdp/current/hadoop-hdfs-client/lib/*:/usr/hdp/current/hadoop-yarn-client/*:/usr/hdp/current/hadoop-yarn-client/lib/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.1.0-129/hadoop/lib/hadoop-lzo-0.6.0.2.6.1.0-129.jar:/etc/hadoop/conf/secure" export SPARK_YARN_MODE="true" export HADOOP_TOKEN_FILE_LOCATION="/hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007/container_1499804611229_0007_01_000001/container_tokens" export NM_AUX_SERVICE_spark_shuffle="" export SPARK_USER="elyra" export LOCAL_USER_DIRS="/hadoop/yarn/local/usercache/elyra/" export HADOOP_HOME="/usr/hdp/2.6.1.0-129/hadoop" export HOME="/home/" export NM_AUX_SERVICE_spark2_shuffle="" export CONTAINER_ID="container_1499804611229_0007_01_000001" export MALLOC_ARENA_MAX="4" ln -sf "/hadoop/yarn/local/usercache/elyra/filecache/15/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar" "toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/hadoop/yarn/local/usercache/elyra/filecache/14/toree-launcher_2.11-1.jar" "__app__.jar" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/hadoop/yarn/local/usercache/elyra/filecache/13/__spark_conf__.zip" "__spark_conf__" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi ln -sf "/hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz" "__spark_libs__" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi # Creating copy of launch script cp "launch_container.sh" "/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001/launch_container.sh" chmod 640 "/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001/launch_container.sh" # Determining directory contents echo "ls -l:" 1>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001/directory.info" ls -l 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001/directory.info" echo "find -L . -maxdepth 5 -ls:" 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001/directory.info" find -L . -maxdepth 5 -ls 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001/directory.info" echo "broken symlinks(find -L . -maxdepth 5 -type l -ls):" 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001/directory.info" find -L . -maxdepth 5 -type l -ls 1>>"/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001/directory.info" exec /bin/bash -c "LD_LIBRARY_PATH="/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH" $JAVA_HOME/bin/java -server -Xmx1024m -Djava.io.tmpdir=$PWD/tmp -Dhdp.version=2.6.1.0-129 -Dspark.yarn.app.container.log.dir=/hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'launcher.ToreeLauncher' --jar file:/usr/local/share/jupyter/kernels/spark_2.1_scala_yarn_cluster/lib/toree-launcher_2.11-1.jar --arg '--response-address' --arg 'ERROR__NO__KERNEL_RESPONSE_ADDRESS' --arg '--profile' --arg '/var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json' --properties-file $PWD/__spark_conf__/__spark_conf__.properties 1> /hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001/stdout 2> /hadoop/yarn/log/application_1499804611229_0007/container_1499804611229_0007_01_000001/stderr" hadoop_shell_errorcode=$? if [ $hadoop_shell_errorcode -ne 0 ] then exit $hadoop_shell_errorcode fi End of LogType:launch_container.sh LogType:stderr Log Upload Time:Wed Jul 12 19:07:15 -0500 2017 LogLength:18503 Log Contents: SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/hadoop/yarn/local/filecache/10/spark2-hdp-yarn-archive.tar.gz/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.6.1.0-129/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 17/07/12 19:04:45 INFO SignalUtils: Registered signal handler for TERM 17/07/12 19:04:45 INFO SignalUtils: Registered signal handler for HUP 17/07/12 19:04:45 INFO SignalUtils: Registered signal handler for INT 17/07/12 19:04:45 INFO ApplicationMaster: Preparing Local resources 17/07/12 19:04:46 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1499804611229_0007_000001 17/07/12 19:04:46 INFO SecurityManager: Changing view acls to: yarn,elyra 17/07/12 19:04:46 INFO SecurityManager: Changing modify acls to: yarn,elyra 17/07/12 19:04:46 INFO SecurityManager: Changing view acls groups to: 17/07/12 19:04:46 INFO SecurityManager: Changing modify acls groups to: 17/07/12 19:04:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, elyra); groups with view permissions: Set(); users with modify permissions: Set(yarn, elyra); groups with modify permissions: Set() 17/07/12 19:04:46 INFO ApplicationMaster: Starting the user application in a separate Thread 17/07/12 19:04:46 INFO ApplicationMaster: Waiting for spark context initialization... 17/07/12 19:04:46 INFO SparkContext: Running Spark version 2.1.1.2.6.1.0-129 17/07/12 19:04:46 INFO SecurityManager: Changing view acls to: yarn,elyra 17/07/12 19:04:46 INFO SecurityManager: Changing modify acls to: yarn,elyra 17/07/12 19:04:46 INFO SecurityManager: Changing view acls groups to: 17/07/12 19:04:46 INFO SecurityManager: Changing modify acls groups to: 17/07/12 19:04:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, elyra); groups with view permissions: Set(); users with modify permissions: Set(yarn, elyra); groups with modify permissions: Set() 17/07/12 19:04:46 INFO Utils: Successfully started service 'sparkDriver' on port 44474. 17/07/12 19:04:46 INFO SparkEnv: Registering MapOutputTracker 17/07/12 19:04:46 INFO SparkEnv: Registering BlockManagerMaster 17/07/12 19:04:46 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 17/07/12 19:04:46 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 17/07/12 19:04:46 INFO DiskBlockManager: Created local directory at /hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007/blockmgr-f1eb9ce2-6fab-46df-a282-7f16c7a58bb3 17/07/12 19:04:46 INFO MemoryStore: MemoryStore started with capacity 366.3 MB 17/07/12 19:04:46 INFO SparkEnv: Registering OutputCommitCoordinator 17/07/12 19:04:47 INFO log: Logging initialized @2391ms 17/07/12 19:04:47 INFO JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 17/07/12 19:04:47 INFO Server: jetty-9.2.z-SNAPSHOT 17/07/12 19:04:47 INFO Server: Started @2467ms 17/07/12 19:04:47 INFO ServerConnector: Started ServerConnector@12123286{HTTP/1.1}{0.0.0.0:32912} 17/07/12 19:04:47 INFO Utils: Successfully started service 'SparkUI' on port 32912. 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6b81e31b{/jobs,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@64a9a5fc{/jobs/json,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1c08a30a{/jobs/job,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4f11fead{/jobs/job/json,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1dea8255{/stages,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1bef58c{/stages/json,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1ea46424{/stages/stage,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@3a01e2d4{/stages/stage/json,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@68043d7b{/stages/pool,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@13cb69c8{/stages/pool/json,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@43f38da6{/storage,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@647c210c{/storage/json,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@1dfd66ab{/storage/rdd,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@385ee5c7{/storage/rdd/json,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@6e1321de{/environment,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@62f93dae{/environment/json,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@636ad494{/executors,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@51a21520{/executors/json,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@701d3f91{/executors/threadDump,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@21c6f1e5{/executors/threadDump/json,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@4947c454{/static,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@30f2f77c{/,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@58e1d40d{/api,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@2bcb24f4{/jobs/job/kill,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@427063a7{/stages/stage/kill,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://169.45.103.145:32912 17/07/12 19:04:47 INFO YarnClusterScheduler: Created YarnClusterScheduler 17/07/12 19:04:47 INFO SchedulerExtensionServices: Starting Yarn extension services with app application_1499804611229_0007 and attemptId Some(appattempt_1499804611229_0007_000001) 17/07/12 19:04:47 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37275. 17/07/12 19:04:47 INFO NettyBlockTransferService: Server created on 169.45.103.145:37275 17/07/12 19:04:47 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 17/07/12 19:04:47 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 169.45.103.145, 37275, None) 17/07/12 19:04:47 INFO BlockManagerMasterEndpoint: Registering block manager 169.45.103.145:37275 with 366.3 MB RAM, BlockManagerId(driver, 169.45.103.145, 37275, None) 17/07/12 19:04:47 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 169.45.103.145, 37275, None) 17/07/12 19:04:47 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 169.45.103.145, 37275, None) 17/07/12 19:04:47 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@507572d3{/metrics/json,null,AVAILABLE,@Spark} 17/07/12 19:04:47 INFO EventLoggingListener: Logging events to hdfs:///spark2-history/application_1499804611229_0007_1 17/07/12 19:04:47 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark://YarnAM@169.45.103.145:44474) 17/07/12 19:04:47 INFO ApplicationMaster: =============================================================================== YARN executor launch context: env: CLASSPATH -> {{PWD}}{{PWD}}/__spark_conf__{{PWD}}/__spark_libs__/*/etc/hadoop/conf/usr/hdp/current/hadoop-client/*/usr/hdp/current/hadoop-client/lib/*/usr/hdp/current/hadoop-hdfs-client/*/usr/hdp/current/hadoop-hdfs-client/lib/*/usr/hdp/current/hadoop-yarn-client/*/usr/hdp/current/hadoop-yarn-client/lib/*$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:$PWD/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.1.0-129/hadoop/lib/hadoop-lzo-0.6.0.2.6.1.0-129.jar:/etc/hadoop/conf/secure SPARK_YARN_STAGING_DIR -> hdfs://spark-02.softlayer.com:8020/user/elyra/.sparkStaging/application_1499804611229_0007 SPARK_USER -> elyra SPARK_YARN_MODE -> true command: LD_LIBRARY_PATH="/usr/hdp/current/hadoop-client/lib/native:/usr/hdp/current/hadoop-client/lib/native/Linux-amd64-64:$LD_LIBRARY_PATH" \ {{JAVA_HOME}}/bin/java \ -server \ -Xmx1024m \ -Djava.io.tmpdir={{PWD}}/tmp \ '-Dspark.history.ui.port=18081' \ -Dspark.yarn.app.container.log.dir= \ -XX:OnOutOfMemoryError='kill %p' \ org.apache.spark.executor.CoarseGrainedExecutorBackend \ --driver-url \ spark://CoarseGrainedScheduler@169.45.103.145:44474 \ --executor-id \ \ --hostname \ \ --cores \ 1 \ --app-id \ application_1499804611229_0007 \ --user-class-path \ file:$PWD/__app__.jar \ --user-class-path \ file:$PWD/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar \ 1>/stdout \ 2>/stderr resources: __app__.jar -> resource { scheme: "hdfs" host: "spark-02.softlayer.com" port: 8020 file: "/user/elyra/.sparkStaging/application_1499804611229_0007/toree-launcher_2.11-1.jar" } size: 14747 timestamp: 1499904283798 type: FILE visibility: PRIVATE __spark_libs__ -> resource { scheme: "hdfs" host: "spark-02.softlayer.com" port: 8020 file: "/hdp/apps/2.6.1.0-129/spark2/spark2-hdp-yarn-archive.tar.gz" } size: 180371223 timestamp: 1499723886531 type: ARCHIVE visibility: PUBLIC __spark_conf__ -> resource { scheme: "hdfs" host: "spark-02.softlayer.com" port: 8020 file: "/user/elyra/.sparkStaging/application_1499804611229_0007/__spark_conf__.zip" } size: 109450 timestamp: 1499904283968 type: ARCHIVE visibility: PRIVATE toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar -> resource { scheme: "hdfs" host: "spark-02.softlayer.com" port: 8020 file: "/user/elyra/.sparkStaging/application_1499804611229_0007/toree-assembly-0.2.0.dev1-incubating-SNAPSHOT.jar" } size: 27591359 timestamp: 1499904283911 type: FILE visibility: PRIVATE =============================================================================== 17/07/12 19:04:47 INFO RMProxy: Connecting to ResourceManager at spark-02.softlayer.com/169.45.103.134:8030 17/07/12 19:04:47 INFO YarnRMClient: Registering the ApplicationMaster 17/07/12 19:04:47 INFO YarnAllocator: Will request 2 executor container(s), each with 1 core(s) and 1408 MB memory (including 384 MB of overhead) 17/07/12 19:04:47 INFO YarnAllocator: Submitted 2 unlocalized container requests. 17/07/12 19:04:47 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals 17/07/12 19:04:47 INFO AMRMClientImpl: Received new token for : spark-02.softlayer.com:45454 17/07/12 19:04:47 INFO YarnAllocator: Launching container container_1499804611229_0007_01_000002 on host spark-02.softlayer.com 17/07/12 19:04:47 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. 17/07/12 19:04:47 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 17/07/12 19:04:47 INFO ContainerManagementProtocolProxy: Opening proxy : spark-02.softlayer.com:45454 17/07/12 19:04:48 INFO AMRMClientImpl: Received new token for : spark-04.softlayer.com:45454 17/07/12 19:04:48 INFO YarnAllocator: Launching container container_1499804611229_0007_01_000003 on host spark-04.softlayer.com 17/07/12 19:04:48 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them. 17/07/12 19:04:48 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0 17/07/12 19:04:48 INFO ContainerManagementProtocolProxy: Opening proxy : spark-04.softlayer.com:45454 17/07/12 19:04:50 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (169.45.103.134:45877) with ID 1 17/07/12 19:04:50 INFO BlockManagerMasterEndpoint: Registering block manager spark-02.softlayer.com:53007 with 366.3 MB RAM, BlockManagerId(1, spark-02.softlayer.com, 53007, None) 17/07/12 19:04:50 INFO YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (169.45.103.145:53554) with ID 2 17/07/12 19:04:50 INFO YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 17/07/12 19:04:50 INFO YarnClusterScheduler: YarnClusterScheduler.postStartHook done 17/07/12 19:04:50 INFO BlockManagerMasterEndpoint: Registering block manager spark-04.softlayer.com:40705 with 366.3 MB RAM, BlockManagerId(2, spark-04.softlayer.com, 40705, None) 17/07/12 19:04:50 INFO Main$$anon$1: Kernel version: 0.2.0.dev1-incubating-SNAPSHOT 17/07/12 19:04:50 INFO Main$$anon$1: Scala version: 2.11.8 17/07/12 19:04:50 INFO Main$$anon$1: ZeroMQ (JeroMQ) version: 3.2.5 17/07/12 19:04:50 INFO Main$$anon$1: Initializing internal actor system 17/07/12 19:04:51 INFO Slf4jLogger: Slf4jLogger started 17/07/12 19:04:51 INFO Main$$anon$1: Connection Profile: { "stdin_port" : 38724, "control_port" : 41509, "hb_port" : 35500, "shell_port" : 39864, "iopub_port" : 39681, "ip" : "0.0.0.0", "transport" : "tcp", "signature_scheme" : "hmac-sha256", "key" : "c39381e5-6a1b-4020-a9d9-325196da7a31" } 17/07/12 19:04:51 INFO EmptyLocalActorRef: Message [org.apache.toree.kernel.protocol.v5.KernelMessage] from Actor[akka://spark-kernel-actor-system/user/kernel_message_relay#-829891308] to Actor[akka://spark-kernel-actor-system/user/status] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. 17/07/12 19:04:52 WARN Main$$anon$1: No external magics provided to PluginManager! 17/07/12 19:04:54 INFO Main$$anon$1: 13 internal plugins loaded 17/07/12 19:04:54 INFO Main$$anon$1: 0 external plugins loaded 17/07/12 19:04:55 INFO KernelMessageRelay: Not ready for messages! Stashing until ready! 17/07/12 19:04:56 WARN ScalaInterpreter: kernel variable: org.apache.toree.boot.layer.StandardComponentInitialization$$anon$1@31f6e08b 17/07/12 19:04:56 WARN ScalaInterpreter: Binding List(@transient implicit) kernel org.apache.toree.kernel.api.Kernel org.apache.toree.boot.layer.StandardComponentInitialization$$anon$1@31f6e08b 17/07/12 19:04:57 INFO ScalaInterpreter: Binding SparkContext into interpreter as sc 17/07/12 19:04:58 INFO Main$$anon$1: Marking relay as ready for receiving messages 17/07/12 19:04:58 INFO KernelMessageRelay: Unstashing all messages received! 17/07/12 19:04:58 INFO KernelMessageRelay: Relay is now fully ready to receive messages! 17/07/12 19:07:13 INFO Main$$anon$1: Shutting down kernel 17/07/12 19:07:13 INFO Main$$anon$1: Shutting down interpreters 17/07/12 19:07:13 INFO ScalaInterpreter: Shutting down interpreter 17/07/12 19:07:13 INFO SparkContext: Invoking stop() from shutdown hook 17/07/12 19:07:13 INFO Main$$anon$1: Shutting down actor system 17/07/12 19:07:13 INFO ServerConnector: Stopped Spark@12123286{HTTP/1.1}{0.0.0.0:0} 17/07/12 19:07:13 INFO SparkUI: Stopped Spark web UI at http://169.45.103.145:32912 17/07/12 19:07:13 INFO YarnAllocator: Driver requested a total number of 0 executor(s). 17/07/12 19:07:13 INFO YarnClusterSchedulerBackend: Shutting down all executors 17/07/12 19:07:13 INFO YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down 17/07/12 19:07:13 INFO SchedulerExtensionServices: Stopping SchedulerExtensionServices (serviceOption=None, services=List(), started=false) 17/07/12 19:07:13 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/07/12 19:07:13 INFO Main$$anon$1: Shutting down interpreters 17/07/12 19:07:13 INFO ScalaInterpreter: Shutting down interpreter 17/07/12 19:07:13 INFO Main$$anon$1: Shutting down actor system 17/07/12 19:07:13 INFO SparkContext: SparkContext already stopped. 17/07/12 19:07:13 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0 17/07/12 19:07:13 INFO MemoryStore: MemoryStore cleared 17/07/12 19:07:13 INFO BlockManager: BlockManager stopped 17/07/12 19:07:13 INFO BlockManagerMaster: BlockManagerMaster stopped 17/07/12 19:07:13 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/07/12 19:07:13 INFO SparkContext: Successfully stopped SparkContext 17/07/12 19:07:13 INFO ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED 17/07/12 19:07:13 INFO AMRMClientImpl: Waiting for application to be successfully unregistered. 17/07/12 19:07:13 INFO ApplicationMaster: Deleting staging directory hdfs://spark-02.softlayer.com:8020/user/elyra/.sparkStaging/application_1499804611229_0007 17/07/12 19:07:13 INFO ShutdownHookManager: Shutdown hook called 17/07/12 19:07:13 INFO ShutdownHookManager: Deleting directory /hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007/spark-981df3fe-fb3e-4ee5-a24c-91129552e921/repl-5b0e1726-0075-4714-8ec5-7da2dc87eb07 17/07/12 19:07:13 INFO ShutdownHookManager: Deleting directory /hadoop/yarn/local/usercache/elyra/appcache/application_1499804611229_0007/spark-981df3fe-fb3e-4ee5-a24c-91129552e921 End of LogType:stderr LogType:stdout Log Upload Time:Wed Jul 12 19:07:15 -0500 2017 LogLength:1158 Log Contents: Now creating a Spark Session... --Toree launcher arguments: --response-address ERROR__NO__KERNEL_RESPONSE_ADDRESS --profile /var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json --------------------------- The profile /var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json doesn't exist, now creating it... /var/run/elyra/runtime/kernel-c1081493-a48b-43df-ad65-c7adeb835b7e.json saved JSON connection content: { "hb_port" : 35500, "control_port" : 41509, "iopub_port" : 39681, "stdin_port" : 38724, "shell_port" : 39864, "key" : "c39381e5-6a1b-4020-a9d9-325196da7a31", "kernel_name" : "Apache Toree Scala", "signature_scheme" : "hmac-sha256", "transport" : "tcp", "ip" : "0.0.0.0" } Socket address ERROR__NO__KERNEL_RESPONSE_ADDRESS not in format, abort writing to socket. =========End (Scala,org.apache.toree.kernel.interpreter.scala.ScalaInterpreter@36385529) (PySpark,org.apache.toree.kernel.interpreter.pyspark.PySparkInterpreter@1fb58a6c) (SparkR,org.apache.toree.kernel.interpreter.sparkr.SparkRInterpreter@35acf211) (SQL,org.apache.toree.kernel.interpreter.sql.SqlInterpreter@4c31f759) End of LogType:stdout [elyra@spark-02 root]$