yarn logs -applicationId application_1682065044112_0121 WARNING: YARN_OPTS has been replaced by HADOOP_OPTS. Using value of YARN_OPTS. 23/10/28 23:40:24 INFO client.RMProxy: Connecting to ResourceManager at node274-ib.ara/192.168.210.19:8032 Container: container_1682065044112_0121_01_000001 on node274-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:application.master.log LogLastModifiedTime:Sat Oct 28 23:28:12 +0200 2023 LogLength:2926 LogContents: 23/10/28 23:28:06 INFO skein.ApplicationMaster: Starting Skein version 0.8.2 23/10/28 23:28:07 INFO skein.ApplicationMaster: Running as user myusername 23/10/28 23:28:07 INFO conf.Configuration: resource-types.xml not found 23/10/28 23:28:07 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'. 23/10/28 23:28:07 INFO skein.ApplicationMaster: Application specification successfully loaded 23/10/28 23:28:07 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 23/10/28 23:28:08 INFO client.RMProxy: Connecting to ResourceManager at node274-ib.ara/192.168.210.19:8030 23/10/28 23:28:08 INFO skein.ApplicationMaster: gRPC server started at node274-ib.ara:44547 23/10/28 23:28:08 INFO skein.ApplicationMaster: WebUI server started at node274-ib.ara:43581 23/10/28 23:28:08 INFO skein.ApplicationMaster: Registering application with resource manager 23/10/28 23:28:09 INFO client.RMProxy: Connecting to ResourceManager at node274-ib.ara/192.168.210.19:8032 23/10/28 23:28:09 INFO skein.ApplicationMaster: Initializing service 'dask.worker'. 23/10/28 23:28:09 INFO skein.ApplicationMaster: WAITING: dask.worker_0 23/10/28 23:28:09 INFO skein.ApplicationMaster: WAITING: dask.worker_1 23/10/28 23:28:09 INFO skein.ApplicationMaster: WAITING: dask.worker_2 23/10/28 23:28:09 INFO skein.ApplicationMaster: WAITING: dask.worker_3 23/10/28 23:28:09 INFO skein.ApplicationMaster: Initializing service 'dask.scheduler'. 23/10/28 23:28:09 INFO skein.ApplicationMaster: REQUESTED: dask.scheduler_0 23/10/28 23:28:10 INFO skein.ApplicationMaster: Starting container_1682065044112_0121_01_000002... 23/10/28 23:28:10 INFO skein.ApplicationMaster: RUNNING: dask.scheduler_0 on container_1682065044112_0121_01_000002 23/10/28 23:28:10 INFO skein.ApplicationMaster: REQUESTED: dask.worker_0 23/10/28 23:28:10 INFO skein.ApplicationMaster: REQUESTED: dask.worker_1 23/10/28 23:28:10 INFO skein.ApplicationMaster: REQUESTED: dask.worker_2 23/10/28 23:28:10 INFO skein.ApplicationMaster: REQUESTED: dask.worker_3 23/10/28 23:28:12 INFO skein.ApplicationMaster: Starting container_1682065044112_0121_01_000003... 23/10/28 23:28:12 INFO skein.ApplicationMaster: RUNNING: dask.worker_0 on container_1682065044112_0121_01_000003 23/10/28 23:28:12 INFO skein.ApplicationMaster: Starting container_1682065044112_0121_01_000004... 23/10/28 23:28:12 INFO skein.ApplicationMaster: RUNNING: dask.worker_1 on container_1682065044112_0121_01_000004 23/10/28 23:28:12 INFO skein.ApplicationMaster: Starting container_1682065044112_0121_01_000005... 23/10/28 23:28:12 INFO skein.ApplicationMaster: RUNNING: dask.worker_2 on container_1682065044112_0121_01_000005 23/10/28 23:28:12 INFO skein.ApplicationMaster: Starting container_1682065044112_0121_01_000006... 23/10/28 23:28:12 INFO skein.ApplicationMaster: RUNNING: dask.worker_3 on container_1682065044112_0121_01_000006 End of LogType:application.master.log.This log file belongs to a running container (container_1682065044112_0121_01_000001) and so may not be complete. *************************************************************************************** End of LogType:prelaunch.err.This log file belongs to a running container (container_1682065044112_0121_01_000001) and so may not be complete. ****************************************************************************** Container: container_1682065044112_0121_01_000001 on node274-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:prelaunch.out LogLastModifiedTime:Sat Oct 28 23:28:06 +0200 2023 LogLength:70 LogContents: Setting up env variables Setting up job resources Launching container End of LogType:prelaunch.out.This log file belongs to a running container (container_1682065044112_0121_01_000001) and so may not be complete. ****************************************************************************** End of LogType:prelaunch.err.This log file belongs to a running container (container_1682065044112_0121_01_000002) and so may not be complete. ****************************************************************************** Container: container_1682065044112_0121_01_000002 on node280-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:dask.scheduler.log LogLastModifiedTime:Sat Oct 28 23:28:17 +0200 2023 LogLength:3377 LogContents: /hdfs/mnt1/yarn/nm/usercache/myusername/appcache/application_1682065044112_0121/container_1682065044112_0121_01_000002/environment/lib/python3.8/site-packages/dask_yarn/core.py:16: FutureWarning: format_bytes is deprecated and will be removed in a future release. Please use dask.utils.format_bytes instead. from distributed.utils import ( /hdfs/mnt1/yarn/nm/usercache/myusername/appcache/application_1682065044112_0121/container_1682065044112_0121_01_000002/environment/lib/python3.8/site-packages/dask_yarn/core.py:16: FutureWarning: parse_timedelta is deprecated and will be removed in a future release. Please use dask.utils.parse_timedelta instead. from distributed.utils import ( /hdfs/mnt1/yarn/nm/usercache/myusername/appcache/application_1682065044112_0121/container_1682065044112_0121_01_000002/environment/lib/python3.8/site-packages/dask_yarn/cli.py:16: DeprecationWarning: the distributed.cli.utils module is deprecated from distributed.cli.utils import install_signal_handlers /hdfs/mnt1/yarn/nm/usercache/myusername/appcache/application_1682065044112_0121/container_1682065044112_0121_01_000002/environment/lib/python3.8/site-packages/dask_yarn/cli.py:398: DeprecationWarning: the loop kwarg to Scheduler is deprecated scheduler = Scheduler(loop=loop, dashboard_address=("", 0)) 2023-10-28 23:28:15,490 - distributed.http.proxy - INFO - To route to workers diagnostics web server please install jupyter-server-proxy: python -m pip install jupyter-server-proxy 2023-10-28 23:28:15,519 - distributed.scheduler - INFO - State start 2023-10-28 23:28:15,522 - distributed.scheduler - INFO - Scheduler at: tcp://192.168.194.25:43505 2023-10-28 23:28:15,523 - distributed.scheduler - INFO - dashboard at: http://192.168.194.25:37326/status 2023-10-28 23:28:16,790 - distributed.scheduler - INFO - Register worker 2023-10-28 23:28:17,043 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.194.25:33219 2023-10-28 23:28:17,043 - distributed.core - INFO - Starting established connection to tcp://192.168.194.25:48852 2023-10-28 23:28:17,044 - distributed.scheduler - INFO - Register worker 2023-10-28 23:28:17,044 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.194.25:34246 2023-10-28 23:28:17,044 - distributed.core - INFO - Starting established connection to tcp://192.168.194.25:48850 2023-10-28 23:28:17,045 - distributed.scheduler - INFO - Register worker 2023-10-28 23:28:17,045 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.194.25:45044 2023-10-28 23:28:17,045 - distributed.core - INFO - Starting established connection to tcp://192.168.194.25:48854 2023-10-28 23:28:17,045 - distributed.scheduler - INFO - Register worker 2023-10-28 23:28:17,046 - distributed.scheduler - INFO - Starting worker compute stream, tcp://192.168.194.25:39089 2023-10-28 23:28:17,046 - distributed.core - INFO - Starting established connection to tcp://192.168.194.25:48856 End of LogType:dask.scheduler.log.This log file belongs to a running container (container_1682065044112_0121_01_000002) and so may not be complete. *********************************************************************************** Container: container_1682065044112_0121_01_000002 on node280-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:prelaunch.out LogLastModifiedTime:Sat Oct 28 23:28:14 +0200 2023 LogLength:70 LogContents: Setting up env variables Setting up job resources Launching container End of LogType:prelaunch.out.This log file belongs to a running container (container_1682065044112_0121_01_000002) and so may not be complete. ****************************************************************************** End of LogType:prelaunch.err.This log file belongs to a running container (container_1682065044112_0121_01_000003) and so may not be complete. ****************************************************************************** Container: container_1682065044112_0121_01_000003 on node280-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:dask.worker.log LogLastModifiedTime:Sat Oct 28 23:28:17 +0200 2023 LogLength:1481 LogContents: 2023-10-28 23:28:16,261 - distributed.nanny - INFO - Start Nanny at: 'tcp://192.168.194.25:42913' 2023-10-28 23:28:16,636 - distributed.worker - INFO - Start worker at: tcp://192.168.194.25:39089 2023-10-28 23:28:16,636 - distributed.worker - INFO - Listening to: tcp://192.168.194.25:39089 2023-10-28 23:28:16,636 - distributed.worker - INFO - Worker name: dask.worker_0 2023-10-28 23:28:16,636 - distributed.worker - INFO - dashboard at: 192.168.194.25:37003 2023-10-28 23:28:16,636 - distributed.worker - INFO - Waiting to connect to: tcp://192.168.194.25:43505 2023-10-28 23:28:16,636 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:16,636 - distributed.worker - INFO - Threads: 2 2023-10-28 23:28:16,636 - distributed.worker - INFO - Memory: 3.73 GiB 2023-10-28 23:28:16,636 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-bzltam8h 2023-10-28 23:28:16,636 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:17,047 - distributed.worker - INFO - Registered to: tcp://192.168.194.25:43505 2023-10-28 23:28:17,047 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:17,049 - distributed.core - INFO - Starting established connection to tcp://192.168.194.25:43505 End of LogType:dask.worker.log.This log file belongs to a running container (container_1682065044112_0121_01_000003) and so may not be complete. ******************************************************************************** Container: container_1682065044112_0121_01_000003 on node280-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:prelaunch.out LogLastModifiedTime:Sat Oct 28 23:28:14 +0200 2023 LogLength:70 LogContents: Setting up env variables Setting up job resources Launching container End of LogType:prelaunch.out.This log file belongs to a running container (container_1682065044112_0121_01_000003) and so may not be complete. ****************************************************************************** End of LogType:prelaunch.err.This log file belongs to a running container (container_1682065044112_0121_01_000004) and so may not be complete. ****************************************************************************** Container: container_1682065044112_0121_01_000004 on node280-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:dask.worker.log LogLastModifiedTime:Sat Oct 28 23:28:17 +0200 2023 LogLength:1481 LogContents: 2023-10-28 23:28:16,231 - distributed.nanny - INFO - Start Nanny at: 'tcp://192.168.194.25:35504' 2023-10-28 23:28:16,609 - distributed.worker - INFO - Start worker at: tcp://192.168.194.25:45044 2023-10-28 23:28:16,609 - distributed.worker - INFO - Listening to: tcp://192.168.194.25:45044 2023-10-28 23:28:16,609 - distributed.worker - INFO - Worker name: dask.worker_1 2023-10-28 23:28:16,609 - distributed.worker - INFO - dashboard at: 192.168.194.25:39786 2023-10-28 23:28:16,609 - distributed.worker - INFO - Waiting to connect to: tcp://192.168.194.25:43505 2023-10-28 23:28:16,609 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:16,609 - distributed.worker - INFO - Threads: 2 2023-10-28 23:28:16,609 - distributed.worker - INFO - Memory: 3.73 GiB 2023-10-28 23:28:16,609 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-676fpcvm 2023-10-28 23:28:16,609 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:17,045 - distributed.worker - INFO - Registered to: tcp://192.168.194.25:43505 2023-10-28 23:28:17,046 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:17,047 - distributed.core - INFO - Starting established connection to tcp://192.168.194.25:43505 End of LogType:dask.worker.log.This log file belongs to a running container (container_1682065044112_0121_01_000004) and so may not be complete. ******************************************************************************** Container: container_1682065044112_0121_01_000004 on node280-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:prelaunch.out LogLastModifiedTime:Sat Oct 28 23:28:14 +0200 2023 LogLength:70 LogContents: Setting up env variables Setting up job resources Launching container End of LogType:prelaunch.out.This log file belongs to a running container (container_1682065044112_0121_01_000004) and so may not be complete. ****************************************************************************** End of LogType:prelaunch.err.This log file belongs to a running container (container_1682065044112_0121_01_000005) and so may not be complete. ****************************************************************************** Container: container_1682065044112_0121_01_000005 on node280-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:dask.worker.log LogLastModifiedTime:Sat Oct 28 23:28:17 +0200 2023 LogLength:1481 LogContents: 2023-10-28 23:28:16,193 - distributed.nanny - INFO - Start Nanny at: 'tcp://192.168.194.25:35987' 2023-10-28 23:28:16,575 - distributed.worker - INFO - Start worker at: tcp://192.168.194.25:34246 2023-10-28 23:28:16,575 - distributed.worker - INFO - Listening to: tcp://192.168.194.25:34246 2023-10-28 23:28:16,575 - distributed.worker - INFO - Worker name: dask.worker_2 2023-10-28 23:28:16,575 - distributed.worker - INFO - dashboard at: 192.168.194.25:43681 2023-10-28 23:28:16,575 - distributed.worker - INFO - Waiting to connect to: tcp://192.168.194.25:43505 2023-10-28 23:28:16,575 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:16,575 - distributed.worker - INFO - Threads: 2 2023-10-28 23:28:16,575 - distributed.worker - INFO - Memory: 3.73 GiB 2023-10-28 23:28:16,575 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-j7h9g52r 2023-10-28 23:28:16,575 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:17,045 - distributed.worker - INFO - Registered to: tcp://192.168.194.25:43505 2023-10-28 23:28:17,045 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:17,047 - distributed.core - INFO - Starting established connection to tcp://192.168.194.25:43505 End of LogType:dask.worker.log.This log file belongs to a running container (container_1682065044112_0121_01_000005) and so may not be complete. ******************************************************************************** Container: container_1682065044112_0121_01_000005 on node280-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:prelaunch.out LogLastModifiedTime:Sat Oct 28 23:28:14 +0200 2023 LogLength:70 LogContents: Setting up env variables Setting up job resources Launching container End of LogType:prelaunch.out.This log file belongs to a running container (container_1682065044112_0121_01_000005) and so may not be complete. ****************************************************************************** End of LogType:prelaunch.err.This log file belongs to a running container (container_1682065044112_0121_01_000006) and so may not be complete. ****************************************************************************** Container: container_1682065044112_0121_01_000006 on node280-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:dask.worker.log LogLastModifiedTime:Sat Oct 28 23:28:17 +0200 2023 LogLength:1481 LogContents: 2023-10-28 23:28:16,202 - distributed.nanny - INFO - Start Nanny at: 'tcp://192.168.194.25:46649' 2023-10-28 23:28:16,579 - distributed.worker - INFO - Start worker at: tcp://192.168.194.25:33219 2023-10-28 23:28:16,579 - distributed.worker - INFO - Listening to: tcp://192.168.194.25:33219 2023-10-28 23:28:16,579 - distributed.worker - INFO - Worker name: dask.worker_3 2023-10-28 23:28:16,579 - distributed.worker - INFO - dashboard at: 192.168.194.25:45964 2023-10-28 23:28:16,579 - distributed.worker - INFO - Waiting to connect to: tcp://192.168.194.25:43505 2023-10-28 23:28:16,579 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:16,579 - distributed.worker - INFO - Threads: 2 2023-10-28 23:28:16,579 - distributed.worker - INFO - Memory: 3.73 GiB 2023-10-28 23:28:16,579 - distributed.worker - INFO - Local Directory: /tmp/dask-scratch-space/worker-swyvcctq 2023-10-28 23:28:16,579 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:17,043 - distributed.worker - INFO - Registered to: tcp://192.168.194.25:43505 2023-10-28 23:28:17,044 - distributed.worker - INFO - ------------------------------------------------- 2023-10-28 23:28:17,046 - distributed.core - INFO - Starting established connection to tcp://192.168.194.25:43505 End of LogType:dask.worker.log.This log file belongs to a running container (container_1682065044112_0121_01_000006) and so may not be complete. ******************************************************************************** Container: container_1682065044112_0121_01_000006 on node280-ib.ara:8041 LogAggregationType: LOCAL ======================================================================== LogType:prelaunch.out LogLastModifiedTime:Sat Oct 28 23:28:14 +0200 2023 LogLength:70 LogContents: Setting up env variables Setting up job resources Launching container End of LogType:prelaunch.out.This log file belongs to a running container (container_1682065044112_0121_01_000006) and so may not be complete. ******************************************************************************