Permalink
Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
147 lines (145 sloc) 16.1 KB
my@octopus:~/dev/git/big-lapin/best-lapin-cretin-spark$ spark-submit --class fr.soat.LapinLePlusCretinWithSparkCluster --master spark://192.168.1.11:7077 target/best-lapin-cretin-spark-1.0-SNAPSHOT.jar
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
17/04/28 21:49:50 INFO SparkContext: Running Spark version 1.6.1
17/04/28 21:49:50 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/04/28 21:49:51 WARN Utils: Your hostname, octopus resolves to a loopback address: 127.0.1.1; using 192.168.1.11 instead (on interface enp7s0)
17/04/28 21:49:51 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
17/04/28 21:49:51 INFO SecurityManager: Changing view acls to: my
17/04/28 21:49:51 INFO SecurityManager: Changing modify acls to: my
17/04/28 21:49:51 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(my); users with modify permissions: Set(my)
17/04/28 21:49:51 INFO Utils: Successfully started service 'sparkDriver' on port 40164.
17/04/28 21:49:51 INFO Slf4jLogger: Slf4jLogger started
17/04/28 21:49:51 INFO Remoting: Starting remoting
17/04/28 21:49:51 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@192.168.1.11:41688]
17/04/28 21:49:51 INFO Utils: Successfully started service 'sparkDriverActorSystem' on port 41688.
17/04/28 21:49:51 INFO SparkEnv: Registering MapOutputTracker
17/04/28 21:49:51 INFO SparkEnv: Registering BlockManagerMaster
17/04/28 21:49:51 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-ca943b65-36b5-46f5-85fb-38b89eed3dd2
17/04/28 21:49:51 INFO MemoryStore: MemoryStore started with capacity 511.1 MB
17/04/28 21:49:52 INFO SparkEnv: Registering OutputCommitCoordinator
17/04/28 21:49:52 INFO Utils: Successfully started service 'SparkUI' on port 4040.
17/04/28 21:49:52 INFO SparkUI: Started SparkUI at http://192.168.1.11:4040
17/04/28 21:49:52 INFO HttpFileServer: HTTP File server directory is /tmp/spark-d9ff765c-8b0b-428f-83a4-59ca7cab757d/httpd-06f891f7-2241-41cf-8038-e3aed6fcb52c
17/04/28 21:49:52 INFO HttpServer: Starting HTTP Server
17/04/28 21:49:52 INFO Utils: Successfully started service 'HTTP file server' on port 45346.
17/04/28 21:49:52 INFO SparkContext: Added JAR file:/home/my/dev/git/big-lapin/best-lapin-cretin-spark/target/best-lapin-cretin-spark-1.0-SNAPSHOT.jar at http://192.168.1.11:45346/jars/best-lapin-cretin-spark-1.0-SNAPSHOT.jar with timestamp 1493408992235
17/04/28 21:49:52 INFO AppClient$ClientEndpoint: Connecting to master spark://192.168.1.11:7077...
17/04/28 21:49:52 INFO SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20170428214952-0000
17/04/28 21:49:52 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40011.
17/04/28 21:49:52 INFO NettyBlockTransferService: Server created on 40011
17/04/28 21:49:52 INFO BlockManagerMaster: Trying to register BlockManager
17/04/28 21:49:52 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.11:40011 with 511.1 MB RAM, BlockManagerId(driver, 192.168.1.11, 40011)
17/04/28 21:49:52 INFO BlockManagerMaster: Registered BlockManager
17/04/28 21:49:52 INFO AppClient$ClientEndpoint: Executor added: app-20170428214952-0000/0 on worker-20170428214752-192.168.1.11-44073 (192.168.1.11:44073) with 4 cores
17/04/28 21:49:52 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170428214952-0000/0 on hostPort 192.168.1.11:44073 with 4 cores, 1024.0 MB RAM
17/04/28 21:49:52 INFO AppClient$ClientEndpoint: Executor added: app-20170428214952-0000/1 on worker-20170428214837-192.168.1.13-35449 (192.168.1.13:35449) with 4 cores
17/04/28 21:49:52 INFO SparkDeploySchedulerBackend: Granted executor ID app-20170428214952-0000/1 on hostPort 192.168.1.13:35449 with 4 cores, 1024.0 MB RAM
17/04/28 21:49:52 INFO AppClient$ClientEndpoint: Executor updated: app-20170428214952-0000/0 is now RUNNING
17/04/28 21:49:52 INFO AppClient$ClientEndpoint: Executor updated: app-20170428214952-0000/1 is now RUNNING
17/04/28 21:49:52 INFO SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
17/04/28 21:49:53 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 127.4 KB, free 127.4 KB)
17/04/28 21:49:53 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 13.9 KB, free 141.3 KB)
17/04/28 21:49:53 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.1.11:40011 (size: 13.9 KB, free: 511.1 MB)
17/04/28 21:49:53 INFO SparkContext: Created broadcast 0 from textFile at LapinLePlusCretinWithSparkCluster.java:87
17/04/28 21:49:53 INFO FileInputFormat: Total input paths to process : 1
17/04/28 21:49:53 INFO SparkContext: Starting job: reduce at LapinLePlusCretinWithSparkCluster.java:91
17/04/28 21:49:53 INFO DAGScheduler: Got job 0 (reduce at LapinLePlusCretinWithSparkCluster.java:91) with 29 output partitions
17/04/28 21:49:53 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at LapinLePlusCretinWithSparkCluster.java:91)
17/04/28 21:49:53 INFO DAGScheduler: Parents of final stage: List()
17/04/28 21:49:53 INFO DAGScheduler: Missing parents: List()
17/04/28 21:49:53 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at map at LapinLePlusCretinWithSparkCluster.java:90), which has no missing parents
17/04/28 21:49:53 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 4.5 KB, free 145.8 KB)
17/04/28 21:49:53 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.5 KB, free 148.3 KB)
17/04/28 21:49:53 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.1.11:40011 (size: 2.5 KB, free: 511.1 MB)
17/04/28 21:49:53 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
17/04/28 21:49:53 INFO DAGScheduler: Submitting 29 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at map at LapinLePlusCretinWithSparkCluster.java:90)
17/04/28 21:49:53 INFO TaskSchedulerImpl: Adding task set 0.0 with 29 tasks
17/04/28 21:49:54 INFO SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (octopus.home:44350) with ID 0
17/04/28 21:49:54 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, octopus.home, partition 0,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:54 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, octopus.home, partition 1,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:54 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, octopus.home, partition 2,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:54 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, octopus.home, partition 3,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:54 INFO BlockManagerMasterEndpoint: Registering block manager octopus.home:35029 with 511.1 MB RAM, BlockManagerId(0, octopus.home, 35029)
17/04/28 21:49:54 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on octopus.home:35029 (size: 2.5 KB, free: 511.1 MB)
17/04/28 21:49:54 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on octopus.home:35029 (size: 13.9 KB, free: 511.1 MB)
17/04/28 21:49:55 INFO SparkDeploySchedulerBackend: Registered executor NettyRpcEndpointRef(null) (kraken.home:41426) with ID 1
17/04/28 21:49:55 INFO TaskSetManager: Starting task 4.0 in stage 0.0 (TID 4, kraken.home, partition 4,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:55 INFO TaskSetManager: Starting task 5.0 in stage 0.0 (TID 5, kraken.home, partition 5,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:55 INFO TaskSetManager: Starting task 6.0 in stage 0.0 (TID 6, kraken.home, partition 6,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:55 INFO TaskSetManager: Starting task 7.0 in stage 0.0 (TID 7, kraken.home, partition 7,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:55 INFO BlockManagerMasterEndpoint: Registering block manager kraken.home:45371 with 511.1 MB RAM, BlockManagerId(1, kraken.home, 45371)
17/04/28 21:49:56 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on kraken.home:45371 (size: 2.5 KB, free: 511.1 MB)
17/04/28 21:49:56 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on kraken.home:45371 (size: 13.9 KB, free: 511.1 MB)
17/04/28 21:49:57 INFO TaskSetManager: Starting task 8.0 in stage 0.0 (TID 8, octopus.home, partition 8,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:58 INFO TaskSetManager: Starting task 9.0 in stage 0.0 (TID 9, octopus.home, partition 9,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:58 INFO TaskSetManager: Starting task 10.0 in stage 0.0 (TID 10, octopus.home, partition 10,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:58 INFO TaskSetManager: Finished task 3.0 in stage 0.0 (TID 3) in 3544 ms on octopus.home (1/29)
17/04/28 21:49:58 INFO TaskSetManager: Finished task 2.0 in stage 0.0 (TID 2) in 3550 ms on octopus.home (2/29)
17/04/28 21:49:58 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 3562 ms on octopus.home (3/29)
17/04/28 21:49:58 INFO TaskSetManager: Starting task 11.0 in stage 0.0 (TID 11, octopus.home, partition 11,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:58 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3580 ms on octopus.home (4/29)
17/04/28 21:49:58 INFO TaskSetManager: Starting task 12.0 in stage 0.0 (TID 12, kraken.home, partition 12,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:58 INFO TaskSetManager: Starting task 13.0 in stage 0.0 (TID 13, kraken.home, partition 13,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:49:58 INFO TaskSetManager: Finished task 7.0 in stage 0.0 (TID 7) in 3165 ms on kraken.home (5/29)
17/04/28 21:49:58 INFO TaskSetManager: Finished task 6.0 in stage 0.0 (TID 6) in 3166 ms on kraken.home (6/29)
17/04/28 21:50:00 INFO TaskSetManager: Starting task 14.0 in stage 0.0 (TID 14, kraken.home, partition 14,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:00 INFO TaskSetManager: Finished task 5.0 in stage 0.0 (TID 5) in 4369 ms on kraken.home (7/29)
17/04/28 21:50:00 INFO TaskSetManager: Starting task 15.0 in stage 0.0 (TID 15, kraken.home, partition 15,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:00 INFO TaskSetManager: Finished task 4.0 in stage 0.0 (TID 4) in 4555 ms on kraken.home (8/29)
17/04/28 21:50:00 INFO TaskSetManager: Starting task 16.0 in stage 0.0 (TID 16, octopus.home, partition 16,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:00 INFO TaskSetManager: Finished task 9.0 in stage 0.0 (TID 9) in 2679 ms on octopus.home (9/29)
17/04/28 21:50:00 INFO TaskSetManager: Starting task 17.0 in stage 0.0 (TID 17, octopus.home, partition 17,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:00 INFO TaskSetManager: Finished task 8.0 in stage 0.0 (TID 8) in 2693 ms on octopus.home (10/29)
17/04/28 21:50:00 INFO TaskSetManager: Starting task 18.0 in stage 0.0 (TID 18, octopus.home, partition 18,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:00 INFO TaskSetManager: Finished task 11.0 in stage 0.0 (TID 11) in 2654 ms on octopus.home (11/29)
17/04/28 21:50:00 INFO TaskSetManager: Starting task 19.0 in stage 0.0 (TID 19, octopus.home, partition 19,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:00 INFO TaskSetManager: Finished task 10.0 in stage 0.0 (TID 10) in 2689 ms on octopus.home (12/29)
17/04/28 21:50:01 INFO TaskSetManager: Starting task 20.0 in stage 0.0 (TID 20, kraken.home, partition 20,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:01 INFO TaskSetManager: Finished task 13.0 in stage 0.0 (TID 13) in 2583 ms on kraken.home (13/29)
17/04/28 21:50:01 INFO TaskSetManager: Starting task 21.0 in stage 0.0 (TID 21, kraken.home, partition 21,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:01 INFO TaskSetManager: Finished task 12.0 in stage 0.0 (TID 12) in 2876 ms on kraken.home (14/29)
17/04/28 21:50:02 INFO TaskSetManager: Starting task 22.0 in stage 0.0 (TID 22, kraken.home, partition 22,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:02 INFO TaskSetManager: Finished task 14.0 in stage 0.0 (TID 14) in 2078 ms on kraken.home (15/29)
17/04/28 21:50:02 INFO TaskSetManager: Starting task 23.0 in stage 0.0 (TID 23, kraken.home, partition 23,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:02 INFO TaskSetManager: Finished task 15.0 in stage 0.0 (TID 15) in 2033 ms on kraken.home (16/29)
17/04/28 21:50:03 INFO TaskSetManager: Starting task 24.0 in stage 0.0 (TID 24, octopus.home, partition 24,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:03 INFO TaskSetManager: Finished task 16.0 in stage 0.0 (TID 16) in 2480 ms on octopus.home (17/29)
17/04/28 21:50:03 INFO TaskSetManager: Starting task 25.0 in stage 0.0 (TID 25, octopus.home, partition 25,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:03 INFO TaskSetManager: Finished task 17.0 in stage 0.0 (TID 17) in 2501 ms on octopus.home (18/29)
17/04/28 21:50:03 INFO TaskSetManager: Starting task 26.0 in stage 0.0 (TID 26, octopus.home, partition 26,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:03 INFO TaskSetManager: Finished task 18.0 in stage 0.0 (TID 18) in 2495 ms on octopus.home (19/29)
17/04/28 21:50:03 INFO TaskSetManager: Starting task 27.0 in stage 0.0 (TID 27, octopus.home, partition 27,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:03 INFO TaskSetManager: Finished task 19.0 in stage 0.0 (TID 19) in 2519 ms on octopus.home (20/29)
17/04/28 21:50:03 INFO TaskSetManager: Starting task 28.0 in stage 0.0 (TID 28, kraken.home, partition 28,PROCESS_LOCAL, 2219 bytes)
17/04/28 21:50:03 INFO TaskSetManager: Finished task 20.0 in stage 0.0 (TID 20) in 2064 ms on kraken.home (21/29)
17/04/28 21:50:03 INFO TaskSetManager: Finished task 21.0 in stage 0.0 (TID 21) in 1785 ms on kraken.home (22/29)
17/04/28 21:50:03 INFO TaskSetManager: Finished task 22.0 in stage 0.0 (TID 22) in 1763 ms on kraken.home (23/29)
17/04/28 21:50:03 INFO TaskSetManager: Finished task 23.0 in stage 0.0 (TID 23) in 1648 ms on kraken.home (24/29)
17/04/28 21:50:04 INFO TaskSetManager: Finished task 28.0 in stage 0.0 (TID 28) in 967 ms on kraken.home (25/29)
17/04/28 21:50:05 INFO TaskSetManager: Finished task 24.0 in stage 0.0 (TID 24) in 1876 ms on octopus.home (26/29)
17/04/28 21:50:05 INFO TaskSetManager: Finished task 25.0 in stage 0.0 (TID 25) in 1850 ms on octopus.home (27/29)
17/04/28 21:50:05 INFO TaskSetManager: Finished task 26.0 in stage 0.0 (TID 26) in 1851 ms on octopus.home (28/29)
17/04/28 21:50:05 INFO TaskSetManager: Finished task 27.0 in stage 0.0 (TID 27) in 1847 ms on octopus.home (29/29)
17/04/28 21:50:05 INFO DAGScheduler: ResultStage 0 (reduce at LapinLePlusCretinWithSparkCluster.java:91) finished in 11,452 s
17/04/28 21:50:05 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool
17/04/28 21:50:05 INFO DAGScheduler: Job 0 finished: reduce at LapinLePlusCretinWithSparkCluster.java:91, took 11,589377 s
Le plus crétin des lapins crétins est :
petit lapin crétin qui fait daaaaaaaaaaa déguisé en pilote de tricycle avec une tronconneuse entre les dents et une balise de chantier dans la main droite
avec un score de : 52
duration : 14466 ms
17/04/28 21:50:05 INFO SparkContext: Invoking stop() from shutdown hook
17/04/28 21:50:05 INFO SparkUI: Stopped Spark web UI at http://192.168.1.11:4040
17/04/28 21:50:05 INFO SparkDeploySchedulerBackend: Shutting down all executors
17/04/28 21:50:05 INFO SparkDeploySchedulerBackend: Asking each executor to shut down
17/04/28 21:50:05 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
17/04/28 21:50:05 INFO MemoryStore: MemoryStore cleared
17/04/28 21:50:05 INFO BlockManager: BlockManager stopped
17/04/28 21:50:05 INFO BlockManagerMaster: BlockManagerMaster stopped
17/04/28 21:50:05 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
17/04/28 21:50:05 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon.
17/04/28 21:50:05 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports.
17/04/28 21:50:05 INFO SparkContext: Successfully stopped SparkContext
17/04/28 21:50:05 INFO ShutdownHookManager: Shutdown hook called
17/04/28 21:50:05 INFO ShutdownHookManager: Deleting directory /tmp/spark-d9ff765c-8b0b-428f-83a4-59ca7cab757d
17/04/28 21:50:05 INFO ShutdownHookManager: Deleting directory /tmp/spark-d9ff765c-8b0b-428f-83a4-59ca7cab757d/httpd-06f891f7-2241-41cf-8038-e3aed6fcb52c
my@octopus:~/dev/git/big-lapin/best-lapin-cretin-spark$