Fabric8 Maven Plugin Example

[YegorMaksymchuk]

Hello !

I've found an error in Fabric8 maven plugin example application

Preconditions:

1. Install maven
2. Install openshift
3. Execute oc cluster up or use remote server with openshift
4. Login to Openshift and select empty project in terminal (oc login and oc new-project <project_name>)

Steps to reproduce:

1. Execute oc create -f https://radanalytics.io/resources.yaml
2. Clone repositories Kafka , WebUI, Spark
3. Go to folder with Kafka and run mvn clean fabric8:deploy
4. Go to folder with WebUI and run mvn clean fabric8:deploy
5. Go to folder with Spark and run mvn clean fabric8:deploy

Actual result:
With Spark versions 2.1.0 and 2.2.0 I got the same result.
All parts of the application were deployed but an integration between Sprak and WebUI is broken.
With errors in Spark-pod-log:
`+ [[ /usr/local/s2i/run == /\u\s\r/\l\o\c\a\l/\s\2\i ]]

• exec /usr/local/s2i/run
  oshinko v0.5.3
  Default spark image: radanalyticsio/openshift-spark:2.3-latest
  grep: /etc/podinfo/labels: No such file or directory
  Found cluster mycluster
  Using shared cluster mycluster
  Waiting for spark master http://mycluster-ui:8080 to be available ...
  Waiting for spark workers (2/2 alive) ...
  All spark workers alive
  Cluster configuration is
  [
  {
  "namespace": "fabric8-maven-plugin",
  "name": "mycluster",
  "href": "/clusters/mycluster",
  "image": "radanalyticsio/openshift-spark:2.3-latest",
  "masterUrl": "spark://mycluster:7077",
  "masterWebUrl": "http://mycluster-ui:8080 ",
  "masterWebRoute": "mycluster-ui-route-fabric8-maven-plugin.127.0.0.1.nip.io",
  "status": "Running",
  "workerCount": 2,
  "masterCount": 1,
  "Config": {
  "MasterCount": 1,
  "WorkerCount": 2,
  "Name": "myconfig",
  "SparkMasterConfig": "",
  "SparkWorkerConfig": "",
  "SparkImage": "radanalyticsio/openshift-spark:2.3-latest",
  "ExposeWebUI": "true",
  "Metrics": "false"
  },
  "ephemeral": "\u003cshared\u003e"
  }
  ]
  spark-submit --class com.voxxed.bigdata.spark.Stream --master spark://mycluster:7077 /opt/app-root/src/voxxed-bigdata-spark-1.0-SNAPSHOT.jar
  18/06/25 10:46:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  18/06/25 10:46:06 INFO SparkContext: Running Spark version 2.3.0
  18/06/25 10:46:06 INFO SparkContext: Submitted application: voxxed-bigdata-spark
  18/06/25 10:46:06 INFO SecurityManager: Changing view acls to: default
  18/06/25 10:46:06 INFO SecurityManager: Changing modify acls to: default
  18/06/25 10:46:06 INFO SecurityManager: Changing view acls groups to:
  18/06/25 10:46:06 INFO SecurityManager: Changing modify acls groups to:
  18/06/25 10:46:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(default); groups with view permissions: Set(); users with modify permissions: Set(default); groups with modify permissions: Set()
  18/06/25 10:46:07 INFO Utils: Successfully started service 'sparkDriver' on port 42485.
  18/06/25 10:46:07 INFO SparkEnv: Registering MapOutputTracker
  18/06/25 10:46:07 INFO SparkEnv: Registering BlockManagerMaster
  18/06/25 10:46:07 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
  18/06/25 10:46:07 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
  18/06/25 10:46:07 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-66ae1f57-b63c-4479-b78a-e292fdf8db26
  18/06/25 10:46:07 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
  18/06/25 10:46:07 INFO SparkEnv: Registering OutputCommitCoordinator
  18/06/25 10:46:07 INFO Utils: Successfully started service 'SparkUI' on port 4040.
  18/06/25 10:46:07 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://voxxed-bigdata-spark-1-tftwx:4040
  18/06/25 10:46:07 INFO SparkContext: Added JAR file:/opt/app-root/src/voxxed-bigdata-spark-1.0-SNAPSHOT.jar at spark://voxxed-bigdata-spark-1-tftwx:42485/jars/voxxed-bigdata-spark-1.0-SNAPSHOT.jar with timestamp 1529923567555
  18/06/25 10:46:07 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://mycluster:7077...
  18/06/25 10:46:07 INFO TransportClientFactory: Successfully created connection to mycluster/172.30.111.85:7077 after 24 ms (0 ms spent in bootstraps)
  18/06/25 10:46:07 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20180625104607-0004
  18/06/25 10:46:07 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37657.
  18/06/25 10:46:07 INFO NettyBlockTransferService: Server created on voxxed-bigdata-spark-1-tftwx:37657
  18/06/25 10:46:07 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
  18/06/25 10:46:07 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, voxxed-bigdata-spark-1-tftwx, 37657, None)
  18/06/25 10:46:07 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20180625104607-0004/0 on worker-20180625104347-172.18.0.2-41937 (172.18.0.2:41937) with 4 core(s)
  18/06/25 10:46:07 INFO StandaloneSchedulerBackend: Granted executor ID app-20180625104607-0004/0 on hostPort 172.18.0.2:41937 with 4 core(s), 1024.0 MB RAM
  18/06/25 10:46:07 INFO BlockManagerMasterEndpoint: Registering block manager voxxed-bigdata-spark-1-tftwx:37657 with 366.3 MB RAM, BlockManagerId(driver, voxxed-bigdata-spark-1-tftwx, 37657, None)
  18/06/25 10:46:07 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20180625104607-0004/1 on worker-20180625104347-172.18.0.11-35055 (172.18.0.11:35055) with 4 core(s)
  18/06/25 10:46:07 INFO StandaloneSchedulerBackend: Granted executor ID app-20180625104607-0004/1 on hostPort 172.18.0.11:35055 with 4 core(s), 1024.0 MB RAM
  18/06/25 10:46:07 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, voxxed-bigdata-spark-1-tftwx, 37657, None)
  18/06/25 10:46:07 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, voxxed-bigdata-spark-1-tftwx, 37657, None)
  18/06/25 10:46:08 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20180625104607-0004/0 is now RUNNING
  18/06/25 10:46:08 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20180625104607-0004/1 is now RUNNING
  18/06/25 10:46:08 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
  18/06/25 10:46:08 INFO ZkEventThread: Starting ZkClient event thread.
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:host.name=voxxed-bigdata-spark-1-tftwx
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:java.version=1.8.0_161
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:java.vendor=Oracle Corporation
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-0.b14.el7_4.x86_64/jre
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:java.class.path=/opt/spark/conf/:/opt/spark/jars/commons-compress-1.4.1.jar:/opt/spark/jars/aopalliance-repackaged-2.4.0-b34.jar:/opt/spark/jars/macro-compat_2.11-1.1.1.jar:/opt/spark/jars/spark-mesos_2.11-2.3.0.jar:/opt/spark/jars/hadoop-client-2.7.3.jar:/opt/spark/jars/spark-streaming_2.11-2.3.0.jar:/opt/spark/jars/jersey-container-servlet-core-2.22.2.jar:/opt/spark/jars/orc-mapreduce-1.4.1-nohive.jar:/opt/spark/jars/jackson-mapper-asl-1.9.13.jar:/opt/spark/jars/flatbuffers-1.2.0-3f79e055.jar:/opt/spark/jars/orc-core-1.4.1-nohive.jar:/opt/spark/jars/breeze_2.11-0.13.2.jar:/opt/spark/jars/joda-time-2.9.3.jar:/opt/spark/jars/parquet-encoding-1.8.2.jar:/opt/spark/jars/metrics-graphite-3.1.5.jar:/opt/spark/jars/parquet-common-1.8.2.jar:/opt/spark/jars/jackson-module-scala_2.11-2.6.7.1.jar:/opt/spark/jars/scala-library-2.11.8.jar:/opt/spark/jars/hadoop-mapreduce-client-core-2.7.3.jar:/opt/spark/jars/jersey-client-2.22.2.jar:/opt/spark/jars/json4s-jackson_2.11-3.2.11.jar:/opt/spark/jars/spark-mllib-local_2.11-2.3.0.jar:/opt/spark/jars/hadoop-yarn-server-web-proxy-2.7.3.jar:/opt/spark/jars/antlr4-runtime-4.7.jar:/opt/spark/jars/datanucleus-api-jdo-3.2.6.jar:/opt/spark/jars/mesos-1.4.0-shaded-protobuf.jar:/opt/spark/jars/calcite-linq4j-1.2.0-incubating.jar:/opt/spark/jars/spark-tags_2.11-2.3.0.jar:/opt/spark/jars/stream-2.7.0.jar:/opt/spark/jars/commons-net-2.2.jar:/opt/spark/jars/janino-3.0.8.jar:/opt/spark/jars/chill-java-0.8.4.jar:/opt/spark/jars/hadoop-annotations-2.7.3.jar:/opt/spark/jars/jsr305-1.3.9.jar:/opt/spark/jars/spark-core_2.11-2.3.0.jar:/opt/spark/jars/hadoop-yarn-common-2.7.3.jar:/opt/spark/jars/slf4j-api-1.7.16.jar:/opt/spark/jars/xbean-asm5-shaded-4.4.jar:/opt/spark/jars/stax-api-1.0.1.jar:/opt/spark/jars/activation-1.1.1.jar:/opt/spark/jars/commons-crypto-1.0.0.jar:/opt/spark/jars/hk2-api-2.4.0-b34.jar:/opt/spark/jars/jersey-common-2.22.2.jar:/opt/spark/jars/commons-beanutils-core-1.8.0.jar:/opt/spark/jars/javax.servlet-api-3.1.0.jar:/opt/spark/jars/leveldbjni-all-1.8.jar:/opt/spark/jars/bcprov-jdk15on-1.58.jar:/opt/spark/jars/RoaringBitmap-0.5.11.jar:/opt/spark/jars/commons-compiler-3.0.8.jar:/opt/spark/jars/zookeeper-3.4.6.jar:/opt/spark/jars/jackson-core-2.6.7.jar:/opt/spark/jars/protobuf-java-2.5.0.jar:/opt/spark/jars/spark-catalyst_2.11-2.3.0.jar:/opt/spark/jars/commons-pool-1.5.4.jar:/opt/spark/jars/hk2-locator-2.4.0-b34.jar:/opt/spark/jars/log4j-1.2.17.jar:/opt/spark/jars/hive-metastore-1.2.1.spark2.jar:/opt/spark/jars/spark-hive-thriftserver_2.11-2.3.0.jar:/opt/spark/jars/snakeyaml-1.15.jar:/opt/spark/jars/javassist-3.18.1-GA.jar:/opt/spark/jars/univocity-parsers-2.5.9.jar:/opt/spark/jars/hive-jdbc-1.2.1.spark2.jar:/opt/spark/jars/paranamer-2.8.jar:/opt/spark/jars/jsp-api-2.1.jar:/opt/spark/jars/jta-1.1.jar:/opt/spark/jars/hadoop-common-2.7.3.jar:/opt/spark/jars/jcl-over-slf4j-1.7.16.jar:/opt/spark/jars/aopalliance-1.0.jar:/opt/spark/jars/hadoop-yarn-api-2.7.3.jar:/opt/spark/jars/kubernetes-model-2.0.0.jar:/opt/spark/jars/ivy-2.4.0.jar:/opt/spark/jars/hadoop-mapreduce-client-shuffle-2.7.3.jar:/opt/spark/jars/arrow-memory-0.8.0.jar:/opt/spark/jars/commons-logging-1.1.3.jar:/opt/spark/jars/commons-httpclient-3.1.jar:/opt/spark/jars/guava-14.0.1.jar:/opt/spark/jars/libfb303-0.9.3.jar:/opt/spark/jars/spark-sql_2.11-2.3.0.jar:/opt/spark/jars/automaton-1.11-8.jar:/opt/spark/jars/xercesImpl-2.9.1.jar:/opt/spark/jars/httpclient-4.5.4.jar:/opt/spark/jars/guice-servlet-3.0.jar:/opt/spark/jars/javax.inject-1.jar:/opt/spark/jars/zstd-jni-1.3.2-2.jar:/opt/spark/jars/logging-interceptor-3.8.1.jar:/opt/spark/jars/parquet-hadoop-1.8.2.jar:/opt/spark/jars/antlr-2.7.7.jar:/opt/spark/jars/calcite-avatica-1.2.0-incubating.jar:/opt/spark/jars/hadoop-yarn-client-2.7.3.jar:/opt/spark/jars/jersey-container-servlet-2.22.2.jar:/opt/spark/jars/parquet-format-2.3.1.jar:/opt/spark/jars/hadoop-mapreduce-client-jobclient-2.7.3.jar:/opt/spark/jars/netty-3.9.9.Final.jar:/opt/spark/jars/jersey-guava-2.22.2.jar:/opt/spark/jars/guice-3.0.jar:/opt/spark/jars/validation-api-1.1.0.Final.jar:/opt/spark/jars/kryo-shaded-3.0.3.jar:/opt/spark/jars/okio-1.13.0.jar:/opt/spark/jars/arrow-vector-0.8.0.jar:/opt/spark/jars/curator-recipes-2.7.1.jar:/opt/spark/jars/jpam-1.1.jar:/opt/spark/jars/hive-cli-1.2.1.spark2.jar:/opt/spark/jars/metrics-jvm-3.1.5.jar:/opt/spark/jars/netty-all-4.1.17.Final.jar:/opt/spark/jars/minlog-1.3.0.jar:/opt/spark/jars/jackson-xc-1.9.13.jar:/opt/spark/jars/commons-configuration-1.6.jar:/opt/spark/jars/commons-dbcp-1.4.jar:/opt/spark/jars/osgi-resource-locator-1.0.1.jar:/opt/spark/jars/base64-2.3.8.jar:/opt/spark/jars/json4s-core_2.11-3.2.11.jar:/opt/spark/jars/java-xmlbuilder-1.1.jar:/opt/spark/jars/antlr-runtime-3.4.jar:/opt/spark/jars/spark-launcher_2.11-2.3.0.jar:/opt/spark/jars/jersey-media-jaxb-2.22.2.jar:/opt/spark/jars/compress-lzf-1.0.3.jar:/opt/spark/jars/generex-1.0.1.jar:/opt/spark/jars/slf4j-log4j12-1.7.16.jar:/opt/spark/jars/jtransforms-2.4.0.jar:/opt/spark/jars/lz4-java-1.4.0.jar:/opt/spark/jars/pyrolite-4.13.jar:/opt/spark/jars/calcite-core-1.2.0-incubating.jar:/opt/spark/jars/metrics-json-3.1.5.jar:/opt/spark/jars/spark-network-shuffle_2.11-2.3.0.jar:/opt/spark/jars/core-1.1.2.jar:/opt/spark/jars/spire-macros_2.11-0.13.0.jar:/opt/spark/jars/datanucleus-core-3.2.10.jar:/opt/spark/jars/parquet-hadoop-bundle-1.6.0.jar:/opt/spark/jars/stringtemplate-3.2.1.jar:/opt/spark/jars/curator-framework-2.7.1.jar:/opt/spark/jars/bonecp-0.8.0.RELEASE.jar:/opt/spark/jars/arpack_combined_all-0.1.jar:/opt/spark/jars/jul-to-slf4j-1.7.16.jar:/opt/spark/jars/jaxb-api-2.2.2.jar:/opt/spark/jars/eigenbase-properties-1.1.5.jar:/opt/spark/jars/avro-1.7.7.jar:/opt/spark/jars/jackson-dataformat-yaml-2.6.7.jar:/opt/spark/jars/spark-kubernetes_2.11-2.3.0.jar:/opt/spark/jars/chill_2.11-0.8.4.jar:/opt/spark/jars/hive-beeline-1.2.1.spark2.jar:/opt/spark/jars/spark-graphx_2.11-2.3.0.jar:/opt/spark/jars/commons-beanutils-1.7.0.jar:/opt/spark/jars/xz-1.0.jar:/opt/spark/jars/spire_2.11-0.13.0.jar:/opt/spark/jars/javax.inject-2.4.0-b34.jar:/opt/spark/jars/okhttp-3.8.1.jar:/opt/spark/jars/commons-collections-3.2.2.jar:/opt/spark/jars/jackson-core-asl-1.9.13.jar:/opt/spark/jars/libthrift-0.9.3.jar:/opt/spark/jars/commons-io-2.4.jar:/opt/spark/jars/jackson-jaxrs-1.9.13.jar:/opt/spark/jars/machinist_2.11-0.6.1.jar:/opt/spark/jars/spark-repl_2.11-2.3.0.jar:/opt/spark/jars/hadoop-hdfs-2.7.3.jar:/opt/spark/jars/metrics-core-3.1.5.jar:/opt/spark/jars/api-util-1.0.0-M20.jar:/opt/spark/jars/jackson-annotations-2.6.7.jar:/opt/spark/jars/shapeless_2.11-2.3.2.jar:/opt/spark/jars/spark-network-common_2.11-2.3.0.jar:/opt/spark/jars/scalap-2.11.8.jar:/opt/spark/jars/jackson-module-paranamer-2.7.9.jar:/opt/spark/jars/scala-compiler-2.11.8.jar:/opt/spark/jars/scala-parser-combinators_2.11-1.0.4.jar:/opt/spark/jars/zjsonpatch-0.3.0.jar:/opt/spark/jars/commons-codec-1.10.jar:/opt/spark/jars/snappy-java-1.1.2.6.jar:/opt/spark/jars/parquet-jackson-1.8.2.jar:/opt/spark/jars/commons-lang3-3.5.jar:/opt/spark/jars/commons-digester-1.8.jar:/opt/spark/jars/spark-sketch_2.11-2.3.0.jar:/opt/spark/jars/spark-kvstore_2.11-2.3.0.jar:/opt/spark/jars/htrace-core-3.1.0-incubating.jar:/opt/spark/jars/apacheds-i18n-2.0.0-M15.jar:/opt/spark/jars/apache-log4j-extras-1.2.17.jar:/opt/spark/jars/commons-lang-2.6.jar:/opt/spark/jars/breeze-macros_2.11-0.13.2.jar:/opt/spark/jars/hadoop-mapreduce-client-app-2.7.3.jar:/opt/spark/jars/py4j-0.10.6.jar:/opt/spark/jars/jline-2.12.1.jar:/opt/spark/jars/spark-yarn_2.11-2.3.0.jar:/opt/spark/jars/snappy-0.2.jar:/opt/spark/jars/hive-exec-1.2.1.spark2.jar:/opt/spark/jars/httpcore-4.4.8.jar:/opt/spark/jars/stax-api-1.0-2.jar:/opt/spark/jars/super-csv-2.2.0.jar:/opt/spark/jars/kubernetes-client-3.0.0.jar:/opt/spark/jars/api-asn1-api-1.0.0-M20.jar:/opt/spark/jars/scala-xml_2.11-1.0.5.jar:/opt/spark/jars/ST4-4.0.4.jar:/opt/spark/jars/javax.annotation-api-1.2.jar:/opt/spark/jars/jersey-server-2.22.2.jar:/opt/spark/jars/spark-unsafe_2.11-2.3.0.jar:/opt/spark/jars/spark-hive_2.11-2.3.0.jar:/opt/spark/jars/hadoop-mapreduce-client-common-2.7.3.jar:/opt/spark/jars/jackson-databind-2.6.7.1.jar:/opt/spark/jars/javax.ws.rs-api-2.0.1.jar:/opt/spark/jars/JavaEWAH-0.3.2.jar:/opt/spark/jars/datanucleus-rdbms-3.2.9.jar:/opt/spark/jars/objenesis-2.1.jar:/opt/spark/jars/jdo-api-3.0.1.jar:/opt/spark/jars/jetty-util-6.1.26.jar:/opt/spark/jars/opencsv-2.3.jar:/opt/spark/jars/parquet-column-1.8.2.jar:/opt/spark/jars/gson-2.2.4.jar:/opt/spark/jars/json4s-ast_2.11-3.2.11.jar:/opt/spark/jars/jodd-core-3.5.2.jar:/opt/spark/jars/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/spark/jars/avro-mapred-1.7.7-hadoop2.jar:/opt/spark/jars/avro-ipc-1.7.7.jar:/opt/spark/jars/hppc-0.7.2.jar:/opt/spark/jars/derby-10.12.1.1.jar:/opt/spark/jars/hadoop-yarn-server-common-2.7.3.jar:/opt/spark/jars/jetty-6.1.26.jar:/opt/spark/jars/hadoop-auth-2.7.3.jar:/opt/spark/jars/commons-math3-3.4.1.jar:/opt/spark/jars/jets3t-0.9.4.jar:/opt/spark/jars/oro-2.0.8.jar:/opt/spark/jars/commons-cli-1.2.jar:/opt/spark/jars/javolution-5.5.1.jar:/opt/spark/jars/curator-client-2.7.1.jar:/opt/spark/jars/hk2-utils-2.4.0-b34.jar:/opt/spark/jars/xmlenc-0.52.jar:/opt/spark/jars/aircompressor-0.8.jar:/opt/spark/jars/jackson-module-jaxb-annotations-2.6.7.jar:/opt/spark/jars/scala-reflect-2.11.8.jar:/opt/spark/jars/spark-mllib_2.11-2.3.0.jar:/opt/spark/jars/arrow-format-0.8.0.jar
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:java.io.tmpdir=/tmp
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:java.compiler=
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:os.name=Linux
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:os.arch=amd64
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:os.version=4.16.15-200.fc27.x86_64
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:user.name=default
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:user.home=/
  18/06/25 10:46:08 INFO ZooKeeper: Client environment:user.dir=/opt/jboss
  18/06/25 10:46:08 INFO ZooKeeper: Initiating client connection, connectString=zookeeper:2181 sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@273842a6
  18/06/25 10:46:08 INFO ZkClient: Waiting for keeper state SyncConnected
  18/06/25 10:46:08 INFO ClientCnxn: Opening socket connection to server zookeeper.fabric8-maven-plugin.svc.cluster.local/172.30.252.173:2181. Will not attempt to authenticate using SASL (unknown error)
  18/06/25 10:46:08 INFO ClientCnxn: Socket connection established to zookeeper.fabric8-maven-plugin.svc.cluster.local/172.30.252.173:2181, initiating session
  18/06/25 10:46:09 INFO ClientCnxn: Session establishment complete on server zookeeper.fabric8-maven-plugin.svc.cluster.local/172.30.252.173:2181, sessionid = 0x1643688effd0009, negotiated timeout = 10000
  18/06/25 10:46:09 INFO ZkClient: zookeeper state changed (SyncConnected)
  kafka.common.TopicExistsException: Topic "stars" already exists.
  at kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:420)
  at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:404)
  at com.voxxed.bigdata.spark.KafkaSupport$.createTopicIfNotPresent(KafkaSupport.scala:41)
  at com.voxxed.bigdata.spark.Stream$.main(Stream.scala:21)
  at com.voxxed.bigdata.spark.Stream.main(Stream.scala)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
  at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
  at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
  at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  18/06/25 10:46:09 INFO ZooKeeper: Initiating client connection, connectString=zookeeper:2181 sessionTimeout=10000 watcher=org.I0Itec.zkclient.ZkClient@1cb19dba
  18/06/25 10:46:09 INFO ZkEventThread: Starting ZkClient event thread.
  18/06/25 10:46:09 INFO ZkClient: Waiting for keeper state SyncConnected
  18/06/25 10:46:09 INFO ClientCnxn: Opening socket connection to server zookeeper.fabric8-maven-plugin.svc.cluster.local/172.30.252.173:2181. Will not attempt to authenticate using SASL (unknown error)
  18/06/25 10:46:09 INFO ClientCnxn: Socket connection established to zookeeper.fabric8-maven-plugin.svc.cluster.local/172.30.252.173:2181, initiating session
  18/06/25 10:46:09 INFO ClientCnxn: Session establishment complete on server zookeeper.fabric8-maven-plugin.svc.cluster.local/172.30.252.173:2181, sessionid = 0x1643688effd000a, negotiated timeout = 10000
  18/06/25 10:46:09 INFO ZkClient: zookeeper state changed (SyncConnected)
  kafka.common.TopicExistsException: Topic "recommendations" already exists.
  at kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:420)
  at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:404)
  at com.voxxed.bigdata.spark.KafkaSupport$.createTopicIfNotPresent(KafkaSupport.scala:41)
  at com.voxxed.bigdata.spark.Stream$.main(Stream.scala:22)
  at com.voxxed.bigdata.spark.Stream.main(Stream.scala)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
  at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
  at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
  at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Exception in thread "main" java.lang.AbstractMethodError
  at org.apache.spark.internal.Logging$class.initializeLogIfNecessary(Logging.scala:99)
  at org.apache.spark.streaming.kafka010.KafkaUtils$.initializeLogIfNecessary(KafkaUtils.scala:40)
  at org.apache.spark.internal.Logging$class.log(Logging.scala:46)
  at org.apache.spark.streaming.kafka010.KafkaUtils$.log(KafkaUtils.scala:40)
  at org.apache.spark.internal.Logging$class.logWarning(Logging.scala:66)
  at org.apache.spark.streaming.kafka010.KafkaUtils$.logWarning(KafkaUtils.scala:40)
  at org.apache.spark.streaming.kafka010.KafkaUtils$.fixKafkaParams(KafkaUtils.scala:208)
  at org.apache.spark.streaming.kafka010.DirectKafkaInputDStream.(DirectKafkaInputDStream.scala:66)
  at org.apache.spark.streaming.kafka010.KafkaUtils$.createDirectStream(KafkaUtils.scala:150)
  at org.apache.spark.streaming.kafka010.KafkaUtils$.createDirectStream(KafkaUtils.scala:127)
  at com.voxxed.bigdata.spark.Stream$.main(Stream.scala:27)
  at com.voxxed.bigdata.spark.Stream.main(Stream.scala)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
  at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:879)
  at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
  at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
  at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
  at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  18/06/25 10:46:10 INFO SparkContext: Invoking stop() from shutdown hook
  18/06/25 10:46:10 INFO SparkUI: Stopped Spark web UI at http://voxxed-bigdata-spark-1-tftwx:4040
  18/06/25 10:46:10 INFO StandaloneSchedulerBackend: Shutting down all executors
  18/06/25 10:46:10 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
  18/06/25 10:46:10 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
  18/06/25 10:46:10 INFO MemoryStore: MemoryStore cleared
  18/06/25 10:46:10 INFO BlockManager: BlockManager stopped
  18/06/25 10:46:10 INFO BlockManagerMaster: BlockManagerMaster stopped
  18/06/25 10:46:10 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
  18/06/25 10:46:10 INFO SparkContext: Successfully stopped SparkContext
  18/06/25 10:46:10 INFO ShutdownHookManager: Shutdown hook called
  18/06/25 10:46:10 INFO ShutdownHookManager: Deleting directory /tmp/spark-5d868bdf-ea56-474c-8caf-183aaefe191c
  18/06/25 10:46:10 INFO ShutdownHookManager: Deleting directory /tmp/spark-0d4f9ac1-a242-4587-a858-15270556afae
  Deleting cluster 'mycluster'
  cluster is not ephemeral
  cluster not deleted 'mycluster'
  `

In case if I change spark version in Spark/pom.xm form 2.1.0 (2.2.0) to 2.3.0, maven can't build project:

[ymaks@ymaks test]$ cd voxxed-bigdata-spark/ [ymaks@ymaks voxxed-bigdata-spark]$ mvn clean fabric8:deploy [INFO] Scanning for projects... [WARNING] [WARNING] Some problems were encountered while building the effective model for com.voxxed.bigdata:voxxed-bigdata-spark:jar:1.0-SNAPSHOT [WARNING] The expression ${build.finalName} is deprecated. Please use ${project.build.finalName} instead. [WARNING] [WARNING] It is highly recommended to fix these problems because they threaten the stability of your build. [WARNING] [WARNING] For this reason, future Maven versions might no longer support building such malformed projects. [WARNING] [INFO] [INFO] --------------< com.voxxed.bigdata:voxxed-bigdata-spark >--------------- [INFO] Building voxxed-bigdata-spark 1.0-SNAPSHOT [INFO] --------------------------------[ jar ]--------------------------------- Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/commons-codec/commons-codec/maven-metadata.xml Downloading from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/maven-metadata.xml Downloading from apache.snapshots: https://repository.apache.org/snapshots/commons-codec/commons-codec/maven-metadata.xml Downloaded from central: https://repo.maven.apache.org/maven2/commons-codec/commons-codec/maven-metadata.xml (642 B at 1.0 kB/s) Downloaded from apache.snapshots: https://repository.apache.org/snapshots/commons-codec/commons-codec/maven-metadata.xml (339 B at 236 B/s) Downloading from apache.snapshots: https://repository.apache.org/snapshots/commons-codec/commons-codec/2.0-SNAPSHOT/maven-metadata.xml Downloading from sonatype-nexus-snapshots: https://oss.sonatype.org/content/repositories/snapshots/commons-codec/commons-codec/2.0-SNAPSHOT/maven-metadata.xml Downloaded from apache.snapshots: https://repository.apache.org/snapshots/commons-codec/commons-codec/2.0-SNAPSHOT/maven-metadata.xml (2.5 kB at 2.0 kB/s) [INFO] [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ voxxed-bigdata-spark --- [INFO] [INFO] >>> fabric8-maven-plugin:3.2.28:deploy (default-cli) > install @ voxxed-bigdata-spark >>> [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ voxxed-bigdata-spark --- [WARNING] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! [INFO] skip non existing resourceDirectory /home/ymaks/sources/IdeaProjects/test/voxxed-bigdata-spark/src/main/resources [INFO] [INFO] --- fabric8-maven-plugin:3.2.28:resource (default) @ voxxed-bigdata-spark --- [INFO] F8: Running in OpenShift mode [INFO] F8: Using docker image name of namespace: fabric8-maven-plugin [INFO] F8: Running generator java-exec [INFO] F8: java-exec: Using Docker image radanalyticsio/radanalytics-java-spark as base / builder [INFO] F8: Using resource templates from /home/ymaks/sources/IdeaProjects/test/voxxed-bigdata-spark/src/main/fabric8 [INFO] F8: fmp-service: Adding a default Service with ports [8080] [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ voxxed-bigdata-spark --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- scala-maven-plugin:3.2.1:compile (default) @ voxxed-bigdata-spark --- [INFO] artifact commons-codec:commons-codec: checking for updates from central [INFO] artifact commons-codec:commons-codec: checking for updates from spy [WARNING] Expected all dependencies to require Scala version: 2.11.8 [WARNING] com.twitter:chill_2.11:0.8.4 requires scala version: 2.11.8 [WARNING] org.apache.spark:spark-core_2.11:2.3.0 requires scala version: 2.11.8 [WARNING] org.json4s:json4s-jackson_2.11:3.2.11 requires scala version: 2.11.0 [WARNING] Multiple versions of scala libraries detected! [INFO] /home/ymaks/sources/IdeaProjects/test/voxxed-bigdata-spark/src/main/scala:-1: info: compiling [INFO] Compiling 5 source files to /home/ymaks/sources/IdeaProjects/test/voxxed-bigdata-spark/target/classes at 1529924157678 [ERROR] /home/ymaks/sources/IdeaProjects/test/voxxed-bigdata-spark/src/main/scala/com/voxxed/bigdata/spark/KafkaSupport.scala:3: error: object admin is not a member of package kafka [ERROR] import kafka.admin.AdminUtils [ERROR] ^ [ERROR] /home/ymaks/sources/IdeaProjects/test/voxxed-bigdata-spark/src/main/scala/com/voxxed/bigdata/spark/KafkaSupport.scala:4: error: object utils is not a member of package kafka [ERROR] import kafka.utils.ZkUtils [ERROR] ^ [ERROR] /home/ymaks/sources/IdeaProjects/test/voxxed-bigdata-spark/src/main/scala/com/voxxed/bigdata/spark/KafkaSupport.scala:39: error: not found: value ZkUtils [ERROR] val zkUtils = ZkUtils("zookeeper:2181", 10000, 10000, isZkSecurityEnabled = false) [ERROR] ^ [ERROR] /home/ymaks/sources/IdeaProjects/test/voxxed-bigdata-spark/src/main/scala/com/voxxed/bigdata/spark/KafkaSupport.scala:39: error: not found: value isZkSecurityEnabled [ERROR] val zkUtils = ZkUtils("zookeeper:2181", 10000, 10000, isZkSecurityEnabled = false) [ERROR] ^ [ERROR] /home/ymaks/sources/IdeaProjects/test/voxxed-bigdata-spark/src/main/scala/com/voxxed/bigdata/spark/KafkaSupport.scala:41: error: not found: value AdminUtils [ERROR] AdminUtils.createTopic(zkUtils, name, 1, 1) [ERROR] ^ [ERROR] 5 errors found [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 27.473 s [INFO] Finished at: 2018-06-25T13:56:02+03:00 [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.1:compile (default) on project voxxed-bigdata-spark: wrap: org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) -> [Help 1] [ERROR]

Expected result:
Application should work in the same scenario like in description of Example

[elmiko]

hi @YegorMaksymchuk , thanks for the report!

i will take a look at the tutorial to see if i can reproduce your results.

[YegorMaksymchuk]

Hi! @elmiko
so for run all spark pods with version 2.1.0 need specify tag:v.0.2.7 in resources.yaml and in Spark\pom.xml in plugin section set:

             <generator>
                   <config>
                       <java-exec>
                           <!-- The radanalytics java-spark image -->
                           <from>radanalyticsio/radanalytics-java-spark:v0.2.7</from>
                       </java-exec>
                   </config>
               </generator>

after that each spark pod will be with version 2.1.0.
But application not work properly you can't get recommendation:
Kafka log : https://pastebin.com/j1MhK4RN
Spark log: https://pastebin.com/ge5eEayq
WebUI:
Part1: https://pastebin.com/3kHx9LtN
Part2: https://pastebin.com/jGcu08Vh
Part3: https://pastebin.com/LRA7M7bV

[elmiko]

After reviewing the source code for this tutorial, i think we should hold off on automated testing for it. i would like to propose that we refactor this code into a new tutorial that just focuses on using the maven fabric8 plugin to deploy spark clusters with oshinko.

[elmiko]

@tmckayus @willb i would love to hear your thoughts on this