Unable to find any JVMs matching version "1.8".
No Java runtime present, try --request to install.
19/09/13 16:40:37 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16:40:37.851 [main] INFO  za.co.absa.hyperdrive.driver.drivers.PropertiesIngestionDriver$ - Going to load ingestion configurations from 'Ingestion.properties.template.txt'.
log4j:WARN No appenders could be found for logger (org.apache.shaded.commons.beanutils.converters.BooleanConverter).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
16:40:37.971 [main] INFO  za.co.absa.hyperdrive.driver.drivers.PropertiesIngestionDriver$ - Configurations loaded. Going to invoke ingestion: [org.apache.commons.configuration2.PropertiesConfiguration@6fff253c]
16:40:37.972 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - Ingestion invoked using the configuration below. Going to instantiate components.
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	component.ingestor = spark
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	component.reader = za.co.absa.hyperdrive.ingestor.implementation.reader.kafka.KafkaStreamReader
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	component.decoder = za.co.absa.hyperdrive.ingestor.implementation.decoder.avro.confluent.ConfluentAvroKafkaStreamDecoder
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	component.manager = za.co.absa.hyperdrive.ingestor.implementation.manager.checkpoint.CheckpointOffsetManager
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	component.transformer = za.co.absa.hyperdrive.ingestor.implementation.transformer.column.selection.ColumnSelectorStreamTransformer
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	component.writer = za.co.absa.hyperdrive.ingestor.implementation.writer.parquet.AllNullableParquetStreamWriter
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	ingestor.spark.app.name = ingestor-app-pane
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	reader.kafka.topic = pageviews
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	reader.kafka.brokers = http://localhost:9092
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	manager.checkpoint.base.location = /tmp/checkpoint-location
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	decoder.avro.schema.registry.url = http://localhost:8081
16:40:37.973 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	decoder.avro.value.schema.id = latest
16:40:37.974 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	decoder.avro.value.schema.naming.strategy = topic.name
16:40:37.974 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	decoder.avro.schema.retention.policy = RETAIN_SELECTED_COLUMN_ONLY
16:40:37.974 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	transformer.columns.to.select = *
16:40:37.974 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	writer.parquet.destination.directory = /tmp/ingestion/destination-directory
16:40:37.974 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	writer.parquet.extra.conf.1 = key1=value1
16:40:37.974 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - 	writer.parquet.extra.conf.2 = key2=value2
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
19/09/13 16:40:38 INFO SparkContext: Running Spark version 2.4.3
19/09/13 16:40:38 INFO SparkContext: Submitted application: ingestor-app-pane
19/09/13 16:40:38 INFO SecurityManager: Changing view acls to: abkw142
19/09/13 16:40:38 INFO SecurityManager: Changing modify acls to: abkw142
19/09/13 16:40:38 INFO SecurityManager: Changing view acls groups to: 
19/09/13 16:40:38 INFO SecurityManager: Changing modify acls groups to: 
19/09/13 16:40:38 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(abkw142); groups with view permissions: Set(); users  with modify permissions: Set(abkw142); groups with modify permissions: Set()
19/09/13 16:40:38 INFO Utils: Successfully started service 'sparkDriver' on port 57054.
19/09/13 16:40:38 INFO SparkEnv: Registering MapOutputTracker
19/09/13 16:40:38 INFO SparkEnv: Registering BlockManagerMaster
19/09/13 16:40:38 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
19/09/13 16:40:38 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
19/09/13 16:40:38 INFO DiskBlockManager: Created local directory at /private/var/folders/x5/xhwdn27d063d3zldgyf1wg700000gq/T/blockmgr-7b156f66-4474-43ee-a52e-bddd56ea13c5
19/09/13 16:40:38 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
19/09/13 16:40:38 INFO SparkEnv: Registering OutputCommitCoordinator
19/09/13 16:40:38 INFO Utils: Successfully started service 'SparkUI' on port 4040.
19/09/13 16:40:38 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.25.12:4040
19/09/13 16:40:38 INFO SparkContext: Added JAR file:/Users/abkw142/my/workingDir/driver-1.0.0-SNAPSHOT.jar at spark://192.168.25.12:57054/jars/driver-1.0.0-SNAPSHOT.jar with timestamp 1568385638531
19/09/13 16:40:38 INFO Executor: Starting executor ID driver on host localhost
19/09/13 16:40:38 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57055.
19/09/13 16:40:38 INFO NettyBlockTransferService: Server created on 192.168.25.12:57055
19/09/13 16:40:38 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
19/09/13 16:40:38 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.25.12, 57055, None)
19/09/13 16:40:38 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.25.12:57055 with 366.3 MB RAM, BlockManagerId(driver, 192.168.25.12, 57055, None)
19/09/13 16:40:38 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.25.12, 57055, None)
19/09/13 16:40:38 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.25.12, 57055, None)
16:40:38.817 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.reader.factories.StreamReaderAbstractFactory$ - Going to load factory for configuration 'component.reader'.
16:40:39.500 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.reader.kafka.KafkaStreamReader$ - Going to create KafkaStreamReader with: topic='pageviews', brokers='http://localhost:9092', extraOptions=Map()
16:40:39.501 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.manager.factories.OffsetManagerAbstractFactory$ - Going to load factory for configuration 'component.manager'.
16:40:39.505 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.manager.checkpoint.CheckpointOffsetManager$ - Going to create CheckpointOffsetManager instance using: topic='pageviews', checkpoint base location='/tmp/checkpoint-location'
16:40:39.506 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.decoder.factories.StreamDecoderAbstractFactory$ - Going to load factory for configuration 'component.decoder'.
16:40:39.515 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.decoder.avro.confluent.ConfluentAvroKafkaStreamDecoder$ - Going to create AvroKafkaStreamDecoder instance using: topic='pageviews', schema retention policy='RETAIN_SELECTED_COLUMN_ONLY', schema registry settings='Map(schema.registry.url -> http://localhost:8081, value.schema.id -> latest, value.schema.naming.strategy -> topic.name)'.
16:40:39.516 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.transformer.factories.StreamTransformerAbstractFactory$ - Going to load factory for configuration 'component.transformer'.
16:40:39.520 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.transformer.column.selection.ColumnSelectorStreamTransformer$ - Going to create ColumnSelectorStreamTransformer using: columns='WrappedArray(*)'
16:40:39.521 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.writer.factories.StreamWriterAbstractFactory$ - Going to load factory for configuration 'component.writer'.
16:40:39.525 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.writer.parquet.AllNullableParquetStreamWriter$ - Going to create AllNullableParquetStreamWriter instance. Injecting ParquetStreamWriter instance.
16:40:39.528 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.writer.parquet.ParquetStreamWriter$ - Going to create ParquetStreamWriter instance using: destination directory='/tmp/ingestion/destination-directory', extra options='Some(Map(key1 -> value1, key2 -> value2))'
16:40:39.528 [main] INFO  za.co.absa.hyperdrive.driver.IngestionDriver - Ingestion components instantiated. Going to invoke SparkIngestor.
16:40:39.529 [main] INFO  za.co.absa.hyperdrive.driver.SparkIngestor$ - STARTING ingestion from 'Kafka topic: pageviews' into '/tmp/ingestion/destination-directory' (id = 527370f5-48ae-4977-a7fc-acb79e89d69c)
19/09/13 16:40:39 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/abkw142/my/workingDir/spark-warehouse').
19/09/13 16:40:39 INFO SharedState: Warehouse path is 'file:/Users/abkw142/my/workingDir/spark-warehouse'.
19/09/13 16:40:39 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
16:40:40.015 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.manager.checkpoint.CheckpointOffsetManager - No offsets to set for topic 'pageviews'.
16:40:40.015 [main] INFO  za.co.absa.hyperdrive.ingestor.implementation.decoder.avro.confluent.ConfluentAvroKafkaStreamDecoder - SchemaRegistry settings: Map(schema.registry.url -> http://localhost:8081, value.schema.id -> latest, value.schema.naming.strategy -> topic.name, schema.registry.topic -> pageviews)
19/09/13 16:40:40 INFO KafkaAvroDeserializerConfig: KafkaAvroDeserializerConfig values: 
	schema.registry.url = [http://localhost:8081]
	basic.auth.user.info = [hidden]
	auto.register.schemas = true
	max.schemas.per.subject = 1000
	basic.auth.credentials.source = URL
	schema.registry.basic.auth.user.info = [hidden]
	specific.avro.reader = false
	value.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy
	key.subject.name.strategy = class io.confluent.kafka.serializers.subject.TopicNameStrategy

19/09/13 16:40:40 INFO SubjectNameStrategyAdapterFactory$: Creating adapter for strategy: topic.name
19/09/13 16:40:40 INFO SchemaManager$: Subject name resolved to: pageviews-value
19/09/13 16:40:40 INFO SchemaManager$: Trying to get latest schema version id for subject 'pageviews-value'
19/09/13 16:40:40 INFO SchemaManager$: Trying to get schema for subject 'pageviews-value' and id '1'
Exception in thread "main" za.co.absa.hyperdrive.shared.exceptions.IngestionStartException: NOT STARTED ingestion 527370f5-48ae-4977-a7fc-acb79e89d69c. This exception was thrown during the starting of the ingestion job. Check the logs for details.
	at za.co.absa.hyperdrive.driver.SparkIngestor$.ingest(SparkIngestor.scala:104)
	at za.co.absa.hyperdrive.driver.IngestionDriver.ingest(IngestionDriver.scala:51)
	at za.co.absa.hyperdrive.driver.drivers.PropertiesIngestionDriver$.main(PropertiesIngestionDriver.scala:50)
	at za.co.absa.hyperdrive.driver.drivers.PropertiesIngestionDriver.main(PropertiesIngestionDriver.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.sql.AnalysisException: Queries with streaming sources must be executed with writeStream.start();;
kafka
	at org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$.org$apache$spark$sql$catalyst$analysis$UnsupportedOperationChecker$$throwError(UnsupportedOperationChecker.scala:389)
	at org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$$anonfun$checkForBatch$1.apply(UnsupportedOperationChecker.scala:38)
	at org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$$anonfun$checkForBatch$1.apply(UnsupportedOperationChecker.scala:36)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:127)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$foreachUp$1.apply(TreeNode.scala:126)
	at scala.collection.immutable.List.foreach(List.scala:392)
	at org.apache.spark.sql.catalyst.trees.TreeNode.foreachUp(TreeNode.scala:126)
	at org.apache.spark.sql.catalyst.analysis.UnsupportedOperationChecker$.checkForBatch(UnsupportedOperationChecker.scala:36)
	at org.apache.spark.sql.execution.QueryExecution.assertSupported(QueryExecution.scala:51)
	at org.apache.spark.sql.execution.QueryExecution.withCachedData$lzycompute(QueryExecution.scala:62)
	at org.apache.spark.sql.execution.QueryExecution.withCachedData(QueryExecution.scala:60)
	at org.apache.spark.sql.execution.QueryExecution.optimizedPlan$lzycompute(QueryExecution.scala:66)
	at org.apache.spark.sql.execution.QueryExecution.optimizedPlan(QueryExecution.scala:66)
	at org.apache.spark.sql.execution.QueryExecution.sparkPlan$lzycompute(QueryExecution.scala:72)
	at org.apache.spark.sql.execution.QueryExecution.sparkPlan(QueryExecution.scala:68)
	at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:77)
	at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:77)
	at org.apache.spark.sql.execution.QueryExecution.toRdd$lzycompute(QueryExecution.scala:80)
	at org.apache.spark.sql.execution.QueryExecution.toRdd(QueryExecution.scala:80)
	at org.apache.spark.sql.Dataset.rdd$lzycompute(Dataset.scala:3037)
	at org.apache.spark.sql.Dataset.rdd(Dataset.scala:3035)
	at za.co.absa.hyperdrive.shared.utils.SparkUtils$.setAllColumnsNullable(SparkUtils.scala:26)
	at za.co.absa.hyperdrive.ingestor.implementation.writer.parquet.AllNullableParquetStreamWriter.write(AllNullableParquetStreamWriter.scala:37)
	at za.co.absa.hyperdrive.driver.SparkIngestor$.ingest(SparkIngestor.scala:101)
	... 15 more
19/09/13 16:40:40 INFO SparkContext: Invoking stop() from shutdown hook
19/09/13 16:40:40 INFO SparkUI: Stopped Spark web UI at http://192.168.25.12:4040
19/09/13 16:40:40 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
19/09/13 16:40:40 INFO MemoryStore: MemoryStore cleared
19/09/13 16:40:40 INFO BlockManager: BlockManager stopped
19/09/13 16:40:40 INFO BlockManagerMaster: BlockManagerMaster stopped
19/09/13 16:40:40 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
19/09/13 16:40:40 INFO SparkContext: Successfully stopped SparkContext
19/09/13 16:40:40 INFO ShutdownHookManager: Shutdown hook called
19/09/13 16:40:40 INFO ShutdownHookManager: Deleting directory /private/var/folders/x5/xhwdn27d063d3zldgyf1wg700000gq/T/spark-d713a512-c286-4c2a-97e5-bac6fe202237
19/09/13 16:40:40 INFO ShutdownHookManager: Deleting directory /private/var/folders/x5/xhwdn27d063d3zldgyf1wg700000gq/T/spark-870b99f8-9665-4465-9a68-cf75e4e24c32
19/09/13 16:40:40 INFO ShutdownHookManager: Deleting directory /private/var/folders/x5/xhwdn27d063d3zldgyf1wg700000gq/T/temporaryReader-28836c8e-e267-42f7-8c79-098d7b63e1b2