Container: container_1602165382040_0066_01_000002 on x.x.x.x_8041
===========================================================================================================
LogType:stderr
Log Upload Time:Wed Oct 14 13:56:55 +0000 2020
LogLength:132797
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt/yarn/usercache/hadoop/filecache/2521/__spark_libs__300261197764166949.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/10/14 13:56:39 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 21147@x.x.x.x
20/10/14 13:56:39 INFO SignalUtils: Registered signal handler for TERM
20/10/14 13:56:39 INFO SignalUtils: Registered signal handler for HUP
20/10/14 13:56:39 INFO SignalUtils: Registered signal handler for INT
20/10/14 13:56:40 INFO SecurityManager: Changing view acls to: yarn,hadoop
20/10/14 13:56:40 INFO SecurityManager: Changing modify acls to: yarn,hadoop
20/10/14 13:56:40 INFO SecurityManager: Changing view acls groups to: 
20/10/14 13:56:40 INFO SecurityManager: Changing modify acls groups to: 
20/10/14 13:56:40 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
20/10/14 13:56:41 INFO TransportClientFactory: Successfully created connection to x.x.x.x/x.x.x.x:37307 after 87 ms (0 ms spent in bootstraps)
20/10/14 13:56:41 INFO SecurityManager: Changing view acls to: yarn,hadoop
20/10/14 13:56:41 INFO SecurityManager: Changing modify acls to: yarn,hadoop
20/10/14 13:56:41 INFO SecurityManager: Changing view acls groups to: 
20/10/14 13:56:41 INFO SecurityManager: Changing modify acls groups to: 
20/10/14 13:56:41 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
20/10/14 13:56:41 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:41 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:41 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:41 INFO TransportClientFactory: Successfully created connection to x.x.x.x/x.x.x.x:37307 after 1 ms (0 ms spent in bootstraps)
20/10/14 13:56:41 INFO DiskBlockManager: Created local directory at /mnt2/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-4131897d-1a83-4aea-8af1-11e02244b842
20/10/14 13:56:41 INFO DiskBlockManager: Created local directory at /mnt3/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-641194e5-eaf2-4a42-aa10-64572e8f29de
20/10/14 13:56:41 INFO DiskBlockManager: Created local directory at /mnt/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-5c7861b2-3e93-492b-9c87-8638ff76111e
20/10/14 13:56:41 INFO DiskBlockManager: Created local directory at /mnt1/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-7b67a73f-7a00-4975-b743-392487a189f4
20/10/14 13:56:41 INFO MemoryStore: MemoryStore started with capacity 10.9 GB
20/10/14 13:56:41 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:41 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:41 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:41 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@x.x.x.x:37307
20/10/14 13:56:41 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:41 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:41 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:41 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
20/10/14 13:56:41 INFO Executor: Starting executor ID 1 on host x.x.x.x
20/10/14 13:56:41 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39115.
20/10/14 13:56:41 INFO NettyBlockTransferService: Server created on x.x.x.x:39115
20/10/14 13:56:41 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/10/14 13:56:41 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(1, x.x.x.x, 39115, None)
20/10/14 13:56:41 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(1, x.x.x.x, 39115, None)
20/10/14 13:56:41 INFO BlockManager: external shuffle service port = 7337
20/10/14 13:56:41 INFO BlockManager: Registering executor with local external shuffle service.
20/10/14 13:56:41 INFO TransportClientFactory: Successfully created connection to x.x.x.x/172.31.25.35:7337 after 2 ms (0 ms spent in bootstraps)
20/10/14 13:56:41 INFO BlockManager: Initialized BlockManager: BlockManagerId(1, x.x.x.x, 39115, None)
20/10/14 13:56:43 INFO CoarseGrainedExecutorBackend: eagerFSInit: Eagerly initialized FileSystem at s3://does/not/exist in 1656 ms
20/10/14 13:56:48 INFO CoarseGrainedExecutorBackend: Got assigned task 1
20/10/14 13:56:48 INFO Executor: Running task 0.0 in stage 1.0 (TID 1)
20/10/14 13:56:48 INFO Executor: Fetching s3://demoiags3/hudi/jars/hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar with timestamp 1602683783966
20/10/14 13:56:49 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:49 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:49 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:49 INFO S3NativeFileSystem: Opening 's3://demoiags3/hudi/jars/hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar' for reading
20/10/14 13:56:49 INFO Utils: Fetching s3://demoiags3/hudi/jars/hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar to /mnt2/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-7adbd6a7-6620-4f09-8370-6bb62d9bf138/fetchFileTemp862498594668888774.tmp
20/10/14 13:56:50 INFO Utils: Copying /mnt2/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-7adbd6a7-6620-4f09-8370-6bb62d9bf138/-4264915221602683783966_cache to /mnt/yarn/usercache/hadoop/appcache/application_1602165382040_0066/container_1602165382040_0066_01_000002/./hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar
20/10/14 13:56:50 INFO Executor: Adding file:/mnt/yarn/usercache/hadoop/appcache/application_1602165382040_0066/container_1602165382040_0066_01_000002/./hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar to class loader
20/10/14 13:56:50 INFO TorrentBroadcast: Started reading broadcast variable 1
20/10/14 13:56:50 INFO TransportClientFactory: Successfully created connection to x.x.x.x/x.x.x.x:45721 after 2 ms (0 ms spent in bootstraps)
20/10/14 13:56:50 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 21.7 KB, free 10.9 GB)
20/10/14 13:56:50 INFO TorrentBroadcast: Reading broadcast variable 1 took 107 ms
20/10/14 13:56:50 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 66.0 KB, free 10.9 GB)
20/10/14 13:56:51 INFO KafkaRDD: Computing topic airflow.public.motor_crash_violation_incidents, partition 0 offsets 15 -> 17
20/10/14 13:56:51 INFO KafkaDataConsumer: Initializing cache 16 64 0.75
20/10/14 13:56:51 INFO ConsumerConfig: ConsumerConfig values: 
	auto.commit.interval.ms = 5000
	auto.offset.reset = none
	bootstrap.servers = [http://x.x.x.x:29092]
	check.crcs = true
	client.id = 
	connections.max.idle.ms = 540000
	default.api.timeout.ms = 60000
	enable.auto.commit = false
	exclude.internal.topics = true
	fetch.max.bytes = 52428800
	fetch.max.wait.ms = 500
	fetch.min.bytes = 1
	group.id = spark-executor-null
	heartbeat.interval.ms = 3000
	interceptor.classes = []
	internal.leave.group.on.close = true
	isolation.level = read_uncommitted
	key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	max.partition.fetch.bytes = 1048576
	max.poll.interval.ms = 300000
	max.poll.records = 500
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	send.buffer.bytes = 131072
	session.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	value.deserializer = class io.confluent.kafka.serializers.KafkaAvroDeserializer

20/10/14 13:56:51 INFO KafkaAvroDeserializerConfig: KafkaAvroDeserializerConfig values: 
	schema.registry.url = [http://x.x.x.x:8081]
	max.schemas.per.subject = 1000
	specific.avro.reader = false

20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'validate.non.null' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.partitionpath.field' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.compact.inline' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.delete.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.recordkey.field' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.upsert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.keygenerator.class' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.deltastreamer.source.kafka.topic' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.registry.url' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.insert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.precombine.field' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.embed.timeline.server' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.bulkinsert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:51 WARN ConsumerConfig: The configuration 'hoodie.filesystem.view.type' was supplied but isn't a known config.
20/10/14 13:56:51 INFO AppInfoParser: Kafka version : 2.0.0
20/10/14 13:56:51 INFO AppInfoParser: Kafka commitId : 3402a8361b734732
20/10/14 13:56:51 INFO CodeGenerator: Code generated in 267.466014 ms
20/10/14 13:56:51 INFO CodeGenerator: Code generated in 37.599887 ms
20/10/14 13:56:51 INFO InternalKafkaConsumer: Initial fetch for spark-executor-null airflow.public.motor_crash_violation_incidents-0 15
20/10/14 13:56:52 INFO Metadata: Cluster ID: cA3sXVaIR-qlM1MPNNYnCw
20/10/14 13:56:53 INFO CodeGenerator: Code generated in 243.334066 ms
20/10/14 13:56:53 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID 1)
java.lang.RuntimeException: Error while encoding: java.lang.NullPointerException: Null value appeared in non-nullable field:
top level row object
If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int).
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(inc_id, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 0, inc_id), IntegerType), year, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 1, year), IntegerType), violation_desc, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 2, violation_desc), StringType), true, false), violation_code, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 3, violation_code), StringType), true, false), case_individual_id, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 4, case_individual_id), IntegerType), flag, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 5, flag), StringType), true, false), last_modified_ts, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 6, last_modified_ts), LongType)) AS before#0
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(inc_id, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 0, inc_id), IntegerType), year, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 1, year), IntegerType), violation_desc, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 2, violation_desc), StringType), true, false), violation_code, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 3, violation_code), StringType), true, false), case_individual_id, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 4, case_individual_id), IntegerType), flag, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 5, flag), StringType), true, false), last_modified_ts, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 6, last_modified_ts), LongType)) AS after#1
named_struct(version, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 0, version), StringType), true, false), connector, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 1, connector), StringType), true, false), name, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 2, name), StringType), true, false), ts_ms, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 3, ts_ms), LongType), snapshot, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 4, snapshot), StringType), true, false), db, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 5, db), StringType), true, false), schema, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 6, schema), StringType), true, false), table, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 7, table), StringType), true, false), txId, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 8, txId), LongType), lsn, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 9, lsn), LongType), xmin, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 10, xmin), LongType)) AS source#2
staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, op), StringType), true, false) AS op#3
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, ts_ms), LongType) AS ts_ms#4L
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(id, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 0, id), StringType), true, false), total_order, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 1, total_order), LongType), data_collection_order, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 2, data_collection_order), LongType)) AS transaction#5
	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:292)
	at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:594)
	at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:594)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:585)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
	at scala.collection.Iterator$class.isEmpty(Iterator.scala:331)
	at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1334)
	at org.apache.hudi.AvroConversionUtils$$anonfun$2.apply(AvroConversionUtils.scala:46)
	at org.apache.hudi.AvroConversionUtils$$anonfun$2.apply(AvroConversionUtils.scala:45)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:123)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: Null value appeared in non-nullable field:
top level row object
If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int).
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:289)
	... 25 more
20/10/14 13:56:54 INFO CoarseGrainedExecutorBackend: Got assigned task 2
20/10/14 13:56:54 INFO Executor: Running task 0.1 in stage 1.0 (TID 2)
20/10/14 13:56:54 INFO KafkaRDD: Computing topic airflow.public.motor_crash_violation_incidents, partition 0 offsets 15 -> 17
20/10/14 13:56:54 INFO ConsumerConfig: ConsumerConfig values: 
	auto.commit.interval.ms = 5000
	auto.offset.reset = none
	bootstrap.servers = [http://x.x.x.x:29092]
	check.crcs = true
	client.id = 
	connections.max.idle.ms = 540000
	default.api.timeout.ms = 60000
	enable.auto.commit = false
	exclude.internal.topics = true
	fetch.max.bytes = 52428800
	fetch.max.wait.ms = 500
	fetch.min.bytes = 1
	group.id = spark-executor-null
	heartbeat.interval.ms = 3000
	interceptor.classes = []
	internal.leave.group.on.close = true
	isolation.level = read_uncommitted
	key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	max.partition.fetch.bytes = 1048576
	max.poll.interval.ms = 300000
	max.poll.records = 500
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	send.buffer.bytes = 131072
	session.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	value.deserializer = class io.confluent.kafka.serializers.KafkaAvroDeserializer

20/10/14 13:56:54 INFO KafkaAvroDeserializerConfig: KafkaAvroDeserializerConfig values: 
	schema.registry.url = [http://x.x.x.x:8081]
	max.schemas.per.subject = 1000
	specific.avro.reader = false

20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'validate.non.null' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.partitionpath.field' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.compact.inline' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.delete.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.recordkey.field' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.upsert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.keygenerator.class' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.deltastreamer.source.kafka.topic' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.registry.url' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.insert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.precombine.field' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.embed.timeline.server' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.bulkinsert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.filesystem.view.type' was supplied but isn't a known config.
20/10/14 13:56:54 INFO AppInfoParser: Kafka version : 2.0.0
20/10/14 13:56:54 INFO AppInfoParser: Kafka commitId : 3402a8361b734732
20/10/14 13:56:54 INFO InternalKafkaConsumer: Initial fetch for spark-executor-null airflow.public.motor_crash_violation_incidents-0 15
20/10/14 13:56:54 INFO Metadata: Cluster ID: cA3sXVaIR-qlM1MPNNYnCw
20/10/14 13:56:54 ERROR Executor: Exception in task 0.1 in stage 1.0 (TID 2)
java.lang.RuntimeException: Error while encoding: java.lang.NullPointerException: Null value appeared in non-nullable field:
top level row object
If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int).
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(inc_id, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 0, inc_id), IntegerType), year, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 1, year), IntegerType), violation_desc, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 2, violation_desc), StringType), true, false), violation_code, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 3, violation_code), StringType), true, false), case_individual_id, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 4, case_individual_id), IntegerType), flag, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 5, flag), StringType), true, false), last_modified_ts, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 6, last_modified_ts), LongType)) AS before#0
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(inc_id, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 0, inc_id), IntegerType), year, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 1, year), IntegerType), violation_desc, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 2, violation_desc), StringType), true, false), violation_code, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 3, violation_code), StringType), true, false), case_individual_id, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 4, case_individual_id), IntegerType), flag, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 5, flag), StringType), true, false), last_modified_ts, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 6, last_modified_ts), LongType)) AS after#1
named_struct(version, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 0, version), StringType), true, false), connector, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 1, connector), StringType), true, false), name, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 2, name), StringType), true, false), ts_ms, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 3, ts_ms), LongType), snapshot, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 4, snapshot), StringType), true, false), db, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 5, db), StringType), true, false), schema, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 6, schema), StringType), true, false), table, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 7, table), StringType), true, false), txId, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 8, txId), LongType), lsn, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 9, lsn), LongType), xmin, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 10, xmin), LongType)) AS source#2
staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, op), StringType), true, false) AS op#3
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, ts_ms), LongType) AS ts_ms#4L
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(id, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 0, id), StringType), true, false), total_order, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 1, total_order), LongType), data_collection_order, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 2, data_collection_order), LongType)) AS transaction#5
	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:292)
	at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:594)
	at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:594)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:585)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
	at scala.collection.Iterator$class.isEmpty(Iterator.scala:331)
	at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1334)
	at org.apache.hudi.AvroConversionUtils$$anonfun$2.apply(AvroConversionUtils.scala:46)
	at org.apache.hudi.AvroConversionUtils$$anonfun$2.apply(AvroConversionUtils.scala:45)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:123)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: Null value appeared in non-nullable field:
top level row object
If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int).
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:289)
	... 25 more
20/10/14 13:56:54 INFO CoarseGrainedExecutorBackend: Got assigned task 3
20/10/14 13:56:54 INFO Executor: Running task 0.2 in stage 1.0 (TID 3)
20/10/14 13:56:54 INFO KafkaRDD: Computing topic airflow.public.motor_crash_violation_incidents, partition 0 offsets 15 -> 17
20/10/14 13:56:54 INFO ConsumerConfig: ConsumerConfig values: 
	auto.commit.interval.ms = 5000
	auto.offset.reset = none
	bootstrap.servers = [http://x.x.x.x:29092]
	check.crcs = true
	client.id = 
	connections.max.idle.ms = 540000
	default.api.timeout.ms = 60000
	enable.auto.commit = false
	exclude.internal.topics = true
	fetch.max.bytes = 52428800
	fetch.max.wait.ms = 500
	fetch.min.bytes = 1
	group.id = spark-executor-null
	heartbeat.interval.ms = 3000
	interceptor.classes = []
	internal.leave.group.on.close = true
	isolation.level = read_uncommitted
	key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	max.partition.fetch.bytes = 1048576
	max.poll.interval.ms = 300000
	max.poll.records = 500
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	send.buffer.bytes = 131072
	session.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	value.deserializer = class io.confluent.kafka.serializers.KafkaAvroDeserializer

20/10/14 13:56:54 INFO KafkaAvroDeserializerConfig: KafkaAvroDeserializerConfig values: 
	schema.registry.url = [http://x.x.x.x:8081]
	max.schemas.per.subject = 1000
	specific.avro.reader = false

20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'validate.non.null' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.partitionpath.field' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.compact.inline' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.delete.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.recordkey.field' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.upsert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.keygenerator.class' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.deltastreamer.source.kafka.topic' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.registry.url' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.insert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.precombine.field' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.embed.timeline.server' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.bulkinsert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.filesystem.view.type' was supplied but isn't a known config.
20/10/14 13:56:54 INFO AppInfoParser: Kafka version : 2.0.0
20/10/14 13:56:54 INFO AppInfoParser: Kafka commitId : 3402a8361b734732
20/10/14 13:56:54 INFO InternalKafkaConsumer: Initial fetch for spark-executor-null airflow.public.motor_crash_violation_incidents-0 15
20/10/14 13:56:54 INFO Metadata: Cluster ID: cA3sXVaIR-qlM1MPNNYnCw
20/10/14 13:56:54 ERROR Executor: Exception in task 0.2 in stage 1.0 (TID 3)
java.lang.RuntimeException: Error while encoding: java.lang.NullPointerException: Null value appeared in non-nullable field:
top level row object
If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int).
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(inc_id, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 0, inc_id), IntegerType), year, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 1, year), IntegerType), violation_desc, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 2, violation_desc), StringType), true, false), violation_code, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 3, violation_code), StringType), true, false), case_individual_id, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 4, case_individual_id), IntegerType), flag, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 5, flag), StringType), true, false), last_modified_ts, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 6, last_modified_ts), LongType)) AS before#0
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(inc_id, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 0, inc_id), IntegerType), year, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 1, year), IntegerType), violation_desc, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 2, violation_desc), StringType), true, false), violation_code, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 3, violation_code), StringType), true, false), case_individual_id, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 4, case_individual_id), IntegerType), flag, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 5, flag), StringType), true, false), last_modified_ts, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 6, last_modified_ts), LongType)) AS after#1
named_struct(version, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 0, version), StringType), true, false), connector, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 1, connector), StringType), true, false), name, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 2, name), StringType), true, false), ts_ms, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 3, ts_ms), LongType), snapshot, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 4, snapshot), StringType), true, false), db, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 5, db), StringType), true, false), schema, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 6, schema), StringType), true, false), table, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 7, table), StringType), true, false), txId, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 8, txId), LongType), lsn, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 9, lsn), LongType), xmin, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 10, xmin), LongType)) AS source#2
staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, op), StringType), true, false) AS op#3
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, ts_ms), LongType) AS ts_ms#4L
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(id, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 0, id), StringType), true, false), total_order, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 1, total_order), LongType), data_collection_order, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 2, data_collection_order), LongType)) AS transaction#5
	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:292)
	at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:594)
	at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:594)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:585)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
	at scala.collection.Iterator$class.isEmpty(Iterator.scala:331)
	at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1334)
	at org.apache.hudi.AvroConversionUtils$$anonfun$2.apply(AvroConversionUtils.scala:46)
	at org.apache.hudi.AvroConversionUtils$$anonfun$2.apply(AvroConversionUtils.scala:45)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:123)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: Null value appeared in non-nullable field:
top level row object
If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int).
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:289)
	... 25 more
20/10/14 13:56:54 INFO CoarseGrainedExecutorBackend: Got assigned task 4
20/10/14 13:56:54 INFO Executor: Running task 0.3 in stage 1.0 (TID 4)
20/10/14 13:56:54 INFO KafkaRDD: Computing topic airflow.public.motor_crash_violation_incidents, partition 0 offsets 15 -> 17
20/10/14 13:56:54 INFO ConsumerConfig: ConsumerConfig values: 
	auto.commit.interval.ms = 5000
	auto.offset.reset = none
	bootstrap.servers = [http://x.x.x.x:29092]
	check.crcs = true
	client.id = 
	connections.max.idle.ms = 540000
	default.api.timeout.ms = 60000
	enable.auto.commit = false
	exclude.internal.topics = true
	fetch.max.bytes = 52428800
	fetch.max.wait.ms = 500
	fetch.min.bytes = 1
	group.id = spark-executor-null
	heartbeat.interval.ms = 3000
	interceptor.classes = []
	internal.leave.group.on.close = true
	isolation.level = read_uncommitted
	key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	max.partition.fetch.bytes = 1048576
	max.poll.interval.ms = 300000
	max.poll.records = 500
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	send.buffer.bytes = 131072
	session.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	value.deserializer = class io.confluent.kafka.serializers.KafkaAvroDeserializer

20/10/14 13:56:54 INFO KafkaAvroDeserializerConfig: KafkaAvroDeserializerConfig values: 
	schema.registry.url = [http://x.x.x.x:8081]
	max.schemas.per.subject = 1000
	specific.avro.reader = false

20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'validate.non.null' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.partitionpath.field' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.compact.inline' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.delete.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.recordkey.field' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.upsert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.keygenerator.class' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.deltastreamer.source.kafka.topic' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.registry.url' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.insert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.precombine.field' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.embed.timeline.server' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.bulkinsert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:54 WARN ConsumerConfig: The configuration 'hoodie.filesystem.view.type' was supplied but isn't a known config.
20/10/14 13:56:54 INFO AppInfoParser: Kafka version : 2.0.0
20/10/14 13:56:54 INFO AppInfoParser: Kafka commitId : 3402a8361b734732
20/10/14 13:56:54 INFO InternalKafkaConsumer: Initial fetch for spark-executor-null airflow.public.motor_crash_violation_incidents-0 15
20/10/14 13:56:54 INFO Metadata: Cluster ID: cA3sXVaIR-qlM1MPNNYnCw
20/10/14 13:56:54 ERROR Executor: Exception in task 0.3 in stage 1.0 (TID 4)
java.lang.RuntimeException: Error while encoding: java.lang.NullPointerException: Null value appeared in non-nullable field:
top level row object
If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int).
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(inc_id, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 0, inc_id), IntegerType), year, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 1, year), IntegerType), violation_desc, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 2, violation_desc), StringType), true, false), violation_code, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 3, violation_code), StringType), true, false), case_individual_id, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 4, case_individual_id), IntegerType), flag, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 5, flag), StringType), true, false), last_modified_ts, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 0, before), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 6, last_modified_ts), LongType)) AS before#0
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(inc_id, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 0, inc_id), IntegerType), year, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 1, year), IntegerType), violation_desc, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 2, violation_desc), StringType), true, false), violation_code, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 3, violation_code), StringType), true, false), case_individual_id, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 4, case_individual_id), IntegerType), flag, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 5, flag), StringType), true, false), last_modified_ts, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 1, after), StructField(inc_id,IntegerType,false), StructField(year,IntegerType,true), StructField(violation_desc,StringType,true), StructField(violation_code,StringType,true), StructField(case_individual_id,IntegerType,true), StructField(flag,StringType,true), StructField(last_modified_ts,LongType,false)), 6, last_modified_ts), LongType)) AS after#1
named_struct(version, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 0, version), StringType), true, false), connector, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 1, connector), StringType), true, false), name, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 2, name), StringType), true, false), ts_ms, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 3, ts_ms), LongType), snapshot, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 4, snapshot), StringType), true, false), db, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 5, db), StringType), true, false), schema, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 6, schema), StringType), true, false), table, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 7, table), StringType), true, false), txId, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 8, txId), LongType), lsn, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 9, lsn), LongType), xmin, if (validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)).isNullAt) null else validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 2, source), StructField(version,StringType,false), StructField(connector,StringType,false), StructField(name,StringType,false), StructField(ts_ms,LongType,false), StructField(snapshot,StringType,true), StructField(db,StringType,false), StructField(schema,StringType,false), StructField(table,StringType,false), StructField(txId,LongType,true), StructField(lsn,LongType,true), StructField(xmin,LongType,true)), 10, xmin), LongType)) AS source#2
staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 3, op), StringType), true, false) AS op#3
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 4, ts_ms), LongType) AS ts_ms#4L
if (assertnotnull(input[0, org.apache.spark.sql.Row, true]).isNullAt) null else named_struct(id, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 0, id), StringType), true, false), total_order, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 1, total_order), LongType), data_collection_order, validateexternaltype(getexternalrowfield(validateexternaltype(getexternalrowfield(assertnotnull(input[0, org.apache.spark.sql.Row, true]), 5, transaction), StructField(id,StringType,false), StructField(total_order,LongType,false), StructField(data_collection_order,LongType,false)), 2, data_collection_order), LongType)) AS transaction#5
	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:292)
	at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:594)
	at org.apache.spark.sql.SparkSession$$anonfun$4.apply(SparkSession.scala:594)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
	at scala.collection.Iterator$$anon$11.next(Iterator.scala:410)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage1.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:585)
	at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
	at scala.collection.Iterator$class.isEmpty(Iterator.scala:331)
	at scala.collection.AbstractIterator.isEmpty(Iterator.scala:1334)
	at org.apache.hudi.AvroConversionUtils$$anonfun$2.apply(AvroConversionUtils.scala:46)
	at org.apache.hudi.AvroConversionUtils$$anonfun$2.apply(AvroConversionUtils.scala:45)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
	at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:823)
	at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
	at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:346)
	at org.apache.spark.rdd.RDD.iterator(RDD.scala:310)
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
	at org.apache.spark.scheduler.Task.run(Task.scala:123)
	at org.apache.spark.executor.Executor$TaskRunner$$anonfun$10.apply(Executor.scala:408)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1405)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:414)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException: Null value appeared in non-nullable field:
top level row object
If the schema is inferred from a Scala tuple/case class, or a Java bean, please try to use scala.Option[_] or other nullable types (e.g. java.lang.Integer instead of int/scala.Int).
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.writeFields_0_0$(Unknown Source)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown Source)
	at org.apache.spark.sql.catalyst.encoders.ExpressionEncoder.toRow(ExpressionEncoder.scala:289)
	... 25 more
20/10/14 13:56:54 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
20/10/14 13:56:54 INFO MemoryStore: MemoryStore cleared
20/10/14 13:56:54 INFO BlockManager: BlockManager stopped
20/10/14 13:56:54 INFO ShutdownHookManager: Shutdown hook called
20/10/14 13:56:54 INFO ShutdownHookManager: Deleting directory /mnt1/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-737d0243-54ad-4968-908e-20f495512503
20/10/14 13:56:54 INFO ShutdownHookManager: Deleting directory /mnt3/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-d3ac6c6d-677f-4882-b570-2e0ccb3448fe
20/10/14 13:56:54 INFO ShutdownHookManager: Deleting directory /mnt2/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-7adbd6a7-6620-4f09-8370-6bb62d9bf138
20/10/14 13:56:54 INFO ShutdownHookManager: Deleting directory /mnt/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-f7cebf7f-8d0d-4429-8d90-8e77ee1bfa47
End of LogType:stderr

LogType:stdout
Log Upload Time:Wed Oct 14 13:56:55 +0000 2020
LogLength:6400
Log Contents:
2020-10-14T13:56:40.514+0000: [GC (Allocation Failure) 2020-10-14T13:56:40.514+0000: [ParNew: 272512K->12449K(306560K), 0.0099352 secs] 272512K->12449K(987904K), 0.0100199 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2020-10-14T13:56:41.315+0000: [GC (Allocation Failure) 2020-10-14T13:56:41.315+0000: [ParNew: 284961K->29916K(306560K), 0.0131299 secs] 284961K->46302K(987904K), 0.0131910 secs] [Times: user=0.08 sys=0.01, real=0.01 secs] 
2020-10-14T13:56:41.328+0000: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(681344K)] 46302K(987904K), 0.0014691 secs] [Times: user=0.00 sys=0.01, real=0.01 secs] 
2020-10-14T13:56:41.330+0000: [CMS-concurrent-mark-start]
2020-10-14T13:56:41.331+0000: [CMS-concurrent-mark: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:41.331+0000: [CMS-concurrent-preclean-start]
2020-10-14T13:56:41.334+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:41.334+0000: [CMS-concurrent-abortable-preclean-start]
2020-10-14T13:56:41.764+0000: [CMS-concurrent-abortable-preclean: 0.226/0.430 secs] [Times: user=1.18 sys=0.02, real=0.43 secs] 
2020-10-14T13:56:41.765+0000: [GC (CMS Final Remark) [YG occupancy: 185775 K (306560 K)]2020-10-14T13:56:41.765+0000: [Rescan (parallel) , 0.0102940 secs]2020-10-14T13:56:41.775+0000: [weak refs processing, 0.0000263 secs]2020-10-14T13:56:41.775+0000: [class unloading, 0.0031113 secs]2020-10-14T13:56:41.778+0000: [scrub symbol table, 0.0019132 secs]2020-10-14T13:56:41.780+0000: [scrub string table, 0.0002369 secs][1 CMS-remark: 16386K(681344K)] 202161K(987904K), 0.0163562 secs] [Times: user=0.08 sys=0.00, real=0.02 secs] 
2020-10-14T13:56:41.781+0000: [CMS-concurrent-sweep-start]
2020-10-14T13:56:41.781+0000: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:41.781+0000: [CMS-concurrent-reset-start]
2020-10-14T13:56:41.877+0000: [CMS-concurrent-reset: 0.095/0.096 secs] [Times: user=0.44 sys=0.08, real=0.09 secs] 
2020-10-14T13:56:42.115+0000: [GC (Allocation Failure) 2020-10-14T13:56:42.115+0000: [ParNew: 302428K->29058K(306560K), 0.0452618 secs] 318814K->68205K(987904K), 0.0453361 secs] [Times: user=0.18 sys=0.02, real=0.05 secs] 
2020-10-14T13:56:43.185+0000: [GC (Allocation Failure) 2020-10-14T13:56:43.185+0000: [ParNew: 301570K->13038K(306560K), 0.0162222 secs] 340717K->70965K(987904K), 0.0162865 secs] [Times: user=0.08 sys=0.01, real=0.02 secs] 
2020-10-14T13:56:49.733+0000: [GC (Allocation Failure) 2020-10-14T13:56:49.733+0000: [ParNew: 285550K->14984K(306560K), 0.0086624 secs] 343477K->72911K(987904K), 0.0087346 secs] [Times: user=0.05 sys=0.01, real=0.01 secs] 
2020-10-14T13:56:49.743+0000: [GC (CMS Initial Mark) [1 CMS-initial-mark: 57926K(681344K)] 72911K(987904K), 0.0029672 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:49.746+0000: [CMS-concurrent-mark-start]
2020-10-14T13:56:49.768+0000: [CMS-concurrent-mark: 0.012/0.022 secs] [Times: user=0.09 sys=0.02, real=0.02 secs] 
2020-10-14T13:56:49.768+0000: [CMS-concurrent-preclean-start]
2020-10-14T13:56:49.770+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2020-10-14T13:56:49.771+0000: [CMS-concurrent-abortable-preclean-start]
2020-10-14T13:56:50.554+0000: [CMS-concurrent-abortable-preclean: 0.424/0.783 secs] [Times: user=1.90 sys=0.17, real=0.78 secs] 
2020-10-14T13:56:50.554+0000: [GC (CMS Final Remark) [YG occupancy: 164661 K (306560 K)]2020-10-14T13:56:50.554+0000: [Rescan (parallel) , 0.0074413 secs]2020-10-14T13:56:50.562+0000: [weak refs processing, 0.0000517 secs]2020-10-14T13:56:50.562+0000: [class unloading, 0.0053289 secs]2020-10-14T13:56:50.567+0000: [scrub symbol table, 0.0053127 secs]2020-10-14T13:56:50.572+0000: [scrub string table, 0.0005441 secs][1 CMS-remark: 57926K(681344K)] 222587K(987904K), 0.0190641 secs] [Times: user=0.07 sys=0.00, real=0.02 secs] 
2020-10-14T13:56:50.573+0000: [CMS-concurrent-sweep-start]
2020-10-14T13:56:50.577+0000: [CMS-concurrent-sweep: 0.004/0.004 secs] [Times: user=0.01 sys=0.01, real=0.01 secs] 
2020-10-14T13:56:50.578+0000: [CMS-concurrent-reset-start]
2020-10-14T13:56:50.610+0000: [CMS-concurrent-reset: 0.033/0.033 secs] [Times: user=0.07 sys=0.00, real=0.03 secs] 
2020-10-14T13:56:51.078+0000: [GC (Allocation Failure) 2020-10-14T13:56:51.078+0000: [ParNew: 287496K->34048K(306560K), 0.0091918 secs] 345233K->93063K(987904K), 0.0092580 secs] [Times: user=0.06 sys=0.01, real=0.00 secs] 
2020-10-14T13:56:52.018+0000: [GC (Allocation Failure) 2020-10-14T13:56:52.019+0000: [ParNew: 306560K->13440K(306560K), 0.0188523 secs] 365575K->99256K(987904K), 0.0189184 secs] [Times: user=0.10 sys=0.02, real=0.01 secs] 
2020-10-14T13:56:52.869+0000: [GC (Allocation Failure) 2020-10-14T13:56:52.869+0000: [ParNew: 285952K->10320K(306560K), 0.0059604 secs] 371768K->96136K(987904K), 0.0060227 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2020-10-14T13:56:54.330+0000: [GC (Allocation Failure) 2020-10-14T13:56:54.330+0000: [ParNew: 282832K->10801K(306560K), 0.0074379 secs] 368648K->96616K(987904K), 0.0075008 secs] [Times: user=0.05 sys=0.00, real=0.00 secs] 
Heap
 par new generation   total 306560K, used 67857K [0x000000031e400000, 0x00000003330a0000, 0x0000000347d90000)
  eden space 272512K,  20% used [0x000000031e400000, 0x0000000321bb8388, 0x000000032ee20000)
  from space 34048K,  31% used [0x0000000330f60000, 0x00000003319ec430, 0x00000003330a0000)
  to   space 34048K,   0% used [0x000000032ee20000, 0x000000032ee20000, 0x0000000330f60000)
 concurrent mark-sweep generation total 681344K, used 85815K [0x0000000347d90000, 0x00000003716f0000, 0x00000007c0000000)
 Metaspace       used 72468K, capacity 73381K, committed 73616K, reserved 1112064K
  class space    used 9821K, capacity 10076K, committed 10128K, reserved 1048576K
l fetch for spark-executor-null airflow.public.motor_crash_violation_incidents-0 15
2020-10-14 13:56:54,286 INFO [Executor task launch worker for task 4] org.apache.spark.streaming.kafka010.KafkaRDD:Computing topic airflow.public.motor_crash_violation_incidents, partition 0 offsets 15 -> 17
2020-10-14 13:56:54,301 INFO [Executor task launch worker for task 4] org.apache.spark.streaming.kafka010.InternalKafkaConsumer:Initial fetch for spark-executor-null airflow.public.motor_crash_violation_incidents-0 15
End of LogType:stdout



Container: container_1602165382040_0066_01_000004 on x.x.x.x_8041
============================================================================================================
LogType:stderr
Log Upload Time:Wed Oct 14 13:56:55 +0000 2020
LogLength:6335
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt2/yarn/usercache/hadoop/filecache/2685/__spark_libs__300261197764166949.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/10/14 13:56:38 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 24508@x.x.x.x
20/10/14 13:56:38 INFO SignalUtils: Registered signal handler for TERM
20/10/14 13:56:38 INFO SignalUtils: Registered signal handler for HUP
20/10/14 13:56:38 INFO SignalUtils: Registered signal handler for INT
20/10/14 13:56:39 INFO SecurityManager: Changing view acls to: yarn,hadoop
20/10/14 13:56:39 INFO SecurityManager: Changing modify acls to: yarn,hadoop
20/10/14 13:56:39 INFO SecurityManager: Changing view acls groups to: 
20/10/14 13:56:39 INFO SecurityManager: Changing modify acls groups to: 
20/10/14 13:56:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
20/10/14 13:56:39 INFO TransportClientFactory: Successfully created connection to x.x.x.x/x.x.x.x:37307 after 101 ms (0 ms spent in bootstraps)
20/10/14 13:56:40 INFO SecurityManager: Changing view acls to: yarn,hadoop
20/10/14 13:56:40 INFO SecurityManager: Changing modify acls to: yarn,hadoop
20/10/14 13:56:40 INFO SecurityManager: Changing view acls groups to: 
20/10/14 13:56:40 INFO SecurityManager: Changing modify acls groups to: 
20/10/14 13:56:40 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:40 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:40 INFO TransportClientFactory: Successfully created connection to x.x.x.x/x.x.x.x:37307 after 2 ms (0 ms spent in bootstraps)
20/10/14 13:56:40 INFO DiskBlockManager: Created local directory at /mnt/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-d1e615d5-ed72-429f-9eb9-9b2e2cc83bc8
20/10/14 13:56:40 INFO DiskBlockManager: Created local directory at /mnt3/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-c1e07bea-4e3d-42dd-8882-9eaf13127a13
20/10/14 13:56:40 INFO DiskBlockManager: Created local directory at /mnt1/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-db596555-9ef5-4fd8-94f7-262cc82cc17c
20/10/14 13:56:40 INFO DiskBlockManager: Created local directory at /mnt2/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-5558b3cb-0cba-4ccf-9aa5-8a6bdf75756f
20/10/14 13:56:40 INFO MemoryStore: MemoryStore started with capacity 10.9 GB
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:40 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:40 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@x.x.x.x:37307
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:40 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:40 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
20/10/14 13:56:40 INFO Executor: Starting executor ID 2 on host x.x.x.x
20/10/14 13:56:41 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37357.
20/10/14 13:56:41 INFO NettyBlockTransferService: Server created on x.x.x.x:37357
20/10/14 13:56:41 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/10/14 13:56:41 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(2, x.x.x.x, 37357, None)
20/10/14 13:56:41 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(2, x.x.x.x, 37357, None)
20/10/14 13:56:41 INFO BlockManager: external shuffle service port = 7337
20/10/14 13:56:41 INFO BlockManager: Registering executor with local external shuffle service.
20/10/14 13:56:41 INFO TransportClientFactory: Successfully created connection to x.x.x.x/172.31.30.142:7337 after 3 ms (0 ms spent in bootstraps)
20/10/14 13:56:41 INFO BlockManager: Initialized BlockManager: BlockManagerId(2, x.x.x.x, 37357, None)
20/10/14 13:56:43 INFO CoarseGrainedExecutorBackend: eagerFSInit: Eagerly initialized FileSystem at s3://does/not/exist in 2330 ms
20/10/14 13:56:54 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
20/10/14 13:56:54 INFO MemoryStore: MemoryStore cleared
20/10/14 13:56:54 INFO BlockManager: BlockManager stopped
20/10/14 13:56:54 INFO ShutdownHookManager: Shutdown hook called
End of LogType:stderr

LogType:stdout
Log Upload Time:Wed Oct 14 13:56:55 +0000 2020
LogLength:3172
Log Contents:
2020-10-14T13:56:39.236+0000: [GC (Allocation Failure) 2020-10-14T13:56:39.236+0000: [ParNew: 272512K->12232K(306560K), 0.0121922 secs] 272512K->12232K(987904K), 0.0122739 secs] [Times: user=0.03 sys=0.01, real=0.01 secs] 
2020-10-14T13:56:40.353+0000: [GC (Allocation Failure) 2020-10-14T13:56:40.353+0000: [ParNew: 284744K->29689K(306560K), 0.0151659 secs] 284744K->46075K(987904K), 0.0152373 secs] [Times: user=0.08 sys=0.02, real=0.02 secs] 
2020-10-14T13:56:40.369+0000: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(681344K)] 47147K(987904K), 0.0028528 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:40.372+0000: [CMS-concurrent-mark-start]
2020-10-14T13:56:40.373+0000: [CMS-concurrent-mark: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:40.373+0000: [CMS-concurrent-preclean-start]
2020-10-14T13:56:40.375+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 
2020-10-14T13:56:40.375+0000: [CMS-concurrent-abortable-preclean-start]
2020-10-14T13:56:40.924+0000: [CMS-concurrent-abortable-preclean: 0.415/0.549 secs] [Times: user=1.48 sys=0.03, real=0.54 secs] 
2020-10-14T13:56:40.925+0000: [GC (CMS Final Remark) [YG occupancy: 168380 K (306560 K)]2020-10-14T13:56:40.925+0000: [Rescan (parallel) , 0.0094974 secs]2020-10-14T13:56:40.934+0000: [weak refs processing, 0.0000297 secs]2020-10-14T13:56:40.934+0000: [class unloading, 0.0043939 secs]2020-10-14T13:56:40.939+0000: [scrub symbol table, 0.0038895 secs]2020-10-14T13:56:40.943+0000: [scrub string table, 0.0003967 secs][1 CMS-remark: 16386K(681344K)] 184766K(987904K), 0.0192756 secs] [Times: user=0.07 sys=0.00, real=0.02 secs] 
2020-10-14T13:56:40.944+0000: [CMS-concurrent-sweep-start]
2020-10-14T13:56:40.944+0000: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:40.947+0000: [CMS-concurrent-reset-start]
2020-10-14T13:56:41.137+0000: [CMS-concurrent-reset: 0.190/0.190 secs] [Times: user=0.58 sys=0.12, real=0.19 secs] 
2020-10-14T13:56:41.562+0000: [GC (Allocation Failure) 2020-10-14T13:56:41.562+0000: [ParNew: 302201K->29596K(306560K), 0.0870264 secs] 318587K->69328K(987904K), 0.0871131 secs] [Times: user=0.27 sys=0.01, real=0.09 secs] 
2020-10-14T13:56:43.049+0000: [GC (Allocation Failure) 2020-10-14T13:56:43.049+0000: [ParNew: 302108K->12761K(306560K), 0.0202815 secs] 341840K->71118K(987904K), 0.0203582 secs] [Times: user=0.10 sys=0.01, real=0.02 secs] 
Heap
 par new generation   total 306560K, used 133686K [0x000000031e400000, 0x00000003330a0000, 0x0000000347d90000)
  eden space 272512K,  44% used [0x000000031e400000, 0x0000000325a173b8, 0x000000032ee20000)
  from space 34048K,  37% used [0x000000032ee20000, 0x000000032fa964d0, 0x0000000330f60000)
  to   space 34048K,   0% used [0x0000000330f60000, 0x0000000330f60000, 0x00000003330a0000)
 concurrent mark-sweep generation total 681344K, used 58356K [0x0000000347d90000, 0x00000003716f0000, 0x00000007c0000000)
 Metaspace       used 41385K, capacity 41786K, committed 42132K, reserved 1085440K
  class space    used 5552K, capacity 5705K, committed 5780K, reserved 1048576K
End of LogType:stdout



Container: container_1602165382040_0066_01_000005 on x.x.x.x_8041
============================================================================================================
LogType:stderr
Log Upload Time:Wed Oct 14 13:56:55 +0000 2020
LogLength:14283
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt2/yarn/usercache/hadoop/filecache/2685/__spark_libs__300261197764166949.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/10/14 13:56:38 INFO CoarseGrainedExecutorBackend: Started daemon with process name: 24510@x.x.x.x
20/10/14 13:56:38 INFO SignalUtils: Registered signal handler for TERM
20/10/14 13:56:38 INFO SignalUtils: Registered signal handler for HUP
20/10/14 13:56:38 INFO SignalUtils: Registered signal handler for INT
20/10/14 13:56:39 INFO SecurityManager: Changing view acls to: yarn,hadoop
20/10/14 13:56:39 INFO SecurityManager: Changing modify acls to: yarn,hadoop
20/10/14 13:56:39 INFO SecurityManager: Changing view acls groups to: 
20/10/14 13:56:39 INFO SecurityManager: Changing modify acls groups to: 
20/10/14 13:56:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
20/10/14 13:56:39 INFO TransportClientFactory: Successfully created connection to x.x.x.x/x.x.x.x:37307 after 101 ms (0 ms spent in bootstraps)
20/10/14 13:56:40 INFO SecurityManager: Changing view acls to: yarn,hadoop
20/10/14 13:56:40 INFO SecurityManager: Changing modify acls to: yarn,hadoop
20/10/14 13:56:40 INFO SecurityManager: Changing view acls groups to: 
20/10/14 13:56:40 INFO SecurityManager: Changing modify acls groups to: 
20/10/14 13:56:40 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:40 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:40 INFO TransportClientFactory: Successfully created connection to x.x.x.x/x.x.x.x:37307 after 2 ms (0 ms spent in bootstraps)
20/10/14 13:56:40 INFO DiskBlockManager: Created local directory at /mnt1/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-082c4ce6-5f9b-4cba-a881-f8e591d28e58
20/10/14 13:56:40 INFO DiskBlockManager: Created local directory at /mnt/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-90079de7-af47-4681-8a14-6a3af45b735b
20/10/14 13:56:40 INFO DiskBlockManager: Created local directory at /mnt2/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-12a02b85-b3e1-4d9a-967a-20b143d3aa3c
20/10/14 13:56:40 INFO DiskBlockManager: Created local directory at /mnt3/yarn/usercache/hadoop/appcache/application_1602165382040_0066/blockmgr-5d62952f-576c-474a-bc9f-6ff465521fab
20/10/14 13:56:40 INFO MemoryStore: MemoryStore started with capacity 10.9 GB
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:40 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:40 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@x.x.x.x:37307
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:40 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:40 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:40 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
20/10/14 13:56:40 INFO Executor: Starting executor ID 3 on host x.x.x.x
20/10/14 13:56:41 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42123.
20/10/14 13:56:41 INFO NettyBlockTransferService: Server created on x.x.x.x:42123
20/10/14 13:56:41 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/10/14 13:56:41 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(3, x.x.x.x, 42123, None)
20/10/14 13:56:41 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(3, x.x.x.x, 42123, None)
20/10/14 13:56:41 INFO BlockManager: external shuffle service port = 7337
20/10/14 13:56:41 INFO BlockManager: Registering executor with local external shuffle service.
20/10/14 13:56:41 INFO TransportClientFactory: Successfully created connection to x.x.x.x/172.31.30.142:7337 after 2 ms (0 ms spent in bootstraps)
20/10/14 13:56:41 INFO BlockManager: Initialized BlockManager: BlockManagerId(3, x.x.x.x, 42123, None)
20/10/14 13:56:41 INFO CoarseGrainedExecutorBackend: Got assigned task 0
20/10/14 13:56:41 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
20/10/14 13:56:41 INFO Executor: Fetching s3://demoiags3/hudi/jars/hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar with timestamp 1602683783966
20/10/14 13:56:41 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:41 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:41 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:43 INFO CoarseGrainedExecutorBackend: eagerFSInit: Eagerly initialized FileSystem at s3://does/not/exist in 2428 ms
20/10/14 13:56:43 INFO S3NativeFileSystem: Opening 's3://demoiags3/hudi/jars/hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar' for reading
20/10/14 13:56:43 INFO Utils: Fetching s3://demoiags3/hudi/jars/hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar to /mnt1/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-292d68c1-ab5a-4e24-87c7-e615a1216824/fetchFileTemp3096751809242468109.tmp
20/10/14 13:56:44 INFO Utils: Copying /mnt1/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-292d68c1-ab5a-4e24-87c7-e615a1216824/-4264915221602683783966_cache to /mnt/yarn/usercache/hadoop/appcache/application_1602165382040_0066/container_1602165382040_0066_01_000005/./hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar
20/10/14 13:56:44 INFO Executor: Adding file:/mnt/yarn/usercache/hadoop/appcache/application_1602165382040_0066/container_1602165382040_0066_01_000005/./hudi-utilities-bundle_2.11-0.6.1-SNAPSHOT.jar to class loader
20/10/14 13:56:44 INFO TorrentBroadcast: Started reading broadcast variable 0
20/10/14 13:56:44 INFO TransportClientFactory: Successfully created connection to x.x.x.x/x.x.x.x:45721 after 2 ms (0 ms spent in bootstraps)
20/10/14 13:56:44 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.2 KB, free 10.9 GB)
20/10/14 13:56:44 INFO TorrentBroadcast: Reading broadcast variable 0 took 130 ms
20/10/14 13:56:45 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 5.2 KB, free 10.9 GB)
20/10/14 13:56:45 INFO KafkaRDD: Computing topic airflow.public.motor_crash_violation_incidents, partition 0 offsets 15 -> 17
20/10/14 13:56:45 INFO KafkaDataConsumer: Initializing cache 16 64 0.75
20/10/14 13:56:45 INFO ConsumerConfig: ConsumerConfig values: 
	auto.commit.interval.ms = 5000
	auto.offset.reset = none
	bootstrap.servers = [http://x.x.x.x:29092]
	check.crcs = true
	client.id = 
	connections.max.idle.ms = 540000
	default.api.timeout.ms = 60000
	enable.auto.commit = false
	exclude.internal.topics = true
	fetch.max.bytes = 52428800
	fetch.max.wait.ms = 500
	fetch.min.bytes = 1
	group.id = spark-executor-null
	heartbeat.interval.ms = 3000
	interceptor.classes = []
	internal.leave.group.on.close = true
	isolation.level = read_uncommitted
	key.deserializer = class org.apache.kafka.common.serialization.StringDeserializer
	max.partition.fetch.bytes = 1048576
	max.poll.interval.ms = 300000
	max.poll.records = 500
	metadata.max.age.ms = 300000
	metric.reporters = []
	metrics.num.samples = 2
	metrics.recording.level = INFO
	metrics.sample.window.ms = 30000
	partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor]
	receive.buffer.bytes = 65536
	reconnect.backoff.max.ms = 1000
	reconnect.backoff.ms = 50
	request.timeout.ms = 30000
	retry.backoff.ms = 100
	sasl.client.callback.handler.class = null
	sasl.jaas.config = null
	sasl.kerberos.kinit.cmd = /usr/bin/kinit
	sasl.kerberos.min.time.before.relogin = 60000
	sasl.kerberos.service.name = null
	sasl.kerberos.ticket.renew.jitter = 0.05
	sasl.kerberos.ticket.renew.window.factor = 0.8
	sasl.login.callback.handler.class = null
	sasl.login.class = null
	sasl.login.refresh.buffer.seconds = 300
	sasl.login.refresh.min.period.seconds = 60
	sasl.login.refresh.window.factor = 0.8
	sasl.login.refresh.window.jitter = 0.05
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
	send.buffer.bytes = 131072
	session.timeout.ms = 10000
	ssl.cipher.suites = null
	ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
	ssl.endpoint.identification.algorithm = https
	ssl.key.password = null
	ssl.keymanager.algorithm = SunX509
	ssl.keystore.location = null
	ssl.keystore.password = null
	ssl.keystore.type = JKS
	ssl.protocol = TLS
	ssl.provider = null
	ssl.secure.random.implementation = null
	ssl.trustmanager.algorithm = PKIX
	ssl.truststore.location = null
	ssl.truststore.password = null
	ssl.truststore.type = JKS
	value.deserializer = class io.confluent.kafka.serializers.KafkaAvroDeserializer

20/10/14 13:56:45 INFO KafkaAvroDeserializerConfig: KafkaAvroDeserializerConfig values: 
	schema.registry.url = [http://x.x.x.x:8081]
	max.schemas.per.subject = 1000
	specific.avro.reader = false

20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'validate.non.null' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.partitionpath.field' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.compact.inline' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.delete.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.recordkey.field' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.upsert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.keygenerator.class' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.deltastreamer.source.kafka.topic' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.deltastreamer.schemaprovider.registry.url' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.insert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.datasource.write.precombine.field' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.embed.timeline.server' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.bulkinsert.shuffle.parallelism' was supplied but isn't a known config.
20/10/14 13:56:45 WARN ConsumerConfig: The configuration 'hoodie.filesystem.view.type' was supplied but isn't a known config.
20/10/14 13:56:45 INFO AppInfoParser: Kafka version : 2.0.0
20/10/14 13:56:45 INFO AppInfoParser: Kafka commitId : 3402a8361b734732
20/10/14 13:56:45 INFO InternalKafkaConsumer: Initial fetch for spark-executor-null airflow.public.motor_crash_violation_incidents-0 15
20/10/14 13:56:45 INFO Metadata: Cluster ID: cA3sXVaIR-qlM1MPNNYnCw
20/10/14 13:56:46 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 1741 bytes result sent to driver
20/10/14 13:56:54 INFO CoarseGrainedExecutorBackend: Driver commanded a shutdown
20/10/14 13:56:54 INFO MemoryStore: MemoryStore cleared
20/10/14 13:56:54 INFO BlockManager: BlockManager stopped
20/10/14 13:56:54 INFO ShutdownHookManager: Shutdown hook called
20/10/14 13:56:54 INFO ShutdownHookManager: Deleting directory /mnt1/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-292d68c1-ab5a-4e24-87c7-e615a1216824
20/10/14 13:56:54 INFO ShutdownHookManager: Deleting directory /mnt2/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-68e76194-1840-49f4-9d8d-c366a54d5be8
20/10/14 13:56:54 INFO ShutdownHookManager: Deleting directory /mnt3/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-ed981205-efd3-4f0e-8668-3d4cf8224a6c
20/10/14 13:56:54 INFO ShutdownHookManager: Deleting directory /mnt/yarn/usercache/hadoop/appcache/application_1602165382040_0066/spark-06a96123-abb3-4df8-9de5-8c88df751f96
End of LogType:stderr

LogType:stdout
Log Upload Time:Wed Oct 14 13:56:55 +0000 2020
LogLength:5441
Log Contents:
2020-10-14T13:56:39.261+0000: [GC (Allocation Failure) 2020-10-14T13:56:39.261+0000: [ParNew: 272512K->12223K(306560K), 0.0141289 secs] 272512K->12223K(987904K), 0.0142444 secs] [Times: user=0.03 sys=0.00, real=0.02 secs] 
2020-10-14T13:56:40.297+0000: [GC (Allocation Failure) 2020-10-14T13:56:40.297+0000: [ParNew: 284735K->29833K(306560K), 0.0132291 secs] 284735K->46219K(987904K), 0.0132908 secs] [Times: user=0.07 sys=0.02, real=0.01 secs] 
2020-10-14T13:56:40.310+0000: [GC (CMS Initial Mark) [1 CMS-initial-mark: 16386K(681344K)] 46219K(987904K), 0.0014545 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:40.312+0000: [CMS-concurrent-mark-start]
2020-10-14T13:56:40.314+0000: [CMS-concurrent-mark: 0.002/0.002 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:40.314+0000: [CMS-concurrent-preclean-start]
2020-10-14T13:56:40.316+0000: [CMS-concurrent-preclean: 0.003/0.003 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 
2020-10-14T13:56:40.316+0000: [CMS-concurrent-abortable-preclean-start]
2020-10-14T13:56:40.898+0000: [CMS-concurrent-abortable-preclean: 0.349/0.582 secs] [Times: user=1.49 sys=0.03, real=0.58 secs] 
2020-10-14T13:56:40.899+0000: [GC (CMS Final Remark) [YG occupancy: 175097 K (306560 K)]2020-10-14T13:56:40.899+0000: [Rescan (parallel) , 0.0354100 secs]2020-10-14T13:56:40.934+0000: [weak refs processing, 0.0000345 secs]2020-10-14T13:56:40.934+0000: [class unloading, 0.0045765 secs]2020-10-14T13:56:40.939+0000: [scrub symbol table, 0.0039345 secs]2020-10-14T13:56:40.943+0000: [scrub string table, 0.0003820 secs][1 CMS-remark: 16386K(681344K)] 191483K(987904K), 0.0454014 secs] [Times: user=0.17 sys=0.00, real=0.04 secs] 
2020-10-14T13:56:40.944+0000: [CMS-concurrent-sweep-start]
2020-10-14T13:56:40.944+0000: [CMS-concurrent-sweep: 0.000/0.000 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:40.944+0000: [CMS-concurrent-reset-start]
2020-10-14T13:56:41.083+0000: [CMS-concurrent-reset: 0.139/0.139 secs] [Times: user=0.45 sys=0.12, real=0.14 secs] 
2020-10-14T13:56:41.374+0000: [GC (Allocation Failure) 2020-10-14T13:56:41.374+0000: [ParNew: 302345K->29315K(306560K), 0.1359638 secs] 318731K->69039K(987904K), 0.1360511 secs] [Times: user=0.53 sys=0.00, real=0.14 secs] 
2020-10-14T13:56:42.556+0000: [GC (Allocation Failure) 2020-10-14T13:56:42.556+0000: [ParNew: 301827K->9624K(306560K), 0.0153224 secs] 341551K->68244K(987904K), 0.0153878 secs] [Times: user=0.05 sys=0.00, real=0.01 secs] 
2020-10-14T13:56:43.946+0000: [GC (Allocation Failure) 2020-10-14T13:56:43.946+0000: [ParNew: 282136K->14635K(306560K), 0.0088503 secs] 340756K->73254K(987904K), 0.0089136 secs] [Times: user=0.06 sys=0.00, real=0.01 secs] 
2020-10-14T13:56:43.955+0000: [GC (CMS Initial Mark) [1 CMS-initial-mark: 58619K(681344K)] 75919K(987904K), 0.0024506 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:43.958+0000: [CMS-concurrent-mark-start]
2020-10-14T13:56:43.971+0000: [CMS-concurrent-mark: 0.010/0.013 secs] [Times: user=0.07 sys=0.01, real=0.01 secs] 
2020-10-14T13:56:43.971+0000: [CMS-concurrent-preclean-start]
2020-10-14T13:56:43.974+0000: [CMS-concurrent-preclean: 0.002/0.002 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:43.974+0000: [CMS-concurrent-abortable-preclean-start]
2020-10-14T13:56:44.918+0000: [CMS-concurrent-abortable-preclean: 0.439/0.944 secs] [Times: user=2.43 sys=0.17, real=0.95 secs] 
2020-10-14T13:56:44.918+0000: [GC (CMS Final Remark) [YG occupancy: 170034 K (306560 K)]2020-10-14T13:56:44.918+0000: [Rescan (parallel) , 0.0087981 secs]2020-10-14T13:56:44.927+0000: [weak refs processing, 0.0000944 secs]2020-10-14T13:56:44.927+0000: [class unloading, 0.0051090 secs]2020-10-14T13:56:44.932+0000: [scrub symbol table, 0.0054921 secs]2020-10-14T13:56:44.938+0000: [scrub string table, 0.0005027 secs][1 CMS-remark: 58619K(681344K)] 228654K(987904K), 0.0203656 secs] [Times: user=0.09 sys=0.00, real=0.02 secs] 
2020-10-14T13:56:44.939+0000: [CMS-concurrent-sweep-start]
2020-10-14T13:56:44.942+0000: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2020-10-14T13:56:44.942+0000: [CMS-concurrent-reset-start]
2020-10-14T13:56:44.975+0000: [CMS-concurrent-reset: 0.033/0.033 secs] [Times: user=0.10 sys=0.00, real=0.03 secs] 
2020-10-14T13:56:45.400+0000: [GC (Allocation Failure) 2020-10-14T13:56:45.400+0000: [ParNew: 287147K->33022K(306560K), 0.0086573 secs] 345576K->91452K(987904K), 0.0087255 secs] [Times: user=0.04 sys=0.00, real=0.01 secs] 
2020-10-14T13:56:54.445+0000: [GC (Allocation Failure) 2020-10-14T13:56:54.445+0000: [ParNew: 305534K->11726K(306560K), 0.0279222 secs] 363964K->95676K(987904K), 0.0279858 secs] [Times: user=0.08 sys=0.02, real=0.03 secs] 
Heap
 par new generation   total 306560K, used 35185K [0x000000031e400000, 0x00000003330a0000, 0x0000000347d90000)
  eden space 272512K,   8% used [0x000000031e400000, 0x000000031fae8be8, 0x000000032ee20000)
  from space 34048K,  34% used [0x0000000330f60000, 0x0000000331ad3a80, 0x00000003330a0000)
  to   space 34048K,   0% used [0x000000032ee20000, 0x000000032ee20000, 0x0000000330f60000)
 concurrent mark-sweep generation total 681344K, used 83949K [0x0000000347d90000, 0x00000003716f0000, 0x00000007c0000000)
 Metaspace       used 58011K, capacity 58749K, committed 59152K, reserved 1099776K
  class space    used 7747K, capacity 7934K, committed 7952K, reserved 1048576K
End of LogType:stdout



Container: container_1602165382040_0066_01_000001 on x.x.x.x_8041
============================================================================================================
LogType:stderr
Log Upload Time:Wed Oct 14 13:56:55 +0000 2020
LogLength:39310
Log Contents:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/mnt2/yarn/usercache/hadoop/filecache/2685/__spark_libs__300261197764166949.zip/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
20/10/14 13:56:33 INFO SignalUtils: Registered signal handler for TERM
20/10/14 13:56:33 INFO SignalUtils: Registered signal handler for HUP
20/10/14 13:56:33 INFO SignalUtils: Registered signal handler for INT
20/10/14 13:56:33 INFO SecurityManager: Changing view acls to: yarn,hadoop
20/10/14 13:56:33 INFO SecurityManager: Changing modify acls to: yarn,hadoop
20/10/14 13:56:33 INFO SecurityManager: Changing view acls groups to: 
20/10/14 13:56:33 INFO SecurityManager: Changing modify acls groups to: 
20/10/14 13:56:33 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yarn, hadoop); groups with view permissions: Set(); users  with modify permissions: Set(yarn, hadoop); groups with modify permissions: Set()
20/10/14 13:56:34 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:34 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:34 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:34 INFO deprecation: mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
20/10/14 13:56:34 INFO deprecation: mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
20/10/14 13:56:34 INFO deprecation: mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
20/10/14 13:56:34 INFO ApplicationMaster: Preparing Local resources
20/10/14 13:56:35 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1602165382040_0066_000001
20/10/14 13:56:35 INFO RMProxy: Connecting to ResourceManager at x.x.x.x/x.x.x.x:8030
20/10/14 13:56:35 INFO YarnRMClient: Registering the ApplicationMaster
20/10/14 13:56:36 INFO TransportClientFactory: Successfully created connection to x.x.x.x/x.x.x.x:37307 after 99 ms (0 ms spent in bootstraps)
20/10/14 13:56:36 INFO ApplicationMaster: 
===============================================================================
YARN executor launch context:
  env:
    CLASSPATH -> /usr/lib/hadoop-lzo/lib/*:/usr/lib/hadoop/hadoop-aws.jar:/usr/share/aws/aws-java-sdk/*:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/*:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/goodies/lib/emr-spark-goodies.jar:/usr/share/aws/emr/security/conf:/usr/share/aws/emr/security/lib/*:/usr/share/aws/hmclient/lib/aws-glue-datacatalog-spark-client.jar:/usr/share/java/Hive-JSON-Serde/hive-openx-serde.jar:/usr/share/aws/sagemaker-spark-sdk/lib/sagemaker-spark-sdk.jar:/usr/share/aws/emr/s3select/lib/emr-s3-select-spark-connector.jar<CPS>{{PWD}}<CPS>{{PWD}}/__spark_conf__<CPS>{{PWD}}/__spark_libs__/*<CPS>$HADOOP_CONF_DIR<CPS>$HADOOP_COMMON_HOME/*<CPS>$HADOOP_COMMON_HOME/lib/*<CPS>$HADOOP_HDFS_HOME/*<CPS>$HADOOP_HDFS_HOME/lib/*<CPS>$HADOOP_MAPRED_HOME/*<CPS>$HADOOP_MAPRED_HOME/lib/*<CPS>$HADOOP_YARN_HOME/*<CPS>$HADOOP_YARN_HOME/lib/*<CPS>/usr/lib/hadoop-lzo/lib/*<CPS>/usr/share/aws/emr/emrfs/conf<CPS>/usr/share/aws/emr/emrfs/lib/*<CPS>/usr/share/aws/emr/emrfs/auxlib/*<CPS>/usr/share/aws/emr/lib/*<CPS>/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar<CPS>/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar<CPS>/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar<CPS>/usr/share/aws/emr/cloudwatch-sink/lib/*<CPS>/usr/share/aws/aws-java-sdk/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*<CPS>$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*<CPS>/usr/lib/hadoop-lzo/lib/*<CPS>/usr/share/aws/emr/emrfs/conf<CPS>/usr/share/aws/emr/emrfs/lib/*<CPS>/usr/share/aws/emr/emrfs/auxlib/*<CPS>/usr/share/aws/emr/lib/*<CPS>/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar<CPS>/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar<CPS>/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar<CPS>/usr/share/aws/emr/cloudwatch-sink/lib/*<CPS>/usr/share/aws/aws-java-sdk/*<CPS>{{PWD}}/__spark_conf__/__hadoop_conf__
    SPARK_YARN_STAGING_DIR -> hdfs://x.x.x.x:8020/user/hadoop/.sparkStaging/application_1602165382040_0066
    SPARK_USER -> hadoop
    SPARK_PUBLIC_DNS -> x.x.x.x

  command:
    LD_LIBRARY_PATH=\"/usr/lib/hadoop/lib/native:/usr/lib/hadoop-lzo/lib/native:$LD_LIBRARY_PATH\" \ 
      {{JAVA_HOME}}/bin/java \ 
      -server \ 
      -Xmx18971m \ 
      '-verbose:gc' \ 
      '-XX:+PrintGCDetails' \ 
      '-XX:+PrintGCDateStamps' \ 
      '-XX:+UseConcMarkSweepGC' \ 
      '-XX:CMSInitiatingOccupancyFraction=70' \ 
      '-XX:MaxHeapFreeRatio=70' \ 
      '-XX:+CMSClassUnloadingEnabled' \ 
      '-XX:OnOutOfMemoryError=kill -9 %p' \ 
      -Djava.io.tmpdir={{PWD}}/tmp \ 
      '-Dspark.driver.port=37307' \ 
      '-Dspark.history.ui.port=18080' \ 
      -Dspark.yarn.app.container.log.dir=<LOG_DIR> \ 
      org.apache.spark.executor.CoarseGrainedExecutorBackend \ 
      --driver-url \ 
      spark://CoarseGrainedScheduler@x.x.x.x:37307 \ 
      --executor-id \ 
      <executorId> \ 
      --hostname \ 
      <hostname> \ 
      --cores \ 
      4 \ 
      --app-id \ 
      application_1602165382040_0066 \ 
      --user-class-path \ 
      file:$PWD/__app__.jar \ 
      --user-class-path \ 
      file:$PWD/hudi-spark-bundle_2.11-0.6.1-SNAPSHOT.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.spark_spark-avro_2.11-2.4.4.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.hadoop_hadoop-aws-2.7.3.jar \ 
      --user-class-path \ 
      file:$PWD/org.spark-project.spark_unused-1.0.0.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.hadoop_hadoop-common-2.7.3.jar \ 
      --user-class-path \ 
      file:$PWD/com.fasterxml.jackson.core_jackson-databind-2.2.3.jar \ 
      --user-class-path \ 
      file:$PWD/com.fasterxml.jackson.core_jackson-annotations-2.2.3.jar \ 
      --user-class-path \ 
      file:$PWD/com.amazonaws_aws-java-sdk-1.7.4.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.hadoop_hadoop-annotations-2.7.3.jar \ 
      --user-class-path \ 
      file:$PWD/com.google.guava_guava-11.0.2.jar \ 
      --user-class-path \ 
      file:$PWD/commons-cli_commons-cli-1.2.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.commons_commons-math3-3.1.1.jar \ 
      --user-class-path \ 
      file:$PWD/xmlenc_xmlenc-0.52.jar \ 
      --user-class-path \ 
      file:$PWD/commons-httpclient_commons-httpclient-3.1.jar \ 
      --user-class-path \ 
      file:$PWD/commons-codec_commons-codec-1.4.jar \ 
      --user-class-path \ 
      file:$PWD/commons-io_commons-io-2.4.jar \ 
      --user-class-path \ 
      file:$PWD/commons-net_commons-net-3.1.jar \ 
      --user-class-path \ 
      file:$PWD/commons-collections_commons-collections-3.2.2.jar \ 
      --user-class-path \ 
      file:$PWD/javax.servlet_servlet-api-2.5.jar \ 
      --user-class-path \ 
      file:$PWD/org.mortbay.jetty_jetty-6.1.26.jar \ 
      --user-class-path \ 
      file:$PWD/org.mortbay.jetty_jetty-util-6.1.26.jar \ 
      --user-class-path \ 
      file:$PWD/com.sun.jersey_jersey-core-1.9.jar \ 
      --user-class-path \ 
      file:$PWD/com.sun.jersey_jersey-json-1.9.jar \ 
      --user-class-path \ 
      file:$PWD/com.sun.jersey_jersey-server-1.9.jar \ 
      --user-class-path \ 
      file:$PWD/commons-logging_commons-logging-1.1.3.jar \ 
      --user-class-path \ 
      file:$PWD/log4j_log4j-1.2.17.jar \ 
      --user-class-path \ 
      file:$PWD/net.java.dev.jets3t_jets3t-0.9.0.jar \ 
      --user-class-path \ 
      file:$PWD/commons-lang_commons-lang-2.6.jar \ 
      --user-class-path \ 
      file:$PWD/commons-configuration_commons-configuration-1.6.jar \ 
      --user-class-path \ 
      file:$PWD/org.slf4j_slf4j-api-1.7.10.jar \ 
      --user-class-path \ 
      file:$PWD/org.codehaus.jackson_jackson-core-asl-1.9.13.jar \ 
      --user-class-path \ 
      file:$PWD/org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.avro_avro-1.7.4.jar \ 
      --user-class-path \ 
      file:$PWD/com.google.protobuf_protobuf-java-2.5.0.jar \ 
      --user-class-path \ 
      file:$PWD/com.google.code.gson_gson-2.2.4.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.hadoop_hadoop-auth-2.7.3.jar \ 
      --user-class-path \ 
      file:$PWD/com.jcraft_jsch-0.1.42.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.curator_curator-client-2.7.1.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.curator_curator-recipes-2.7.1.jar \ 
      --user-class-path \ 
      file:$PWD/com.google.code.findbugs_jsr305-3.0.0.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.htrace_htrace-core-3.1.0-incubating.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.zookeeper_zookeeper-3.4.6.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.commons_commons-compress-1.4.1.jar \ 
      --user-class-path \ 
      file:$PWD/org.codehaus.jettison_jettison-1.1.jar \ 
      --user-class-path \ 
      file:$PWD/com.sun.xml.bind_jaxb-impl-2.2.3-1.jar \ 
      --user-class-path \ 
      file:$PWD/org.codehaus.jackson_jackson-jaxrs-1.9.13.jar \ 
      --user-class-path \ 
      file:$PWD/org.codehaus.jackson_jackson-xc-1.9.13.jar \ 
      --user-class-path \ 
      file:$PWD/javax.xml.bind_jaxb-api-2.2.2.jar \ 
      --user-class-path \ 
      file:$PWD/javax.xml.stream_stax-api-1.0-2.jar \ 
      --user-class-path \ 
      file:$PWD/javax.activation_activation-1.1.jar \ 
      --user-class-path \ 
      file:$PWD/asm_asm-3.2.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.httpcomponents_httpclient-4.2.5.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.httpcomponents_httpcore-4.2.5.jar \ 
      --user-class-path \ 
      file:$PWD/com.jamesmurty.utils_java-xmlbuilder-0.4.jar \ 
      --user-class-path \ 
      file:$PWD/commons-digester_commons-digester-1.8.jar \ 
      --user-class-path \ 
      file:$PWD/commons-beanutils_commons-beanutils-core-1.8.0.jar \ 
      --user-class-path \ 
      file:$PWD/commons-beanutils_commons-beanutils-1.7.0.jar \ 
      --user-class-path \ 
      file:$PWD/com.thoughtworks.paranamer_paranamer-2.3.jar \ 
      --user-class-path \ 
      file:$PWD/org.xerial.snappy_snappy-java-1.0.4.1.jar \ 
      --user-class-path \ 
      file:$PWD/org.tukaani_xz-1.0.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.directory.server_apacheds-kerberos-codec-2.0.0-M15.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.curator_curator-framework-2.7.1.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.directory.server_apacheds-i18n-2.0.0-M15.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.directory.api_api-asn1-api-1.0.0-M20.jar \ 
      --user-class-path \ 
      file:$PWD/org.apache.directory.api_api-util-1.0.0-M20.jar \ 
      --user-class-path \ 
      file:$PWD/org.slf4j_slf4j-log4j12-1.7.10.jar \ 
      --user-class-path \ 
      file:$PWD/io.netty_netty-3.6.2.Final.jar \ 
      --user-class-path \ 
      file:$PWD/javax.servlet.jsp_jsp-api-2.1.jar \ 
      --user-class-path \ 
      file:$PWD/jline_jline-0.9.94.jar \ 
      --user-class-path \ 
      file:$PWD/junit_junit-4.11.jar \ 
      --user-class-path \ 
      file:$PWD/org.hamcrest_hamcrest-core-1.3.jar \ 
      --user-class-path \ 
      file:$PWD/com.fasterxml.jackson.core_jackson-core-2.2.3.jar \ 
      --user-class-path \ 
      file:$PWD/joda-time_joda-time-2.10.6.jar \ 
      1><LOG_DIR>/stdout \ 
      2><LOG_DIR>/stderr

  resources:
    org.apache.hadoop_hadoop-common-2.7.3.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.hadoop_hadoop-common-2.7.3.jar" } size: 3479293 timestamp: 1602683787336 type: FILE visibility: PRIVATE
    commons-digester_commons-digester-1.8.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-digester_commons-digester-1.8.jar" } size: 143602 timestamp: 1602683788379 type: FILE visibility: PRIVATE
    commons-cli_commons-cli-1.2.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-cli_commons-cli-1.2.jar" } size: 41123 timestamp: 1602683787486 type: FILE visibility: PRIVATE
    commons-httpclient_commons-httpclient-3.1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-httpclient_commons-httpclient-3.1.jar" } size: 305001 timestamp: 1602683787550 type: FILE visibility: PRIVATE
    com.sun.jersey_jersey-server-1.9.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.sun.jersey_jersey-server-1.9.jar" } size: 713089 timestamp: 1602683787761 type: FILE visibility: PRIVATE
    org.apache.curator_curator-framework-2.7.1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.curator_curator-framework-2.7.1.jar" } size: 186273 timestamp: 1602683788514 type: FILE visibility: PRIVATE
    commons-beanutils_commons-beanutils-1.7.0.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-beanutils_commons-beanutils-1.7.0.jar" } size: 188671 timestamp: 1602683788418 type: FILE visibility: PRIVATE
    org.codehaus.jettison_jettison-1.1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.codehaus.jettison_jettison-1.1.jar" } size: 67758 timestamp: 1602683788167 type: FILE visibility: PRIVATE
    com.fasterxml.jackson.core_jackson-core-2.2.3.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.fasterxml.jackson.core_jackson-core-2.2.3.jar" } size: 192699 timestamp: 1602683789102 type: FILE visibility: PRIVATE
    org.apache.hadoop_hadoop-annotations-2.7.3.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.hadoop_hadoop-annotations-2.7.3.jar" } size: 40863 timestamp: 1602683787441 type: FILE visibility: PRIVATE
    com.thoughtworks.paranamer_paranamer-2.3.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.thoughtworks.paranamer_paranamer-2.3.jar" } size: 29555 timestamp: 1602683788437 type: FILE visibility: PRIVATE
    net.java.dev.jets3t_jets3t-0.9.0.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/net.java.dev.jets3t_jets3t-0.9.0.jar" } size: 539735 timestamp: 1602683787821 type: FILE visibility: PRIVATE
    org.codehaus.jackson_jackson-core-asl-1.9.13.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.codehaus.jackson_jackson-core-asl-1.9.13.jar" } size: 232248 timestamp: 1602683787902 type: FILE visibility: PRIVATE
    org.apache.directory.server_apacheds-kerberos-codec-2.0.0-M15.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.directory.server_apacheds-kerberos-codec-2.0.0-M15.jar" } size: 691479 timestamp: 1602683788495 type: FILE visibility: PRIVATE
    org.apache.hadoop_hadoop-aws-2.7.3.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.hadoop_hadoop-aws-2.7.3.jar" } size: 126287 timestamp: 1602683787288 type: FILE visibility: PRIVATE
    com.sun.xml.bind_jaxb-impl-2.2.3-1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.sun.xml.bind_jaxb-impl-2.2.3-1.jar" } size: 890168 timestamp: 1602683788188 type: FILE visibility: PRIVATE
    org.apache.hadoop_hadoop-auth-2.7.3.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.hadoop_hadoop-auth-2.7.3.jar" } size: 94150 timestamp: 1602683788006 type: FILE visibility: PRIVATE
    joda-time_joda-time-2.10.6.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/joda-time_joda-time-2.10.6.jar" } size: 643778 timestamp: 1602683789121 type: FILE visibility: PRIVATE
    javax.servlet_servlet-api-2.5.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/javax.servlet_servlet-api-2.5.jar" } size: 105112 timestamp: 1602683787655 type: FILE visibility: PRIVATE
    org.codehaus.jackson_jackson-xc-1.9.13.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.codehaus.jackson_jackson-xc-1.9.13.jar" } size: 27084 timestamp: 1602683788226 type: FILE visibility: PRIVATE
    com.google.code.gson_gson-2.2.4.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.google.code.gson_gson-2.2.4.jar" } size: 190432 timestamp: 1602683787986 type: FILE visibility: PRIVATE
    __spark_conf__ -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/__spark_conf__.zip" } size: 273483 timestamp: 1602683789258 type: ARCHIVE visibility: PRIVATE
    log4j_log4j-1.2.17.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/log4j_log4j-1.2.17.jar" } size: 489884 timestamp: 1602683787800 type: FILE visibility: PRIVATE
    commons-collections_commons-collections-3.2.2.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-collections_commons-collections-3.2.2.jar" } size: 588337 timestamp: 1602683787633 type: FILE visibility: PRIVATE
    junit_junit-4.11.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/junit_junit-4.11.jar" } size: 245039 timestamp: 1602683789066 type: FILE visibility: PRIVATE
    com.fasterxml.jackson.core_jackson-annotations-2.2.3.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.fasterxml.jackson.core_jackson-annotations-2.2.3.jar" } size: 33483 timestamp: 1602683787379 type: FILE visibility: PRIVATE
    org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.codehaus.jackson_jackson-mapper-asl-1.9.13.jar" } size: 780664 timestamp: 1602683787923 type: FILE visibility: PRIVATE
    org.apache.commons_commons-compress-1.4.1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.commons_commons-compress-1.4.1.jar" } size: 241367 timestamp: 1602683788148 type: FILE visibility: PRIVATE
    org.apache.zookeeper_zookeeper-3.4.6.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.zookeeper_zookeeper-3.4.6.jar" } size: 792964 timestamp: 1602683788128 type: FILE visibility: PRIVATE
    org.apache.commons_commons-math3-3.1.1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.commons_commons-math3-3.1.1.jar" } size: 1599627 timestamp: 1602683787509 type: FILE visibility: PRIVATE
    commons-codec_commons-codec-1.4.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-codec_commons-codec-1.4.jar" } size: 58160 timestamp: 1602683787571 type: FILE visibility: PRIVATE
    xmlenc_xmlenc-0.52.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/xmlenc_xmlenc-0.52.jar" } size: 15010 timestamp: 1602683787529 type: FILE visibility: PRIVATE
    commons-io_commons-io-2.4.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-io_commons-io-2.4.jar" } size: 185140 timestamp: 1602683787591 type: FILE visibility: PRIVATE
    org.apache.spark_spark-avro_2.11-2.4.4.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.spark_spark-avro_2.11-2.4.4.jar" } size: 187318 timestamp: 1602683787264 type: FILE visibility: PRIVATE
    com.google.code.findbugs_jsr305-3.0.0.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.google.code.findbugs_jsr305-3.0.0.jar" } size: 33031 timestamp: 1602683788086 type: FILE visibility: PRIVATE
    com.google.protobuf_protobuf-java-2.5.0.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.google.protobuf_protobuf-java-2.5.0.jar" } size: 533455 timestamp: 1602683787965 type: FILE visibility: PRIVATE
    org.mortbay.jetty_jetty-6.1.26.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.mortbay.jetty_jetty-6.1.26.jar" } size: 539912 timestamp: 1602683787676 type: FILE visibility: PRIVATE
    com.sun.jersey_jersey-core-1.9.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.sun.jersey_jersey-core-1.9.jar" } size: 458739 timestamp: 1602683787718 type: FILE visibility: PRIVATE
    org.tukaani_xz-1.0.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.tukaani_xz-1.0.jar" } size: 94672 timestamp: 1602683788475 type: FILE visibility: PRIVATE
    javax.activation_activation-1.1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/javax.activation_activation-1.1.jar" } size: 62983 timestamp: 1602683788283 type: FILE visibility: PRIVATE
    org.codehaus.jackson_jackson-jaxrs-1.9.13.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.codehaus.jackson_jackson-jaxrs-1.9.13.jar" } size: 18336 timestamp: 1602683788206 type: FILE visibility: PRIVATE
    org.apache.httpcomponents_httpclient-4.2.5.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.httpcomponents_httpclient-4.2.5.jar" } size: 433368 timestamp: 1602683788321 type: FILE visibility: PRIVATE
    org.spark-project.spark_unused-1.0.0.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.spark-project.spark_unused-1.0.0.jar" } size: 2777 timestamp: 1602683787309 type: FILE visibility: PRIVATE
    commons-logging_commons-logging-1.1.3.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-logging_commons-logging-1.1.3.jar" } size: 62050 timestamp: 1602683787781 type: FILE visibility: PRIVATE
    commons-beanutils_commons-beanutils-core-1.8.0.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-beanutils_commons-beanutils-core-1.8.0.jar" } size: 206035 timestamp: 1602683788400 type: FILE visibility: PRIVATE
    com.amazonaws_aws-java-sdk-1.7.4.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.amazonaws_aws-java-sdk-1.7.4.jar" } size: 11948376 timestamp: 1602683787423 type: FILE visibility: PRIVATE
    org.apache.htrace_htrace-core-3.1.0-incubating.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.htrace_htrace-core-3.1.0-incubating.jar" } size: 1475955 timestamp: 1602683788108 type: FILE visibility: PRIVATE
    javax.servlet.jsp_jsp-api-2.1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/javax.servlet.jsp_jsp-api-2.1.jar" } size: 100636 timestamp: 1602683789029 type: FILE visibility: PRIVATE
    org.slf4j_slf4j-api-1.7.10.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.slf4j_slf4j-api-1.7.10.jar" } size: 32119 timestamp: 1602683787881 type: FILE visibility: PRIVATE
    org.apache.avro_avro-1.7.4.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.avro_avro-1.7.4.jar" } size: 303139 timestamp: 1602683787944 type: FILE visibility: PRIVATE
    commons-configuration_commons-configuration-1.6.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-configuration_commons-configuration-1.6.jar" } size: 298829 timestamp: 1602683787862 type: FILE visibility: PRIVATE
    org.apache.directory.server_apacheds-i18n-2.0.0-M15.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.directory.server_apacheds-i18n-2.0.0-M15.jar" } size: 44925 timestamp: 1602683788534 type: FILE visibility: PRIVATE
    org.apache.directory.api_api-asn1-api-1.0.0-M20.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.directory.api_api-asn1-api-1.0.0-M20.jar" } size: 16560 timestamp: 1602683788553 type: FILE visibility: PRIVATE
    org.apache.curator_curator-recipes-2.7.1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.curator_curator-recipes-2.7.1.jar" } size: 270342 timestamp: 1602683788066 type: FILE visibility: PRIVATE
    javax.xml.stream_stax-api-1.0-2.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/javax.xml.stream_stax-api-1.0-2.jar" } size: 23346 timestamp: 1602683788265 type: FILE visibility: PRIVATE
    hudi-spark-bundle_2.11-0.6.1-SNAPSHOT.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/hudi-spark-bundle_2.11-0.6.1-SNAPSHOT.jar" } size: 34995129 timestamp: 1602683787241 type: FILE visibility: PRIVATE
    org.hamcrest_hamcrest-core-1.3.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.hamcrest_hamcrest-core-1.3.jar" } size: 45024 timestamp: 1602683789083 type: FILE visibility: PRIVATE
    com.google.guava_guava-11.0.2.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.google.guava_guava-11.0.2.jar" } size: 1648200 timestamp: 1602683787465 type: FILE visibility: PRIVATE
    javax.xml.bind_jaxb-api-2.2.2.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/javax.xml.bind_jaxb-api-2.2.2.jar" } size: 105134 timestamp: 1602683788245 type: FILE visibility: PRIVATE
    org.apache.directory.api_api-util-1.0.0-M20.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.directory.api_api-util-1.0.0-M20.jar" } size: 79912 timestamp: 1602683788972 type: FILE visibility: PRIVATE
    commons-lang_commons-lang-2.6.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-lang_commons-lang-2.6.jar" } size: 284220 timestamp: 1602683787841 type: FILE visibility: PRIVATE
    org.xerial.snappy_snappy-java-1.0.4.1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.xerial.snappy_snappy-java-1.0.4.1.jar" } size: 995968 timestamp: 1602683788456 type: FILE visibility: PRIVATE
    commons-net_commons-net-3.1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/commons-net_commons-net-3.1.jar" } size: 273370 timestamp: 1602683787612 type: FILE visibility: PRIVATE
    org.slf4j_slf4j-log4j12-1.7.10.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.slf4j_slf4j-log4j12-1.7.10.jar" } size: 8866 timestamp: 1602683788991 type: FILE visibility: PRIVATE
    com.jamesmurty.utils_java-xmlbuilder-0.4.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.jamesmurty.utils_java-xmlbuilder-0.4.jar" } size: 18490 timestamp: 1602683788359 type: FILE visibility: PRIVATE
    __spark_libs__ -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/__spark_libs__300261197764166949.zip" } size: 231412540 timestamp: 1602683786626 type: ARCHIVE visibility: PRIVATE
    com.jcraft_jsch-0.1.42.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.jcraft_jsch-0.1.42.jar" } size: 185746 timestamp: 1602683788026 type: FILE visibility: PRIVATE
    io.netty_netty-3.6.2.Final.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/io.netty_netty-3.6.2.Final.jar" } size: 1199572 timestamp: 1602683789011 type: FILE visibility: PRIVATE
    hive-site.xml -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/hive-site.xml" } size: 2168 timestamp: 1602683789139 type: FILE visibility: PRIVATE
    com.fasterxml.jackson.core_jackson-databind-2.2.3.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.fasterxml.jackson.core_jackson-databind-2.2.3.jar" } size: 865838 timestamp: 1602683787359 type: FILE visibility: PRIVATE
    com.sun.jersey_jersey-json-1.9.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/com.sun.jersey_jersey-json-1.9.jar" } size: 147952 timestamp: 1602683787740 type: FILE visibility: PRIVATE
    asm_asm-3.2.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/asm_asm-3.2.jar" } size: 43398 timestamp: 1602683788301 type: FILE visibility: PRIVATE
    org.mortbay.jetty_jetty-util-6.1.26.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.mortbay.jetty_jetty-util-6.1.26.jar" } size: 177131 timestamp: 1602683787697 type: FILE visibility: PRIVATE
    org.apache.httpcomponents_httpcore-4.2.5.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.httpcomponents_httpcore-4.2.5.jar" } size: 227708 timestamp: 1602683788340 type: FILE visibility: PRIVATE
    jline_jline-0.9.94.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/jline_jline-0.9.94.jar" } size: 87325 timestamp: 1602683789047 type: FILE visibility: PRIVATE
    org.apache.curator_curator-client-2.7.1.jar -> resource { scheme: "hdfs" host: "x.x.x.x" port: 8020 file: "/user/hadoop/.sparkStaging/application_1602165382040_0066/org.apache.curator_curator-client-2.7.1.jar" } size: 69500 timestamp: 1602683788048 type: FILE visibility: PRIVATE

===============================================================================
20/10/14 13:56:36 INFO Utils: Using initial executors = 50, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
20/10/14 13:56:36 INFO YarnAllocator: Will request 50 executor container(s), each with 4 core(s) and 22528 MB memory (including 3557 MB of overhead)
20/10/14 13:56:36 INFO YarnAllocator: Submitted 50 unlocalized container requests.
20/10/14 13:56:36 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
20/10/14 13:56:36 INFO AMRMClientImpl: Received new token for : x.x.x.x:8041
20/10/14 13:56:36 INFO YarnAllocator: Launching container container_1602165382040_0066_01_000002 on host x.x.x.x for executor with ID 1
20/10/14 13:56:36 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
20/10/14 13:56:36 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
20/10/14 13:56:36 INFO AMRMClientImpl: Received new token for : x.x.x.x:8041
20/10/14 13:56:36 INFO YarnAllocator: Launching container container_1602165382040_0066_01_000004 on host x.x.x.x for executor with ID 2
20/10/14 13:56:36 INFO YarnAllocator: Launching container container_1602165382040_0066_01_000005 on host x.x.x.x for executor with ID 3
20/10/14 13:56:36 INFO YarnAllocator: Received 2 containers from YARN, launching executors on 2 of them.
20/10/14 13:56:36 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
20/10/14 13:56:36 INFO ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
20/10/14 13:56:40 INFO YarnAllocator: Driver requested a total number of 1 executor(s).
20/10/14 13:56:40 INFO YarnAllocator: Canceling requests for 47 executor container(s) to have a new desired total 1 executors.
20/10/14 13:56:46 INFO YarnAllocator: Driver requested a total number of 0 executor(s).
20/10/14 13:56:54 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. x.x.x.x:37307
20/10/14 13:56:54 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. x.x.x.x:37307
20/10/14 13:56:54 INFO ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0
20/10/14 13:56:54 INFO ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED
20/10/14 13:56:54 INFO AMRMClientImpl: Waiting for application to be successfully unregistered.
20/10/14 13:56:54 INFO ApplicationMaster: Deleting staging directory hdfs://x.x.x.x:8020/user/hadoop/.sparkStaging/application_1602165382040_0066
20/10/14 13:56:54 INFO ShutdownHookManager: Shutdown hook called
End of LogType:stderr

LogType:stdout
Log Upload Time:Wed Oct 14 13:56:55 +0000 2020
LogLength:0
Log Contents:
End of LogType:stdout