Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[HUDI-2747]Support set --sparkMaster for MDT cli #4964

Merged
merged 1 commit into from Mar 7, 2022
Merged

[HUDI-2747]Support set --sparkMaster for MDT cli #4964

merged 1 commit into from Mar 7, 2022

Conversation

zhangyue19921010
Copy link
Contributor

https://issues.apache.org/jira/browse/HUDI-2747

What is the purpose of the pull request

Fix hudi cli metadata commands. Currently when running hudi cli metadata commands locally, it complains about spark master, e.g., yarn master not found. The commands need to take in spark related configs so that they can run locally and in cluster.

Brief change log

Let's take metadata list-partitions as an example,
Before this patch

67411 [Spring Shell] INFO  org.apache.spark.SparkContext  - Running Spark version 2.4.4
67442 [Spring Shell] INFO  org.apache.spark.SparkContext  - Submitted application: HoodieCLI
67515 [Spring Shell] INFO  org.apache.spark.SecurityManager  - Changing view acls to: yuezhang
67515 [Spring Shell] INFO  org.apache.spark.SecurityManager  - Changing modify acls to: yuezhang
67516 [Spring Shell] INFO  org.apache.spark.SecurityManager  - Changing view acls groups to: 
67516 [Spring Shell] INFO  org.apache.spark.SecurityManager  - Changing modify acls groups to: 
67517 [Spring Shell] INFO  org.apache.spark.SecurityManager  - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yuezhang); groups with view permissions: Set(); users  with modify permissions: Set(yuezhang); groups with modify permissions: Set()
67571 [Spring Shell] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.output.compression.codec is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.codec
67571 [Spring Shell] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.output.compress is deprecated. Instead, use mapreduce.output.fileoutputformat.compress
67571 [Spring Shell] INFO  org.apache.hadoop.conf.Configuration.deprecation  - mapred.output.compression.type is deprecated. Instead, use mapreduce.output.fileoutputformat.compress.type
67823 [Spring Shell] INFO  org.apache.spark.util.Utils  - Successfully started service 'sparkDriver' on port 56009.
67851 [Spring Shell] INFO  org.apache.spark.SparkEnv  - Registering MapOutputTracker
67877 [Spring Shell] INFO  org.apache.spark.SparkEnv  - Registering BlockManagerMaster
67881 [Spring Shell] INFO  org.apache.spark.storage.BlockManagerMasterEndpoint  - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
67881 [Spring Shell] INFO  org.apache.spark.storage.BlockManagerMasterEndpoint  - BlockManagerMasterEndpoint up
67896 [Spring Shell] INFO  org.apache.spark.storage.DiskBlockManager  - Created local directory at /private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/blockmgr-956db4c2-91a4-4287-913c-69feff340328
67917 [Spring Shell] INFO  org.apache.spark.storage.memory.MemoryStore  - MemoryStore started with capacity 2004.6 MB
67970 [Spring Shell] INFO  org.apache.spark.SparkEnv  - Registering OutputCommitCoordinator
68082 [Spring Shell] INFO  org.spark_project.jetty.util.log  - Logging initialized @68799ms
68158 [Spring Shell] INFO  org.spark_project.jetty.server.Server  - jetty-9.3.z-SNAPSHOT, build timestamp: 2019-02-16T00:53:49+08:00, git hash: eb70b240169fcf1abbd86af36482d1c49826fa0b
68175 [Spring Shell] INFO  org.spark_project.jetty.server.Server  - Started @68893ms
68200 [Spring Shell] INFO  org.spark_project.jetty.server.AbstractConnector  - Started ServerConnector@6ce10da9{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
68200 [Spring Shell] INFO  org.apache.spark.util.Utils  - Successfully started service 'SparkUI' on port 4040.
68226 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@701a506a{/jobs,null,AVAILABLE,@Spark}
68226 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@43834940{/jobs/json,null,AVAILABLE,@Spark}
68227 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@7bab57ed{/jobs/job,null,AVAILABLE,@Spark}
68227 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@1b91c8aa{/jobs/job/json,null,AVAILABLE,@Spark}
68228 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@42814bd1{/stages,null,AVAILABLE,@Spark}
68228 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@63347853{/stages/json,null,AVAILABLE,@Spark}
68228 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@58b14e8c{/stages/stage,null,AVAILABLE,@Spark}
68229 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@45d10a0b{/stages/stage/json,null,AVAILABLE,@Spark}
68230 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@1746e913{/stages/pool,null,AVAILABLE,@Spark}
68230 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@5ab68f0a{/stages/pool/json,null,AVAILABLE,@Spark}
68230 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@678c3f74{/storage,null,AVAILABLE,@Spark}
68231 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@2ac0e0dc{/storage/json,null,AVAILABLE,@Spark}
68231 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@aee8ae6{/storage/rdd,null,AVAILABLE,@Spark}
68231 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@644e8a6e{/storage/rdd/json,null,AVAILABLE,@Spark}
68232 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@623f7e0c{/environment,null,AVAILABLE,@Spark}
68232 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@4f54a497{/environment/json,null,AVAILABLE,@Spark}
68232 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@3113ae21{/executors,null,AVAILABLE,@Spark}
68233 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@219e5f97{/executors/json,null,AVAILABLE,@Spark}
68233 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@38488ab6{/executors/threadDump,null,AVAILABLE,@Spark}
68233 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@52d2e134{/executors/threadDump/json,null,AVAILABLE,@Spark}
68243 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@7e9c3fe9{/static,null,AVAILABLE,@Spark}
68244 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@5d2fa317{/,null,AVAILABLE,@Spark}
68245 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@1c5344fd{/api,null,AVAILABLE,@Spark}
68246 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@447ff00f{/jobs/job/kill,null,AVAILABLE,@Spark}
68247 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@55a01d8e{/stages/stage/kill,null,AVAILABLE,@Spark}
68250 [Spring Shell] INFO  org.apache.spark.ui.SparkUI  - Bound SparkUI to 0.0.0.0, and started at http://172.24.10.26:4040
68283 [Spring Shell] ERROR org.apache.spark.SparkContext  - Error initializing SparkContext.
org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
	at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117)
	at org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:367)
	at org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:201)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
	at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
	at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
	at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
	at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
	at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
	at java.lang.Thread.run(Thread.java:748)
68302 [Spring Shell] INFO  org.spark_project.jetty.server.AbstractConnector  - Stopped Spark@6ce10da9{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
68304 [Spring Shell] INFO  org.apache.spark.ui.SparkUI  - Stopped Spark web UI at http://172.24.10.26:4040
68318 [dispatcher-event-loop-8] INFO  org.apache.spark.MapOutputTrackerMasterEndpoint  - MapOutputTrackerMasterEndpoint stopped!
68331 [Spring Shell] INFO  org.apache.spark.storage.memory.MemoryStore  - MemoryStore cleared
68332 [Spring Shell] INFO  org.apache.spark.storage.BlockManager  - BlockManager stopped
68346 [Spring Shell] INFO  org.apache.spark.storage.BlockManagerMaster  - BlockManagerMaster stopped
68347 [Spring Shell] WARN  org.apache.spark.metrics.MetricsSystem  - Stopping a MetricsSystem that is not running
68355 [dispatcher-event-loop-1] INFO  org.apache.spark.scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint  - OutputCommitCoordinator stopped!
68364 [Spring Shell] INFO  org.apache.spark.SparkContext  - Successfully stopped SparkContext
68364 [Spring Shell] ERROR org.springframework.shell.core.SimpleExecutionStrategy  - Command failed java.lang.reflect.UndeclaredThrowableException
68364 [Spring Shell] WARN  org.springframework.shell.core.JLineShellComponent.exceptions  - 
java.lang.reflect.UndeclaredThrowableException
	at org.springframework.util.ReflectionUtils.rethrowRuntimeException(ReflectionUtils.java:315)
	at org.springframework.util.ReflectionUtils.handleInvocationTargetException(ReflectionUtils.java:295)
	at org.springframework.util.ReflectionUtils.handleReflectionException(ReflectionUtils.java:279)
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:219)
	at org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
	at org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
	at org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
	at org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
	at org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.spark.SparkException: Could not parse Master URL: 'yarn'
	at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2784)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:493)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117)
	at org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:367)
	at org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:201)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)

After this patch, we can use metadata list-partitions --sparkMaster local[*]

97028 [Spring Shell] WARN  org.apache.spark.SparkContext  - Another SparkContext is being constructed (or threw an exception in its constructor).  This may indicate an error, since only one SparkContext may be running in this JVM (see SPARK-2243). The other SparkContext was created at:
org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
org.apache.hudi.cli.utils.SparkUtil.initJavaSparkConf(SparkUtil.java:117)
org.apache.hudi.cli.commands.MetadataCommand.initJavaSparkContext(MetadataCommand.java:367)
org.apache.hudi.cli.commands.MetadataCommand.listPartitions(MetadataCommand.java:201)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:216)
org.springframework.shell.core.SimpleExecutionStrategy.invoke(SimpleExecutionStrategy.java:68)
org.springframework.shell.core.SimpleExecutionStrategy.execute(SimpleExecutionStrategy.java:59)
org.springframework.shell.core.AbstractShell.executeCommand(AbstractShell.java:134)
org.springframework.shell.core.JLineShell.promptLoop(JLineShell.java:533)
org.springframework.shell.core.JLineShell.run(JLineShell.java:179)
java.lang.Thread.run(Thread.java:748)
97028 [Spring Shell] INFO  org.apache.spark.SparkContext  - Running Spark version 2.4.4
97029 [Spring Shell] INFO  org.apache.spark.SparkContext  - Submitted application: HoodieCLI
97030 [Spring Shell] INFO  org.apache.spark.SecurityManager  - Changing view acls to: yuezhang
97030 [Spring Shell] INFO  org.apache.spark.SecurityManager  - Changing modify acls to: yuezhang
97030 [Spring Shell] INFO  org.apache.spark.SecurityManager  - Changing view acls groups to: 
97030 [Spring Shell] INFO  org.apache.spark.SecurityManager  - Changing modify acls groups to: 
97030 [Spring Shell] INFO  org.apache.spark.SecurityManager  - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(yuezhang); groups with view permissions: Set(); users  with modify permissions: Set(yuezhang); groups with modify permissions: Set()
97062 [Spring Shell] INFO  org.apache.spark.util.Utils  - Successfully started service 'sparkDriver' on port 56031.
97064 [Spring Shell] INFO  org.apache.spark.SparkEnv  - Registering MapOutputTracker
97065 [Spring Shell] INFO  org.apache.spark.SparkEnv  - Registering BlockManagerMaster
97065 [Spring Shell] INFO  org.apache.spark.storage.BlockManagerMasterEndpoint  - Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
97065 [Spring Shell] INFO  org.apache.spark.storage.BlockManagerMasterEndpoint  - BlockManagerMasterEndpoint up
97066 [Spring Shell] INFO  org.apache.spark.storage.DiskBlockManager  - Created local directory at /private/var/folders/61/77xdhf3x0x9g3t_vdd1c9_nwr4wznp/T/blockmgr-3f3b2949-3a00-49d7-a600-f0f585e9e7fc
97066 [Spring Shell] INFO  org.apache.spark.storage.memory.MemoryStore  - MemoryStore started with capacity 2004.6 MB
97067 [Spring Shell] INFO  org.apache.spark.SparkEnv  - Registering OutputCommitCoordinator
97074 [Spring Shell] INFO  org.spark_project.jetty.server.Server  - jetty-9.3.z-SNAPSHOT, build timestamp: 2019-02-16T00:53:49+08:00, git hash: eb70b240169fcf1abbd86af36482d1c49826fa0b
97075 [Spring Shell] INFO  org.spark_project.jetty.server.Server  - Started @97793ms
97075 [Spring Shell] INFO  org.spark_project.jetty.server.AbstractConnector  - Started ServerConnector@53404266{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
97075 [Spring Shell] INFO  org.apache.spark.util.Utils  - Successfully started service 'SparkUI' on port 4040.
97076 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@25c98e66{/jobs,null,AVAILABLE,@Spark}
97076 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@58ad3d9b{/jobs/json,null,AVAILABLE,@Spark}
97076 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@1ba0d88{/jobs/job,null,AVAILABLE,@Spark}
97076 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@50383cab{/jobs/job/json,null,AVAILABLE,@Spark}
97077 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@396b3274{/stages,null,AVAILABLE,@Spark}
97077 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@81b8127{/stages/json,null,AVAILABLE,@Spark}
97077 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@599c0be1{/stages/stage,null,AVAILABLE,@Spark}
97077 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@48d0a91d{/stages/stage/json,null,AVAILABLE,@Spark}
97078 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@28c8cae1{/stages/pool,null,AVAILABLE,@Spark}
97078 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@7c8e7466{/stages/pool/json,null,AVAILABLE,@Spark}
97078 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@4574ab15{/storage,null,AVAILABLE,@Spark}
97078 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@8216ae7{/storage/json,null,AVAILABLE,@Spark}
97079 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@665036c0{/storage/rdd,null,AVAILABLE,@Spark}
97079 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@69f24328{/storage/rdd/json,null,AVAILABLE,@Spark}
97079 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@9be6100{/environment,null,AVAILABLE,@Spark}
97079 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@acb4482{/environment/json,null,AVAILABLE,@Spark}
97080 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@6d4a952d{/executors,null,AVAILABLE,@Spark}
97080 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@165a70af{/executors/json,null,AVAILABLE,@Spark}
97080 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@6c16b505{/executors/threadDump,null,AVAILABLE,@Spark}
97080 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@3ca90351{/executors/threadDump/json,null,AVAILABLE,@Spark}
97081 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@5731a14{/static,null,AVAILABLE,@Spark}
97081 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@40e9cd1d{/,null,AVAILABLE,@Spark}
97081 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@198ee698{/api,null,AVAILABLE,@Spark}
97082 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@7077c974{/jobs/job/kill,null,AVAILABLE,@Spark}
97082 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@7ef86342{/stages/stage/kill,null,AVAILABLE,@Spark}
97082 [Spring Shell] INFO  org.apache.spark.ui.SparkUI  - Bound SparkUI to 0.0.0.0, and started at http://172.24.10.26:4040
97179 [Spring Shell] INFO  org.apache.spark.executor.Executor  - Starting executor ID driver on host localhost
97254 [Spring Shell] INFO  org.apache.spark.util.Utils  - Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 56032.
97254 [Spring Shell] INFO  org.apache.spark.network.netty.NettyBlockTransferService  - Server created on 172.24.10.26:56032
97256 [Spring Shell] INFO  org.apache.spark.storage.BlockManager  - Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
97284 [Spring Shell] INFO  org.apache.spark.storage.BlockManagerMaster  - Registering BlockManager BlockManagerId(driver, 172.24.10.26, 56032, None)
97285 [dispatcher-event-loop-10] INFO  org.apache.spark.storage.BlockManagerMasterEndpoint  - Registering block manager 172.24.10.26:56032 with 2004.6 MB RAM, BlockManagerId(driver, 172.24.10.26, 56032, None)
97288 [Spring Shell] INFO  org.apache.spark.storage.BlockManagerMaster  - Registered BlockManager BlockManagerId(driver, 172.24.10.26, 56032, None)
97289 [Spring Shell] INFO  org.apache.spark.storage.BlockManager  - Initialized BlockManager: BlockManagerId(driver, 172.24.10.26, 56032, None)
97416 [Spring Shell] INFO  org.spark_project.jetty.server.handler.ContextHandler  - Started o.s.j.s.ServletContextHandler@71d6123f{/metrics/json,null,AVAILABLE,@Spark}
97536 [Spring Shell] INFO  org.apache.hudi.common.table.HoodieTableMetaClient  - Loading HoodieTableMetaClient from /Users/yuezhang/tmp/hudiAfTable/forecast_agg
97537 [Spring Shell] INFO  org.apache.hudi.common.table.HoodieTableConfig  - Loading table properties from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/hoodie.properties
97538 [Spring Shell] INFO  org.apache.hudi.common.table.HoodieTableMetaClient  - Finished Loading Table of type COPY_ON_WRITE(version=1, baseFileFormat=PARQUET) from /Users/yuezhang/tmp/hudiAfTable/forecast_agg
97538 [Spring Shell] INFO  org.apache.hudi.common.table.HoodieTableMetaClient  - Loading HoodieTableMetaClient from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata
97539 [Spring Shell] INFO  org.apache.hudi.common.table.HoodieTableConfig  - Loading table properties from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/.hoodie/hoodie.properties
97540 [Spring Shell] INFO  org.apache.hudi.common.table.HoodieTableMetaClient  - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata
97543 [Spring Shell] INFO  org.apache.hudi.metadata.HoodieTableMetadataUtil  - Loading latest merged file slices for metadata table partition files
97566 [Spring Shell] INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline  - Loaded instants upto : Option{val=[20220214171053821__deltacommit__COMPLETED]}
97572 [Spring Shell] INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView  - Took 3 ms to read  0 instants, 0 replaced file groups
97587 [Spring Shell] INFO  org.apache.hudi.common.util.ClusteringUtils  - Found 0 files in pending clustering operations
97588 [Spring Shell] INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView  - Building file system view for partition (files)
97600 [Spring Shell] INFO  org.apache.hudi.common.table.view.AbstractTableFileSystemView  - addFilesToView: NumFiles=12, NumFileGroups=1, FileGroupsCreationTime=8, StoreTimeTaken=1
97611 [Spring Shell] INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline  - Loaded instants upto : Option{val=[20220214171053821__commit__COMPLETED]}
97743 [Spring Shell] INFO  org.apache.hudi.common.table.HoodieTableMetaClient  - Loading HoodieTableMetaClient from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata
97744 [Spring Shell] INFO  org.apache.hudi.common.table.HoodieTableConfig  - Loading table properties from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/.hoodie/hoodie.properties
97745 [Spring Shell] INFO  org.apache.hudi.common.table.HoodieTableMetaClient  - Finished Loading Table of type MERGE_ON_READ(version=1, baseFileFormat=HFILE) from /Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata
97760 [Spring Shell] INFO  org.apache.hudi.common.table.timeline.HoodieActiveTimeline  - Loaded instants upto : Option{val=[20220214171053821__deltacommit__COMPLETED]}
97768 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-0-0', fileLen=-1}
97774 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Reading a delete block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-0-0
97778 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieLogFormatReader  - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-14-29', fileLen=-1}
97778 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-14-29', fileLen=-1}
97783 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.1_0-14-29 at instant 00000000000000
97784 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieLogFormatReader  - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.2_0-27-40', fileLen=-1}
97785 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.2_0-27-40', fileLen=-1}
97785 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.2_0-27-40 at instant 20220214165727173
97785 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Number of remaining logblocks to merge 2
97842 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Number of remaining logblocks to merge 1
97861 [Spring Shell] INFO  org.apache.hadoop.hbase.io.hfile.CacheConfig  - Allocating LruBlockCache size=1.42 GB, blockSize=64 KB
97875 [Spring Shell] INFO  org.apache.hadoop.hbase.io.hfile.CacheConfig  - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
98021 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98029 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98030 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98096 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98123 [Spring Shell] INFO  org.apache.hudi.common.util.collection.ExternalSpillableMap  - Estimated Payload size => 328
98143 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieLogFormatReader  - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.3_0-11-20', fileLen=-1}
98143 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.3_0-11-20', fileLen=-1}
98144 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.3_0-11-20 at instant 20220214170139680
98144 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Number of remaining logblocks to merge 1
98155 [Spring Shell] INFO  org.apache.hadoop.hbase.io.hfile.CacheConfig  - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
98155 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98155 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98156 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98156 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98159 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieLogFormatReader  - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.4_0-11-20', fileLen=-1}
98159 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.4_0-11-20', fileLen=-1}
98160 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.4_0-11-20 at instant 20220214170233074
98160 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Number of remaining logblocks to merge 1
98170 [Spring Shell] INFO  org.apache.hadoop.hbase.io.hfile.CacheConfig  - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
98171 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98171 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98171 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98171 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98173 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieLogFormatReader  - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.5_0-11-20', fileLen=-1}
98174 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.5_0-11-20', fileLen=-1}
98174 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.5_0-11-20 at instant 20220214170320009
98174 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Number of remaining logblocks to merge 1
98185 [Spring Shell] INFO  org.apache.hadoop.hbase.io.hfile.CacheConfig  - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
98185 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98185 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98185 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98186 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98188 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieLogFormatReader  - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.6_0-11-20', fileLen=-1}
98189 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.6_0-11-20', fileLen=-1}
98189 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.6_0-11-20 at instant 20220214170407096
98189 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Number of remaining logblocks to merge 1
98200 [Spring Shell] INFO  org.apache.hadoop.hbase.io.hfile.CacheConfig  - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
98200 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98200 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98200 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98200 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98203 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieLogFormatReader  - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.7_0-11-20', fileLen=-1}
98204 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.7_0-11-20', fileLen=-1}
98204 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.7_0-11-20 at instant 20220214170451797
98204 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Number of remaining logblocks to merge 1
98218 [Spring Shell] INFO  org.apache.hadoop.hbase.io.hfile.CacheConfig  - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
98218 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98218 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98218 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98219 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98224 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieLogFormatReader  - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.8_0-11-20', fileLen=-1}
98225 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.8_0-11-20', fileLen=-1}
98225 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.8_0-11-20 at instant 20220214170547258
98225 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Number of remaining logblocks to merge 1
98238 [Spring Shell] INFO  org.apache.hadoop.hbase.io.hfile.CacheConfig  - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
98238 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98238 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98238 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98238 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98241 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieLogFormatReader  - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.9_0-11-20', fileLen=-1}
98241 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.9_0-11-20', fileLen=-1}
98241 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.9_0-11-20 at instant 20220214170815557
98241 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Number of remaining logblocks to merge 1
98252 [Spring Shell] INFO  org.apache.hadoop.hbase.io.hfile.CacheConfig  - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
98252 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98253 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98253 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98253 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98255 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieLogFormatReader  - Moving to the next reader for logfile HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.10_0-11-20', fileLen=-1}
98256 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Scanning log file HoodieLogFile{pathStr='file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.10_0-11-20', fileLen=-1}
98256 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Reading a data block from file file:/Users/yuezhang/tmp/hudiAfTable/forecast_agg/.hoodie/metadata/files/.files-0000_00000000000000.log.10_0-11-20 at instant 20220214171053821
98256 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Number of remaining logblocks to merge 1
98267 [Spring Shell] INFO  org.apache.hadoop.hbase.io.hfile.CacheConfig  - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
98267 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98267 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98267 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98267 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98268 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Merging the final data blocks
98268 [Spring Shell] INFO  org.apache.hudi.common.table.log.AbstractHoodieLogRecordReader  - Number of remaining logblocks to merge 1
98279 [Spring Shell] INFO  org.apache.hadoop.hbase.io.hfile.CacheConfig  - Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1567280, freeSize=1525578832, maxSize=1527146112, heapSize=1567280, minSize=1450788736, minFactor=0.95, multiSize=725394368, multiFactor=0.5, singleSize=362697184, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
98279 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98279 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98279 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98280 [Spring Shell] INFO  org.apache.hadoop.io.compress.CodecPool  - Got brand-new decompressor [.gz]
98281 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner  - Number of log files scanned => 11
98281 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner  - MaxMemoryInBytes allowed for compaction => 1073741824
98281 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner  - Number of entries in MemoryBasedMap in ExternalSpillableMap => 2
98281 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner  - Total size in bytes of MemoryBasedMap in ExternalSpillableMap => 656
98281 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner  - Number of entries in BitCaskDiskMap in ExternalSpillableMap => 0
98281 [Spring Shell] INFO  org.apache.hudi.common.table.log.HoodieMergedLogRecordScanner  - Size of file spilled to disk => 0
98282 [Spring Shell] INFO  org.apache.hudi.metadata.HoodieBackedTableMetadata  - Opened 11 metadata log files (dataset instant=20220214171053821, metadata instant=20220214171053821) in 677 ms
98294 [Spring Shell] INFO  org.apache.hudi.metadata.BaseTableMetadata  - Listed partitions from metadata: #partitions=1
98302 [Spring Shell] INFO  org.springframework.shell.core.JLineShellComponent  - ╔═══════════╗
║ partition ║
╠═══════════╣
║ 20210623  ║
╚═══════════╝

Verify this pull request

(Please pick either of the following options)

This pull request is a trivial rework / code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end.
  • Added HoodieClientWriteTest to verify the change.
  • Manually verified the change by running a job locally.

Committer checklist

  • Has a corresponding JIRA in PR title & commit

  • Commit message is descriptive of the change

  • CI is green

  • Necessary doc changes done or have another open PR

  • For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.

@zhangyue19921010
Copy link
Contributor Author

@yihua Hi ethan would you mind to take a look? Thanks a lot!

@hudi-bot
Copy link

hudi-bot commented Mar 7, 2022

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@nsivabalan nsivabalan self-assigned this Mar 7, 2022
@nsivabalan nsivabalan added this to Under Discussion PRs in PR Tracker Board via automation Mar 7, 2022
PR Tracker Board automation moved this from Under Discussion PRs to Nearing Landing Mar 7, 2022
@nsivabalan nsivabalan merged commit 53826d6 into apache:master Mar 7, 2022
PR Tracker Board automation moved this from Nearing Landing to Done Mar 7, 2022
yihua
yihua reviewed Mar 8, 2022
View reviewed changes
Copy link
Contributor

@yihua yihua left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

vingov pushed a commit to vingov/hudi that referenced this pull request Apr 3, 2022
@zhangyue19921010
[HUDI-2747] support set --sparkMaster for MDT cli (apache#4964)
d278e36 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
stayrascal pushed a commit to stayrascal/hudi that referenced this pull request Apr 12, 2022
@zhangyue19921010 @stayrascal
[HUDI-2747] support set --sparkMaster for MDT cli (apache#4964)
6fa32a0 
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yihua yihua yihua left review comments

@nsivabalan nsivabalan nsivabalan approved these changes

Assignees

@nsivabalan nsivabalan

Labels
None yet
Projects
PR Tracker Board
Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@zhangyue19921010 @hudi-bot @nsivabalan @yihua