Skip to content

Comments

[HUDI-5893] Mark additional advanced configs#8329

Merged
yihua merged 1 commit intoapache:masterfrom
yihua:HUDI-5893-mark-advanced-config
Apr 14, 2023
Merged

[HUDI-5893] Mark additional advanced configs#8329
yihua merged 1 commit intoapache:masterfrom
yihua:HUDI-5893-mark-advanced-config

Conversation

@yihua
Copy link
Contributor

@yihua yihua commented Mar 30, 2023

Change Logs

This PR marks 582 additional advanced configs. After this PR, we have 52 basic configs (8%) and 586 advanced configs (excluding Flink specific options).

Here are the configs that are going to show up in the "Basic Configuration" page (other configs are marked advanced by this PR):

Spark configs (in total, 6)

  • Spark Read Configs (3)
hoodie.datasource.read.begin.instanttime
hoodie.datasource.read.end.instanttime
hoodie.datasource.query.type
  • Spark Write Configs (3)
hoodie.datasource.write.operation
hoodie.datasource.write.table.type
hoodie.sql.insert.mode

General write and table service configs (in total, 32)

  • Clean Configs (2)
hoodie.clean.async
hoodie.cleaner.commits.retained
  • Archival Configs (2)
hoodie.keep.max.commits
hoodie.keep.min.commits
  • Key Generator Options (3)
hoodie.datasource.write.partitionpath.field
hoodie.datasource.write.recordkey.field
hoodie.datasource.write.hive_style_partitioning
  • Storage Configs (2)
hoodie.parquet.compression.codec
hoodie.parquet.max.file.size
  • Metadata Configs (3)
hoodie.metadata.enable
hoodie.metadata.index.bloom.filter.enable
hoodie.metadata.index.column.stats.enable
  • Write Configurations (4)
hoodie.base.path 
hoodie.table.name
hoodie.datasource.write.precombine.field
hoodie.write.concurrency.mode
  • Compaction Configs (2)
hoodie.compact.inline 
hoodie.compact.inline.max.delta.commits
  • Clustering Configs (4)
hoodie.clustering.async.enabled
hoodie.clustering.inline
hoodie.clustering.plan.strategy.small.file.limit
hoodie.clustering.plan.strategy.target.file.max.bytes
  • Bootstrap Configs (1)
hoodie.bootstrap.base.path
  • Index Configs (1)
hoodie.index.type
  • Common Metadata Sync Configs (1)
hoodie.datasource.meta.sync.enable
  • Hive Sync Configs (4)
hoodie.datasource.hive_sync.mode
hoodie.datasource.hive_sync.enable
hoodie.datasource.hive_sync.jdbcurl
hoodie.datasource.hive_sync.metastore.uris
  • Metrics Configurations (2)
hoodie.metrics.on
hoodie.metrics.reporter.type
  • Kafka Sink Connect Configurations (1)
bootstrap.servers

Deltastreamer configs (in total, 14)

  • DeltaStreamer SQL Transformer Configs (2)
hoodie.deltastreamer.transformer.sql
hoodie.deltastreamer.transformer.sql.file
  • SQL Source Configs (1)
hoodie.deltastreamer.source.sql.sql.query
  • Hudi Incremental Source Configs (1)
hoodie.deltastreamer.source.hoodieincr.path
  • Pulsar Source Configs (3)
hoodie.deltastreamer.source.pulsar.topic
hoodie.deltastreamer.source.pulsar.endpoint.admin.url
hoodie.deltastreamer.source.pulsar.endpoint.service.url
  • Kafka Source Configs (1)
hoodie.deltastreamer.source.kafka.topic
  • S3 Source Configs (1)
hoodie.deltastreamer.s3.source.queue.url
  • DFS Path Selector Configs (1)
hoodie.deltastreamer.source.dfs.root
  • DeltaStreamer Schema Provider Configs (2)
hoodie.deltastreamer.schemaprovider.registry.targetUrl
hoodie.deltastreamer.schemaprovider.registry.url
  • File-based Schema Provider Configs (2)
hoodie.deltastreamer.schemaprovider.source.schema.file
hoodie.deltastreamer.schemaprovider.target.schema.file

Impact

Advanced configs are not shown in the "Basic Configuration" page in our docs for simplicity.

Risk level

none

Documentation Update

HUDI-5783 for docs update.

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@yihua yihua added the configs label Mar 30, 2023
@yihua yihua force-pushed the HUDI-5893-mark-advanced-config branch 3 times, most recently from a7b0e52 to 44487a1 Compare April 14, 2023 00:28
@yihua yihua requested a review from nsivabalan April 14, 2023 01:32
@yihua yihua force-pushed the HUDI-5893-mark-advanced-config branch from 44487a1 to 054fbfe Compare April 14, 2023 03:00
@apache apache deleted a comment from hudi-bot Apr 14, 2023
@yihua
Copy link
Contributor Author

yihua commented Apr 14, 2023

CI is green.
Screenshot 2023-04-14 at 11 27 45

@yihua yihua merged commit b937b08 into apache:master Apr 14, 2023
yihua added a commit that referenced this pull request Apr 18, 2023
This commit marks 85 advanced Flink configs. After this commit, we have 33 basic configs and 85 advanced configs for Hudi on Flink. The criteria of marking advanced configs for Hudi on Flink is the same as #8329.
stayrascal pushed a commit to stayrascal/hudi that referenced this pull request Apr 20, 2023
This PR marks 582 additional advanced configs. After this PR, we have 52 basic configs (8%) and 586 advanced configs (excluding Flink-specific options).
- Spark configs: in total, 6
- General write and table service configs: in total, 32
- Deltastreamer configs: in total, 14
stayrascal pushed a commit to stayrascal/hudi that referenced this pull request Apr 20, 2023
This commit marks 85 advanced Flink configs. After this commit, we have 33 basic configs and 85 advanced configs for Hudi on Flink. The criteria of marking advanced configs for Hudi on Flink is the same as apache#8329.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

2 participants