Skip to content

[spark] Extend get_cluster_configs procedure output with default value, is_default and description columns#3690

Open
XuQianJin-Stars wants to merge 1 commit into
apache:mainfrom
XuQianJin-Stars:feat/spark-get-cluster-configs-columns
Open

[spark] Extend get_cluster_configs procedure output with default value, is_default and description columns#3690
XuQianJin-Stars wants to merge 1 commit into
apache:mainfrom
XuQianJin-Stars:feat/spark-get-cluster-configs-columns

Conversation

@XuQianJin-Stars

Copy link
Copy Markdown
Contributor

Purpose

Linked issue: close #xxx

Extend the sys.get_cluster_configs Spark procedure output so users can inspect not only the current value and source of each cluster config, but also its declared default value, whether the current value equals the default, and a human-readable description. This makes the procedure far more useful for diagnostics and onboarding.

Brief change log

  • Extended GetClusterConfigsProcedure output schema from 3 columns to 6 columns:
    • config_key (String)
    • config_value (String)
    • config_source (String: DYNAMIC / STATIC / DEFAULT)
    • default_value (String, nullable)
    • is_default (Boolean)
    • description (String, nullable)
  • Built a config_key -> ConfigOption lookup by reflecting over the static ConfigOptions fields, so known keys are enriched with their declared default value, default flag, and description.
  • For unknown keys (not declared in ConfigOptions): default_value and description are NULL, and is_default is reported as true.
  • Updated Spark engine documentation (procedures.md) with the new columns and usage examples.
  • Updated GetClusterConfigsProcedureTest with assertions covering the new columns.

Tests

  • GetClusterConfigsProcedureTest:
    • Verified the result schema has 6 columns with the expected field names.
    • Verified default_value, is_default, and description for an overridden key (e.g. kv.snapshot.interval).
    • Verified a key with no declared default reports NULL default and is_default = true.

API and Format

  • Output schema (column set and order) of the sys.get_cluster_configs procedure is extended. This is an additive change; existing columns keep their positions. Downstream consumers reading by column index should be aware of the new trailing columns.

Documentation

  • Updated website/docs/engine-spark/procedures.md to document the three new output columns and provide example output.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant