Skip to content

Conversation

@alamb
Copy link
Contributor

@alamb alamb commented Apr 14, 2024

Which issue does this PR close?

N/A

Rationale for this change

When I want to see all the config options in datafusion CLI I use SHOW ALL.
However there are more than 40 rows (the default before truncation) and I almost
always forget to set --maxrows and thus can't see all of them:

(venv) andrewlamb@Andrews-MacBook-Pro:~/Software/arrow-datafusion2$ datafusion-cli
DataFusion CLI v37.0.0
> show all;
+-------------------------------------------------------------------------+---------------------------+
| name                                                                    | value                     |
+-------------------------------------------------------------------------+---------------------------+
| datafusion.catalog.create_default_catalog_and_schema                    | true                      |
| datafusion.catalog.default_catalog                                      | datafusion                |
| datafusion.catalog.default_schema                                       | public                    |
| datafusion.catalog.format                                               |                           |
| datafusion.catalog.has_header                                           | false                     |
| datafusion.catalog.information_schema                                   | true                      |
...
| datafusion.execution.parquet.reorder_filters                            | false                     |
| datafusion.execution.parquet.skip_metadata                              | true                      |
| datafusion.execution.parquet.statistics_enabled                         |                           |
| datafusion.execution.parquet.write_batch_size                           | 1024                      |
| .                                                                                                   |
| .                                                                                                   |
| .                                                                                                   |
+-------------------------------------------------------------------------+---------------------------+
73 row(s) fetched. (First 40 displayed. Use --maxrows to adjust)
Elapsed 0.007 seconds.

I would like to see them all

What changes are included in this PR?

  1. Refactor code for adjusting PrintOptions into its own structure
  2. Add case for SHOW VARIABLE

Are these changes tested?

I tested this manually

Are there any user-facing changes?

I can see all variables now with SHOW ALL:

Details

DataFusion CLI v37.0.0
> show all;
+-------------------------------------------------------------------------+---------------------------+
| name                                                                    | value                     |
+-------------------------------------------------------------------------+---------------------------+
| datafusion.catalog.create_default_catalog_and_schema                    | true                      |
| datafusion.catalog.default_catalog                                      | datafusion                |
| datafusion.catalog.default_schema                                       | public                    |
| datafusion.catalog.format                                               |                           |
| datafusion.catalog.has_header                                           | false                     |
| datafusion.catalog.information_schema                                   | true                      |
| datafusion.catalog.location                                             |                           |
| datafusion.execution.aggregate.scalar_update_factor                     | 10                        |
| datafusion.execution.batch_size                                         | 8192                      |
| datafusion.execution.coalesce_batches                                   | true                      |
| datafusion.execution.collect_statistics                                 | false                     |
| datafusion.execution.enable_recursive_ctes                              | true                      |
| datafusion.execution.listing_table_ignore_subdirectory                  | true                      |
| datafusion.execution.max_buffered_batches_per_output_file               | 2                         |
| datafusion.execution.meta_fetch_concurrency                             | 32                        |
| datafusion.execution.minimum_parallel_output_files                      | 4                         |
| datafusion.execution.parquet.allow_single_file_parallelism              | true                      |
| datafusion.execution.parquet.bloom_filter_enabled                       | false                     |
| datafusion.execution.parquet.bloom_filter_fpp                           |                           |
| datafusion.execution.parquet.bloom_filter_ndv                           |                           |
| datafusion.execution.parquet.column_index_truncate_length               |                           |
| datafusion.execution.parquet.compression                                | zstd(3)                   |
| datafusion.execution.parquet.created_by                                 | datafusion version 37.0.0 |
| datafusion.execution.parquet.data_page_row_count_limit                  | 18446744073709551615      |
| datafusion.execution.parquet.data_pagesize_limit                        | 1048576                   |
| datafusion.execution.parquet.dictionary_enabled                         |                           |
| datafusion.execution.parquet.dictionary_page_size_limit                 | 1048576                   |
| datafusion.execution.parquet.enable_page_index                          | true                      |
| datafusion.execution.parquet.encoding                                   |                           |
| datafusion.execution.parquet.max_row_group_size                         | 1048576                   |
| datafusion.execution.parquet.max_statistics_size                        |                           |
| datafusion.execution.parquet.maximum_buffered_record_batches_per_stream | 2                         |
| datafusion.execution.parquet.maximum_parallel_row_group_writers         | 1                         |
| datafusion.execution.parquet.metadata_size_hint                         |                           |
| datafusion.execution.parquet.pruning                                    | true                      |
| datafusion.execution.parquet.pushdown_filters                           | false                     |
| datafusion.execution.parquet.reorder_filters                            | false                     |
| datafusion.execution.parquet.skip_metadata                              | true                      |
| datafusion.execution.parquet.statistics_enabled                         |                           |
| datafusion.execution.parquet.write_batch_size                           | 1024                      |
| datafusion.execution.parquet.writer_version                             | 1.0                       |
| datafusion.execution.planning_concurrency                               | 16                        |
| datafusion.execution.soft_max_rows_per_output_file                      | 50000000                  |
| datafusion.execution.sort_in_place_threshold_bytes                      | 1048576                   |
| datafusion.execution.sort_spill_reservation_bytes                       | 10485760                  |
| datafusion.execution.target_partitions                                  | 16                        |
| datafusion.execution.time_zone                                          | +00:00                    |
| datafusion.explain.logical_plan_only                                    | false                     |
| datafusion.explain.physical_plan_only                                   | false                     |
| datafusion.explain.show_sizes                                           | true                      |
| datafusion.explain.show_statistics                                      | false                     |
| datafusion.optimizer.allow_symmetric_joins_without_pruning              | true                      |
| datafusion.optimizer.default_filter_selectivity                         | 20                        |
| datafusion.optimizer.enable_distinct_aggregation_soft_limit             | true                      |
| datafusion.optimizer.enable_round_robin_repartition                     | true                      |
| datafusion.optimizer.enable_topk_aggregation                            | true                      |
| datafusion.optimizer.filter_null_join_keys                              | false                     |
| datafusion.optimizer.hash_join_single_partition_threshold               | 1048576                   |
| datafusion.optimizer.hash_join_single_partition_threshold_rows          | 131072                    |
| datafusion.optimizer.max_passes                                         | 3                         |
| datafusion.optimizer.prefer_existing_sort                               | false                     |
| datafusion.optimizer.prefer_hash_join                                   | true                      |
| datafusion.optimizer.repartition_aggregations                           | true                      |
| datafusion.optimizer.repartition_file_min_size                          | 10485760                  |
| datafusion.optimizer.repartition_file_scans                             | true                      |
| datafusion.optimizer.repartition_joins                                  | true                      |
| datafusion.optimizer.repartition_sorts                                  | true                      |
| datafusion.optimizer.repartition_windows                                | true                      |
| datafusion.optimizer.skip_failed_rules                                  | false                     |
| datafusion.optimizer.top_down_join_key_reordering                       | true                      |
| datafusion.sql_parser.dialect                                           | generic                   |
| datafusion.sql_parser.enable_ident_normalization                        | true                      |
| datafusion.sql_parser.parse_float_as_decimal                            | false                     |
+-------------------------------------------------------------------------+---------------------------+
73 row(s) fetched.
Elapsed 0.026 seconds.

>


let statements = DFParser::parse_sql_with_dialect(&sql, dialect.as_ref())?;
for statement in statements {
let adjusted =
Copy link
Contributor Author

@alamb alamb Apr 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I extracted the existing code to update the print options based on statement into its own structure which I think makes it clearer what is going on here

@alamb
Copy link
Contributor Author

alamb commented Apr 15, 2024

Thanks for the review @Ted-Jiang 🙏

@alamb alamb merged commit 635faa6 into apache:main Apr 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants