Skip to content

HIVE-29032: Enhance qt:database option to expose connection properties in qfiles #5919

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

zabetak
Copy link
Member

@zabetak zabetak commented Jul 2, 2025

What changes were proposed in this pull request?

  1. Allow to define name/id for each database used via the qt:database directive
  2. Expose database connection properties via system properties and manage them during the life-cycle of the database

Why are the changes needed?

  1. Facilitate writing of qtests for the JDBC storage handler
  2. Avoid harcoding connection properties in qfiles
  3. Opens up the stage for supporting multiple databases of the same kind in qtests

Does this PR introduce any user-facing change?

No, it only affects the test infrastructure.

How was this patch tested?

mvn test -Dtest=TestMiniLlapLocalCliDriver -Dqfile=cbo_jdbc_joincost.q,dataconnector_mysql.q,jdbc_filter_expand_row_operator.q,jdbc_partition_table_pruned_pcolumn.q,jdbc_project_pushdown.q,jdbc_table_dml_postgres.q,jdbc_table_limit_postgres.q,jdbc_table_with_schema_mariadb.q,jdbc_table_with_schema_mssql.q,jdbc_table_with_schema_oracle.q,jdbc_table_with_schema_postgres.q,qt_database_mssql.q,qt_database_mysql.q,qt_database_postgres.q -Dtest.output.overwrite

Copy link

sonarqubecloud bot commented Jul 7, 2025

* Syntax: qt:database:DatabaseType:path_to_init_script
*
* The database type ({@link DatabaseType}) is obligatory argument. The initialization script can be omitted.
* Syntax: qt:database:DatabaseType:databaseName:path_to_init_script
Copy link
Member

@deniskuzZ deniskuzZ Jul 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make it a bit more user-friendly?

qt:database:DatabaseType:schema:DatabaseName:path_to_init_script

--!qt:database:mysql:schema:qdb:q_test_author_book_tables.sql

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will make the option parsing trickier and maybe it's not worth it given that most devs will copy-paste from one file to another.

For the record, I also considered using commons-cli and Options to allow the following syntax:

qt:database:--type mysql --name qdb --script q_test_author_book_tables.sql
qt:database:-t mysql -n qdb -s q_test_author_book_tables.sql

But it felt a bit of an overkill especially since databaseType and databaseName are mandatory args.

If the rest of the changes in this PR look good how about merging this (since tests are green) and following up with another ticket if you feel that its worth it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, though I like this syntax more:

qt:database:--type mysql --name qdb --script q_test_author_book_tables.sql

Copy link
Member

@deniskuzZ deniskuzZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@@ -82,52 +80,48 @@ AbstractExternalDB create() {
abstract AbstractExternalDB create();
}

private final Map<DatabaseType, String> databaseToScript = new EnumMap<>(DatabaseType.class);
private final String scriptsDir;
private final List<AbstractExternalDB> databases = new ArrayList<>();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zabetak, should it be a Set? What if the used defines init scripts for multiple schemas?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants