Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-36526][SQL] DSV2 Index Support: Add supportsIndex interface #33754
[SPARK-36526][SQL] DSV2 Index Support: Add supportsIndex interface #33754
Changes from 3 commits
a84794c
b89b321
f9f4e37
d4c1931
14a819a
120b477
f6ca2e8
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
public final class
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shall we name it
TableIndex
to be more specific?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What
indexType
could be? Is it up to data source implementation?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's up to the data source implementation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we have example index type in mind?
String
seems vague to me as it can contain arbitrary stuff. Why don't we go with enum or class? Another perspective is why we cannot put the index type insideproperties
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The example index types are
BLOOM_FILTER_INDEX
,Z_ORDERING_INDEX
,BTREE_INDEX
. I actually started with enum and changed to String.It's more convenient to have index type outside of
properties
. in majority of the data sources, the create index syntax isCREATE [index_type] INDEX index_name ON [TABLE] table_name (column_name [ , . . . ])[OPTIONS indexPropertyList]
@cloud-fan Shall we use enum instead of String for index type? Even though the catalog implementation is responsible for recognizing and taking care of user-specified index type, we still need to add these specific index types in
SqlBase
? We probably want to use enum to let users know what index types are supported?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Index is for performance only and Spark doesn't need to define the semantic, I think String is more convenient. Data source can define whatever index type they like and ask end-users to use. Spark doesn't need to be in the middle.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should return
properties
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shall we also add
Map<String, String> columnProperties(col: FieldReference)
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we refine the classdoc?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed. Thanks
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shall we follow
SupportsPartitionManagement
and make it extendsTable
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nvm, index has a unique name. DROP INDEX does not need a table.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, not all databases make index name globally unique, see https://www.w3schools.com/sql/sql_ref_drop_index.asp
I think we can still make
SupportsIndex
extendsTable
, if the SQL syntax isDROP INDEX index_name ON table_name;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After thinking more about it, I think
DROP INDEX index_name ON [TABLE] table_name
is better, as it's more consistent with theCREATE INDEX
syntax.This is also more flexible: the index name only need to be unique within the table.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For partitioned table, do we plan to support index creation on table level (for all partitions), or on individual partition level?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is up to the data source implementation. I think it makes more sense at file level (each data file has an index file).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I prefer to support index creation on individual partition level.
For the existing data in the production environment, if only support index creation on table level, it is likely to be an impossible job for users.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry I think I didn't explain this clear: the index creation is actually done at the underlying data source. It's up to the data source's implementation on which level the index is created. For the implementation in file based data source, I believe the index is created at file level, not at table level or partition level.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for your explain
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
UnsupportedOperationException
is not a checked java exception, we don't need to put it in thethrows
clause.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what's "soft delete"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what's the corresponding SQL syntax for delete and drop index?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should follow
TableCatalog.alterTable
It's not very clear to me what can be changed in ALTER INDEX though.