Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature: add user-defined index access method origin #169

Closed
wants to merge 16 commits into from

Conversation

hw118118
Copy link
Contributor

@hw118118 hw118118 commented Aug 30, 2023

The purpose of the commit is in support of user-defined index access method. Currently, there are many restrictions for index access methods and new index access methods are incompatible with internal framework. At the same time, it's impossible for user to change default index access method. So I build some hooks that are for changing default behavior and being compatible for those strange restrictions.

For express the meaning of the modification, I show a example as below.

For example , I want to add 7 kinds of new index access method that based on diffrent storage engine (maybe unionstore storage) and they are correspond to internal index access methods as below:

  • usbtree ---btree
  • usgin --- gin
  • usgist --- gist
  • usspgist --- spgist
  • ushash --- hash
  • usbitmap --- bitmap

The main diffrence between them is index data can separate from other data and support Compute and Storage Separation.

Suppose we want to use these new functions, we can create a new extension and we can run sql (for example by create extension unionstore;) command and you can find new access methods :

regression=# select * from pg_am;
  oid   |    amname     |         amhandler         | amtype
--------+---------------+---------------------------+--------
      2 | heap          | heap_tableam_handler      | t
    403 | btree         | bthandler                 | i
    405 | hash          | hashhandler               | i
    783 | gist          | gisthandler               | i
   2742 | gin           | ginhandler                | i
   4000 | spgist        | spghandler                | i
   3580 | brin          | brinhandler               | i
   7024 | ao_row        | ao_row_tableam_handler    | t
   7166 | ao_column     | ao_column_tableam_handler | t
   7013 | bitmap        | bmhandler                 | i
  16394 | union_store   | heap_tableam_handler      | t
  16402 | ushash        | ushashhandler             | i
  16403 | usbtree       | usbthandler               | i
  16404 | usgist        | usgisthandler             | i
  16405 | usgin         | usginhandler              | i
  16406 | usspgist      | usspghandler              | i
  16407 | usbrin        | usbrinhandler             | i
  16408 | usbitmap      | usbmhandler               | i
  22820 | heap2         | heap_tableam_handler      | t
 111831 | ao_row_testam | ao_row_tableam_handler    | t
 111847 | ao_col_testam | ao_column_tableam_handler | t
 111850 | heap_testam   | heap_tableam_handler      | t

So you can use these new index access methods as internal index access method. At the same time, I add new guc variable and you can correct default index type by set default_index_type = usbtree or show relevant info by show default_index_type;

In conclusion, it's more flexible and it decoupe from other part.

Copy link
Collaborator

@avamingli avamingli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shall we add cases to test new hooks?

src/include/utils/sync_guc_name.h Outdated Show resolved Hide resolved
src/backend/utils/misc/guc.c Outdated Show resolved Hide resolved
src/backend/utils/cache/lsyscache.c Outdated Show resolved Hide resolved
src/backend/mock.mk Show resolved Hide resolved
@hw118118 hw118118 force-pushed the index_am branch 2 times, most recently from 7ba80b6 to f77dd80 Compare October 30, 2023 06:37
@avamingli
Copy link
Collaborator

Hi, I test the branch with the added GUC, there are sth we need to fix.

  1. set default_index_type, and create a new index, the GUC doesn't take effect.
    We set it to brin, but new created index is also btree.
gpadmin=# set default_index_type = brin;
SET
gpadmin=# show default_index_type;
 default_index_type
--------------------
 brin
(1 row)

gpadmin=# create index on t2(c1);
CREATE INDEX
gpadmin=# \d+ t2;
                                           Table "public.t2"
 Column |  Type   | Collation | Nullable | Default | Storage | Compression | Stats target | Description
--------+---------+-----------+----------+---------+---------+-------------+--------------+-------------
 c1     | integer |           |          |         | plain   |             |              |
 c2     | integer |           |          |         | plain   |             |              |
Indexes:
    "t2_c1_idx" btree (c1)
Distributed by: (c1)
Access method: heap
  1. We lack of test when setting this GUC to check the invalidation.
    User could set a not-exists value, it's risky.
gpadmin=# select * from pg_am;
 oid  |  amname   |         amhandler         | amtype
------+-----------+---------------------------+--------
    2 | heap      | heap_tableam_handler      | t
  403 | btree     | bthandler                 | i
  405 | hash      | hashhandler               | i
  783 | gist      | gisthandler               | i
 2742 | gin       | ginhandler                | i
 4000 | spgist    | spghandler                | i
 3580 | brin      | brinhandler               | i
 7024 | ao_row    | ao_row_tableam_handler    | t
 7166 | ao_column | ao_column_tableam_handler | t
 7013 | bitmap    | bmhandler                 | i
(10 rows)

gpadmin=# set default_index_type = not_exist;
SET
gpadmin=# show default_index_type;
 default_index_type
--------------------
 not_exist
(1 row)

gpadmin=# create table t3(c1 int unique, c2 int);
ERROR:  access method "not_exist" does not exist
  1. We changed the codes when creating a table in parse stage.
    But doesn't have a case to check, like
create unique index on t3(c1);

And as the GUC is added by CBDB, we should put it in guc_gp.c, not guc.c
I see the GUC is only used at parse stage, does it need to sync to segments(src/include/utils/sync_guc_name.h)?

@avamingli
Copy link
Collaborator

Add some test cases?

@hw118118
Copy link
Contributor Author

hw118118 commented Nov 7, 2023

Add some test cases?

There are many test cases but place into extension UnionStore, because it mainly comply with UnionStore extension.

@HuSen8891 HuSen8891 closed this Nov 24, 2023
@HuSen8891 HuSen8891 reopened this Nov 24, 2023
The purpose of the commit is in support of user-defined index access
method. Currently, there are many restrictions for index access methods
and new index access methods are incompatible with internal framework.
At the same time, it's impossible for user to change default index
access method. So I build some hooks that are for changing default
behavior and being compatible for those strange restrictions.
In assignProcTypes interface, like-btree hook is missing and fix it.
Fix some problems for brin and nodescan and support new index access
methods better.
Restore data that delete by mistake in brin.c.
Combine diffrent index access method hook into a index access method
hook and add a function for whether a index am oid is of a kind of index
am type.
Fix some problems for guc var default_table_access_method and make it
resonable.
Add rely files for index.h
Correct unique rule for transformIndexConstraint and only btree supports
uniqueness.
Correct unique rule for transformIndexConstraint and only btree supports
uniqueness.
Correct unique rule for transformIndexConstraint and only btree supports
uniqueness.
Adjust is_index_access_method_hook hook name
Fix the issue for storage of create operator class.
Fix the issue for storage of create operator class and add test for brin.
Fix the issue for create operator class v1
Fix the issue for create operator class v2.
Fix delete delete.
@hw118118 hw118118 closed this Nov 27, 2023
@hw118118 hw118118 changed the title Feature: add user-defined index access method Feature: add user-defined index access method origin Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants