PHOENIX-3547 Supporting more number of indices per table. #324
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently the number of indices per Phoenix table is bound to maximum of 65535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices. This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtably big enough to cover this requirement. Any existing Phoenix table will still continue to support only maximum of 65535 of indices. A new boolean column (USE_LONG_VIEW_INDEX BOOLEAN DEFAULT FALSE) is added to SYSTEM.CATALOG to specify each Phoenix table's support for large number of indices. On each new Phoenix table creation the value for USE_LONG_VIEW_INDEX will be set to `true` while this value would be false for any existing table.
Currently the number of indices per Phoenix table is bound to maximum of 65535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices. This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtably big enough to cover this requirement. Any existing Phoenix table will still continue to support only maximum of 65535 of indices. A new boolean column (USE_LONG_VIEW_INDEX BOOLEAN DEFAULT FALSE) is added to SYSTEM.CATALOG to specify each Phoenix table's support for large number of indices. On each new Phoenix table creation the value for USE_LONG_VIEW_INDEX will be set to `true` while this value would be false for any existing table.
Closes apache#318 Signed-off-by: Josh Elser <elserj@apache.org>
…ist to ASC sort order always
Currently the number of indices per Phoenix table is bound to maximum of 65535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices. This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtably big enough to cover this requirement. Any existing Phoenix table will still continue to support only maximum of 65535 of indices. A new int column (VIEW_INDEX_ID_DATA_TYPE TINYINT) is added to SYSTEM.CATALOG to specify each Phoenix table's vewIndex data type. On each new Phoenix table creation the value for VIEW_INDEX_ID_DATA_TYPE will be set to `Long` while this value would be `Short` for any existing table.
Currently the number of indices per Phoenix table is bound to maximum of 65535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices. This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtably big enough to cover this requirement. Any existing Phoenix table will still continue to support only maximum of 65535 of indices. A new int column (VIEW_INDEX_ID_DATA_TYPE TINYINT) is added to SYSTEM.CATALOG to specify each Phoenix table's vewIndex data type. On each new Phoenix table creation the value for VIEW_INDEX_ID_DATA_TYPE will be set to `Long` while this value would be `Short` for any existing table.
|
@twdsilva Tested the viewIndexId change from |
|
@m2je Just to confirm the old client was still able to use the view that it had created right ? |
twdsilva
requested changes
Aug 17, 2018
| @@ -382,6 +389,7 @@ | |||
| private static final int DISABLE_WAL_INDEX = TABLE_KV_COLUMNS.indexOf(DISABLE_WAL_KV); | |||
| private static final int MULTI_TENANT_INDEX = TABLE_KV_COLUMNS.indexOf(MULTI_TENANT_KV); | |||
| private static final int VIEW_TYPE_INDEX = TABLE_KV_COLUMNS.indexOf(VIEW_TYPE_KV); | |||
| private static final int VIEW_INDEX_ID_DATA_TYPE = TABLE_KV_COLUMNS.indexOf(VIEW_INDEX_ID_DATA_TYPE_BYTES_KV); | |||
There was a problem hiding this comment.
nit: rename to VIEW_INDEX_ID_DATA_TYPE_INDEX
| } | ||
|
|
||
| private PDataType getViewIndexType(Cell[] tableKeyValues) { | ||
| Cell dataTypeKv = tableKeyValues[VIEW_INDEX_ID_DATA_TYPE]; |
There was a problem hiding this comment.
I think you need a null check here similar to the getViewIndexId. If the view index was created by an old client it won't have the VIEW_INDEX_ID_DATA_TYPE and you can assume its a short.
added 2 commits
August 19, 2018 17:25
Currently the number of indices per Phoenix table is bound to maximum of 65535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices. This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtably big enough to cover this requirement. Any existing Phoenix table will still continue to support only maximum of 65535 of indices. A new int column (VIEW_INDEX_ID_DATA_TYPE TINYINT) is added to SYSTEM.CATALOG to specify each Phoenix table's viewIndex data type. On each new Phoenix table creation the value for VIEW_INDEX_ID_DATA_TYPE will be set to `Long` while this value would be `Short` for any existing table. According to Protobuf documentation https://developers.google.com/protocol-buffers/docs/proto#updating we can change the type of viewIndexId from int32 to int64 and maintain the backward compatibility for older clients. We did a manual verification for this scenario by creating a view with the old client and verify the new client is able to connect and read/write to the view after changing the viewIndexId type and adding the viewIndexType.
… of 65535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices. This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtably big enough to cover this requirement. Any existing Phoenix table will still continue to support only maximum of 65535 of indices. A new int column (VIEW_INDEX_ID_DATA_TYPE TINYINT) is added to SYSTEM.CATALOG to specify each Phoenix table's viewIndex data type. On each new Phoenix table creation the value for VIEW_INDEX_ID_DATA_TYPE will be set to `Long` while this value would be `Short` for any existing table. According to Protobuf documentation https://developers.google.com/protocol-buffers/docs/proto#updating we can change the type of viewIndexId from int32 to int64 and maintain the backward compatibility for older clients. We did a manual verification for this scenario by creating a view with the old client and verify the new client is able to connect and read/write to the view after changing the viewIndexId type and adding the viewIndexType.
|
@twdsilva Yes, the old client was able to interact with no problem. |
|
@m2je This PR has a bunch of many commits, can you please rebase and squash all your commits into a single commit. After that you can generate a .patch file and attach it to the JIRA and the click on the "Submit Patch" button so that we can get a test run. |
|
👍 Closed this and follow up in #334 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently the number of indices per Phoenix table is bound to maximum of 65535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices.
This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtably big enough to cover this requirement.
Any existing Phoenix table will still continue to support only maximum of 65535 of indices.
A new int column (VIEW_INDEX_ID_DATA_TYPE TINYINT) is added to SYSTEM.CATALOG to specify each Phoenix table's viewIndex data type.
On each new Phoenix table creation the value for VIEW_INDEX_ID_DATA_TYPE will be set to
Longwhile this value would beShortfor any existing table.According to Protobuf documentation https://developers.google.com/protocol-buffers/docs/proto#updating we can change the type of viewIndexId from int32 to int64 and maintain the backward compatibility for older clients. We did a manual verification for this scenario by
creating a view with the old client and verify the new client is able to connect and read/write to the view after changing the viewIndexId type and adding the viewIndexType.