PHOENIX-3547 Supporting more number of indices per table. #334
Conversation
|
@twdsilva I can see the Jenkins job is complaining about the some lines being bigger than 100 chars. Many of those had been there and my commit just changed part of it. I can fix them all but not sure if this is expected from this PR. |
| } | ||
| if (result.getViewIndexId() != null) { | ||
| builder.setViewIndexId(result.getViewIndexId()); | ||
| builder.setViewIndexType(result.getViewIndexType().getSqlType()); |
There was a problem hiding this comment.
Can you add a null check for result.getViewIndexType()?
| @@ -100,7 +100,7 @@ public static TableRef contructSchemaTable(PhoenixStatement statement, List<Quer | |||
| UNION_SCHEMA_NAME, UNION_TABLE_NAME, PTableType.SUBQUERY, null, | |||
| HConstants.LATEST_TIMESTAMP, scn == null ? HConstants.LATEST_TIMESTAMP : scn, | |||
| null, null, projectedColumns, null, null, null, true, null, null, null, true, | |||
| true, true, null, null, null, false, null, 0, 0L, | |||
| true, true, null,null, null, null, false, null, 0, 0L, | |||
m2je
force-pushed
the
PHOENIX-3547
branch
2 times, most recently
from
338584f
to
0708ae9
Compare
Currently the number of indices per Phoenix table is bound to maximum of 65,535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices. This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtedly big enough to cover this requirement. Any existing Phoenix table will still continue to support only maximum of 65535 of indices. A new int column (VIEW_INDEX_ID_DATA_TYPE TINYINT) is added to SYSTEM.CATALOG to specify each Phoenix table's viewIndex data type. On each new Phoenix table creation the value for VIEW_INDEX_ID_DATA_TYPE will be set to Long while this value would be Short for any existing table. According to Protobuf documentation https://developers.google.com/protocol-buffers/docs/proto#updating we can change the type of viewIndexId from int32 to int64 and maintain the backward compatibility for older clients. We did a manual verification for this scenario by creating a view with the old client and verify the new client is able to connect and read/write to the view after changing the viewIndexId type and adding the viewIndexType.
|
@twdsilva all done 👍 |
|
@m2je I was able to apply this PR to the 4.x branches, but it didn't apply cleanly to the master branch. Can you please attach a patch the the JIRA that applies cleanly to the master branch ? |
|
@twdsilva Sure, The only issue I'm seeing is the each 4.x branch also need a separated patches. Let me prepare them. |
|
@twdsilva Added all patches to the ticket. Unfortunately they seem to all need to have their own patch except
|
|
@m2je Thank you for the contribution. I have committed this patch to the 4.x and master branches. |
|
Thanks @twdsilva |
Currently the number of indices per Phoenix table is bound to maximum of 65,535 (java.lang.Short) which is a limitation for applications requiring to have unlimited number of indices.
This change will consider any new table created in Phoenix to support view index ids to be in the range of -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 (java.lang.Long) which is undoubtedly big enough to cover this requirement.
Any existing Phoenix table will still continue to support only maximum of 65535 of indices.
A new int column (VIEW_INDEX_ID_DATA_TYPE TINYINT) is added to SYSTEM.CATALOG to specify each Phoenix table's viewIndex data type.
On each new Phoenix table creation the value for VIEW_INDEX_ID_DATA_TYPE will be set to Long while this value would be Short for any existing table.
According to Protobuf documentation https://developers.google.com/protocol-buffers/docs/proto#updating we can change the type of viewIndexId from int32 to int64 and maintain the backward compatibility for older clients. We did a manual verification for this scenario by
creating a view with the old client and verify the new client is able to connect and read/write to the view after changing the viewIndexId type and adding the viewIndexType.