Skip to content

Phoenix 6219 GlobalIndexChecker doesn't work for SingleCell indexes#993

Closed
gokceni wants to merge 1 commit into
apache:4.x-PHOENIX-5923from
gokceni:PHOENIX-6219
Closed

Phoenix 6219 GlobalIndexChecker doesn't work for SingleCell indexes#993
gokceni wants to merge 1 commit into
apache:4.x-PHOENIX-5923from
gokceni:PHOENIX-6219

Conversation

@gokceni
Copy link
Copy Markdown
Contributor

@gokceni gokceni commented Dec 1, 2020

No description provided.

@gokceni gokceni changed the title Phoenix 6219 Phoenix 6219 GlobalIndexChecker doesn't work for SingleCell indexes Dec 1, 2020
@gokceni
Copy link
Copy Markdown
Contributor Author

gokceni commented Dec 1, 2020

private final boolean mutable;
private final boolean useSnapshot;

private static final Logger LOGGER = LoggerFactory.getLogger(IndexExtendedIT.class);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This object is not being used

@@ -326,10 +326,10 @@ private void repairIndexRows(byte[] indexRowKey, long ts, List<Cell> row) throws
buildIndexScan.setAttribute(BaseScannerRegionObserver.REBUILD_INDEXES, TRUE_BYTES);
buildIndexScan.setAttribute(BaseScannerRegionObserver.SKIP_REGION_BOUNDARY_CHECK, Bytes.toBytes(true));
// Scan only columns included in the index table plus the empty column
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this comment still applicable ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it is. The difference is that in getAllColumns, you get index cols. The getAllColumnsForDataTable, you get the mapping columns in data table. For example: in index there is Col1 but in hbase it is in a cell with others with column qualifier:1. But in data table it is in its own cell with col qualifier Col1.

if (!dataTable.isTransactional()
|| !dataTable.getTransactionProvider().getTransactionProvider().isUnsupported(Feature.MAINTAIN_LOCAL_INDEX_ON_SERVER)) {
indexesItr = maintainedLocalIndexes(indexes.iterator());
indexesItr = maintainedLocalOrGlobalIndexesWithoutMatchingStorageScheme(dataTable, indexes.iterator());
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this still be maintainedLocalIndexes since it is called inside if (onlyLocalIndexex)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately onlyLocalIndexes variable name is misleading. So it is immutableIndexes or transactional indexes
boolean onlyLocalIndexes = dataTable.isImmutableRows() || dataTable.isTransactional();

Copy link
Copy Markdown
Contributor

@gjacoby126 gjacoby126 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some nits

conn.commit();

stmt.execute(String.format("CREATE " + (localIndex ? "LOCAL" : "") + " INDEX %s ON %s (UPPER(NAME, 'en_US')) ASYNC ", indexTableName,dataTableFullName));
stmt.execute(String.format("CREATE " + (localIndex ? "LOCAL" : "") + " INDEX %s ON %s (UPPER(NAME, 'en_US')) ASYNC " + this.indexDDLOptions, indexTableName,dataTableFullName));
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can use String.format parameter rather than concatenating indexDDLOptions


stmt.execute(
String.format("CREATE INDEX %s ON %s (UPPER(NAME, 'en_US')) ", indexName,
String.format("CREATE INDEX %s ON %s (UPPER(NAME, 'en_US')) " + this.indexDDLOptions, indexName,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto, String.format vs concatenation

conn.commit();
String
sql =
"UPSERT /*+ NO_INDEX */ INTO " + idxName + "(\":PK1\",\":INT_PK\",\"0:V1\",\"0:V2\",\"0:V4\") SELECT /*+ NO_INDEX */ PK1,INT_PK, V1, V2,V4 FROM "
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does a NO_INDEX hint in an upsert clause do anything?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I copy pasted this from PostIndexDDLCompilerTest. I think it is valid but not mean much since this is already the index. Let me remove it

conn.commit();
String
sql =
"UPSERT /*+ NO_INDEX */ INTO " + idxName + "(\":PK1\",\":INT_PK\",\"0:V1\",\"0:V2\",\"0:V4\") SELECT /*+ NO_INDEX */ PK1,INT_PK, V1, V2,V4 FROM "
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: overly long line

@stoty
Copy link
Copy Markdown
Contributor

stoty commented Dec 4, 2020

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 5s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x-PHOENIX-5923 Compile Tests _
+1 💚 mvninstall 10m 53s 4.x-PHOENIX-5923 passed
+1 💚 compile 0m 56s 4.x-PHOENIX-5923 passed
+1 💚 checkstyle 1m 7s 4.x-PHOENIX-5923 passed
+1 💚 javadoc 0m 43s 4.x-PHOENIX-5923 passed
+0 🆗 spotbugs 2m 56s phoenix-core in 4.x-PHOENIX-5923 has 946 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚 mvninstall 5m 11s the patch passed
+1 💚 compile 0m 54s the patch passed
+1 💚 javac 0m 54s the patch passed
-1 ❌ checkstyle 1m 7s phoenix-core: The patch generated 45 new + 1488 unchanged - 26 fixed = 1533 total (was 1514)
+1 💚 whitespace 0m 0s The patch has no whitespace issues.
+1 💚 javadoc 0m 42s the patch passed
-1 ❌ spotbugs 3m 7s phoenix-core generated 1 new + 944 unchanged - 2 fixed = 945 total (was 946)
_ Other Tests _
-1 ❌ unit 186m 42s phoenix-core in the patch failed.
-1 ❌ asflicense 0m 41s The patch generated 1 ASF License warnings.
218m 47s
Reason Tests
FindBugs module:phoenix-core
index must be non-null but is marked as nullable At IndexMaintainer.java:is marked as nullable At IndexMaintainer.java:[line 188]
Failed junit tests phoenix.end2end.RebuildIndexConnectionPropsIT
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #993
Optional Tests dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
uname Linux 518799a48a85 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev/phoenix-personality.sh
git revision 4.x-PHOENIX-5923 / a3df4dc
Default Java Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
spotbugs https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/2/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html
unit https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/2/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Results https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/2/testReport/
asflicense https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/2/artifact/yetus-general-check/output/patch-asflicense-problems.txt
Max. process+thread count 3482 (vs. ulimit of 30000)
modules C: phoenix-core U: phoenix-core
Console output https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/2/console
versions git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Copy Markdown
Contributor

@gjacoby126 gjacoby126 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, thanks @gokceni

@gokceni
Copy link
Copy Markdown
Contributor Author

gokceni commented Dec 8, 2020

Thanks @gjacoby126 for the review! I added one more test to SingleCellIndexIT called testMultipleColumnFamilies

@stoty
Copy link
Copy Markdown
Contributor

stoty commented Dec 8, 2020

💔 -1 overall

Vote Subsystem Runtime Comment
+0 🆗 reexec 1m 12s Docker mode activated.
_ Prechecks _
+1 💚 dupname 0m 0s No case conflicting files found.
+1 💚 hbaseanti 0m 0s Patch does not have any anti-patterns.
+1 💚 @author 0m 0s The patch does not contain any @author tags.
-1 ❌ test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
_ 4.x-PHOENIX-5923 Compile Tests _
+1 💚 mvninstall 10m 48s 4.x-PHOENIX-5923 passed
+1 💚 compile 0m 56s 4.x-PHOENIX-5923 passed
+1 💚 checkstyle 1m 7s 4.x-PHOENIX-5923 passed
+1 💚 javadoc 0m 42s 4.x-PHOENIX-5923 passed
+0 🆗 spotbugs 2m 53s phoenix-core in 4.x-PHOENIX-5923 has 946 extant spotbugs warnings.
_ Patch Compile Tests _
+1 💚 mvninstall 5m 13s the patch passed
+1 💚 compile 0m 56s the patch passed
+1 💚 javac 0m 56s the patch passed
-1 ❌ checkstyle 1m 9s phoenix-core: The patch generated 38 new + 1496 unchanged - 19 fixed = 1534 total (was 1515)
+1 💚 whitespace 0m 1s The patch has no whitespace issues.
+1 💚 javadoc 0m 42s the patch passed
-1 ❌ spotbugs 3m 8s phoenix-core generated 1 new + 944 unchanged - 2 fixed = 945 total (was 946)
_ Other Tests _
-1 ❌ unit 192m 33s phoenix-core in the patch failed.
-1 ❌ asflicense 0m 41s The patch generated 1 ASF License warnings.
224m 45s
Reason Tests
FindBugs module:phoenix-core
index must be non-null but is marked as nullable At IndexMaintainer.java:is marked as nullable At IndexMaintainer.java:[line 188]
Failed junit tests TEST-[RangeScanIT_0]
phoenix.end2end.ParameterizedIndexUpgradeToolIT
Subsystem Report/Notes
Docker ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/3/artifact/yetus-general-check/output/Dockerfile
GITHUB PR #993
Optional Tests dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
uname Linux cd2a74a09081 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool maven
Personality dev/phoenix-personality.sh
git revision 4.x-PHOENIX-5923 / a3df4dc
Default Java Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/3/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
spotbugs https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/3/artifact/yetus-general-check/output/new-spotbugs-phoenix-core.html
unit https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/3/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Results https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/3/testReport/
asflicense https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/3/artifact/yetus-general-check/output/patch-asflicense-problems.txt
Max. process+thread count 6250 (vs. ulimit of 30000)
modules C: phoenix-core U: phoenix-core
Console output https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-993/3/console
versions git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

Copy link
Copy Markdown
Contributor

@kadirozde kadirozde left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, Thanks

@gokceni
Copy link
Copy Markdown
Contributor Author

gokceni commented Dec 8, 2020

Merged

@gokceni gokceni closed this Dec 8, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants