PHOENIX-6192 : Use tenant connection to resolve tenant views in syncUpdateCacheFreqAllIndexes() #928

virajjasani · 2020-10-19T13:20:16Z

No description provided.

stoty · 2020-10-19T17:53:30Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	1m 6s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	36m 32s	master passed
+1 💚	compile	0m 57s	master passed
+1 💚	checkstyle	0m 59s	master passed
+1 💚	javadoc	0m 43s	master passed
+0 🆗	spotbugs	2m 51s	phoenix-core in master has 973 extant spotbugs warnings.
		_ Patch Compile Tests _
+1 💚	mvninstall	32m 54s	the patch passed
+1 💚	compile	0m 50s	the patch passed
+1 💚	javac	0m 50s	the patch passed
-1 ❌	checkstyle	1m 0s	phoenix-core: The patch generated 12 new + 1278 unchanged - 11 fixed = 1290 total (was 1289)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 42s	the patch passed
+1 💚	spotbugs	3m 2s	the patch passed
		_ Other Tests _
-1 ❌	unit	172m 45s	phoenix-core in the patch failed.
+1 💚	asflicense	0m 26s	The patch does not generate ASF License warnings.
		257m 29s

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-928/2/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#928
Optional Tests	dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
uname	Linux 1884fa74806d 4.15.0-65-generic #74-Ubuntu SMP Tue Sep 17 17:06:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / `5b58d9b`
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-928/2/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-928/2/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-928/2/testReport/
Max. process+thread count	6751 (vs. ulimit of 30000)
modules	C: phoenix-core U: phoenix-core
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-928/2/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

yanxinyi · 2020-10-21T18:13:04Z

phoenix-core/src/it/java/org/apache/phoenix/end2end/GlobalConnectionTenantTable2IT.java

+            long cacheFreq = rs.getLong(1);
+            assertEquals(0, cacheFreq);
+
+            updateCacheFreq(conn, null, TABLE_NAME, 500);


nit: maybe use tableNameFreqVal instead of entering a numerical value?

ChinmaySKulkarni · 2020-10-21T17:46:00Z

phoenix-core/src/it/java/org/apache/phoenix/end2end/GlobalConnectionTenantTable2IT.java

+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class GlobalConnectionTenantTable2IT extends BaseTest {


Just wondering, can these tests can't be added to GlobalConnectionTenantTableIT itself?

In case the answer is no for some reason, can we rename this test class to be more representative of the actual test i.e. all tests are related to syncUpdateCacheFreq so maybe rename it to show that

ChinmaySKulkarni · 2020-10-21T17:48:14Z

phoenix-core/src/it/java/org/apache/phoenix/end2end/GlobalConnectionTenantTable2IT.java

+        }
+    }
+
+    private void updateCacheFreq(Connection conn, String tenantId,


nit: Maybe this method should be called updateUpdateCacheFreq() ;)

ChinmaySKulkarni · 2020-10-21T17:51:56Z

phoenix-core/src/it/java/org/apache/phoenix/end2end/GlobalConnectionTenantTable2IT.java

+                "SELECT UPDATE_CACHE_FREQUENCY FROM SYSTEM.CATALOG WHERE TABLE_NAME='"
+                    + TABLE_NAME + "'");
+            rs.next();
+            long cacheFreq = rs.getLong(1);


nit: Not sure we need to assign rs.get..() to a variable each time for tenantId, ucf, etc. Up to you if you think it's good for clarity (I don't feel strongly about it).

ChinmaySKulkarni · 2020-10-21T17:54:48Z

phoenix-core/src/it/java/org/apache/phoenix/end2end/GlobalConnectionTenantTable2IT.java

+            assertEquals(500, cacheFreq);
+
+            rs = conn.createStatement().executeQuery(
+                "SELECT TENANT_ID,TABLE_SCHEM,UPDATE_CACHE_FREQUENCY FROM SYSTEM.CATALOG"


Extract this query into a private final String member variable to avoid duplication

ChinmaySKulkarni · 2020-10-21T17:55:44Z

phoenix-core/src/it/java/org/apache/phoenix/end2end/GlobalConnectionTenantTable2IT.java

+            updateCacheFreq(conn, TENANT_NAME, VIEW1_NAME, 999);
+            updateCacheFreq(conn, TENANT_NAME, VIEW1_INDEX_NAME, 888);
+
+            rs = conn.createStatement().executeQuery(


nit: these 4-5 steps are repeated multiple times so maybe worth extracting to a small helper method..something like assertUpdateCacheFreqValue())

ChinmaySKulkarni · 2020-10-21T17:56:23Z

phoenix-core/src/it/java/org/apache/phoenix/end2end/GlobalConnectionTenantTable2IT.java

+            cacheFreq = rs.getLong(3);
+            assertEquals(TENANT_NAME,tenantId);
+            assertEquals(SCHEMA_NAME, schemaName);
+            assertEquals(500, cacheFreq);


Similarly, shouldn't this be 0 since VIEW2 has 0 set?

ChinmaySKulkarni · 2020-10-21T17:58:16Z

phoenix-core/src/it/java/org/apache/phoenix/end2end/GlobalConnectionTenantTable2IT.java

+        try (Connection conn = getTenantConnection(TENANT_NAME)) {
+            createView(conn, SCHEMA_NAME, VIEW1_NAME, TABLE_NAME);
+            createViewIndex(conn, SCHEMA_NAME, VIEW1_INDEX_NAME, VIEW1_NAME, VIEW_INDEX_COL);
+            createView(conn, SCHEMA_NAME, VIEW2_NAME, VIEW1_NAME);


Would be better to try out a couple of different TENANT_IDs to make sure that syncUpdateCacheFreqAllIndexes() changes the tenant_id to use as per each view that is found

ChinmaySKulkarni · 2020-10-21T18:14:02Z

phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java

+            throws SQLException {
+        String viewName = SchemaUtil.getTableName(tableInfo.getSchemaName(),
+            tableInfo.getTableName());
+        String viewTenantId = Bytes.toString(tableInfo.getTenantId());


It will be better to resolve the PTable for the view (either using tenanted connection if required or a global connection if a global view) and then call syncUpdateCacheFreqForIndexesOfTable() just once. Basically, combine this method and getViewAndSyncCacheFreqForIndexes().

ChinmaySKulkarni · 2020-10-21T18:14:36Z

phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java

@@ -1400,6 +1393,43 @@ public static void syncUpdateCacheFreqAllIndexes(PhoenixConnection conn, PTable
        }
    }

+    private static void iterateOverChildViewAndSyncCacheFreq(


Rename this method since we aren't iterating over anything. Maybe after combining this with getViewAndSyncCacheFreqForIndexes() we can just call it the latter.

ChinmaySKulkarni · 2020-10-21T18:15:05Z

phoenix-core/src/main/java/org/apache/phoenix/util/UpgradeUtil.java

+            view = PhoenixRuntime.getTable(conn, viewName);
+        } catch (TableNotFoundException e) {
+            // Ignore
+            LOGGER.warn("Error getting PTable for view: {}", viewName);


This should be ERROR level

yanxinyi · 2020-10-21T19:03:09Z

phoenix-core/src/it/java/org/apache/phoenix/end2end/GlobalConnectionTenantTable2IT.java

+            cacheFreq = rs.getLong(1);
+            assertEquals(0, cacheFreq);
+
+            updateCacheFreq(conn, TENANT_NAME, VIEW1_NAME, 999);


nit: maybe use viewName1FreqVal instead of entering a numerical value?

yanxinyi · 2020-10-21T19:04:56Z

@virajjasani thanks for the patch. Can you rebase and solve the conflict?

…pdateCacheFreqAllIndexes()

stoty · 2020-10-22T12:41:57Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	0m 0s	Docker mode activated.
-1 ❌	docker	3m 33s	Docker failed to build yetus/phoenix:10ef71711.

Subsystem	Report/Notes
GITHUB PR	#928
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-928/3/console
versions	git=2.17.1
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

virajjasani · 2020-10-22T12:44:37Z

Thanks for the review @ChinmaySKulkarni @yanxinyi . I have addressed your comments, could you please take a look?

stoty · 2020-10-22T16:27:28Z

💔 -1 overall

Vote	Subsystem	Runtime	Comment
+0 🆗	reexec	6m 42s	Docker mode activated.
		_ Prechecks _
+1 💚	dupname	0m 0s	No case conflicting files found.
+1 💚	hbaseanti	0m 0s	Patch does not have any anti-patterns.
+1 💚	@author	0m 0s	The patch does not contain any @author tags.
-1 ❌	test4tests	0m 0s	The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
		_ master Compile Tests _
+1 💚	mvninstall	15m 46s	master passed
+1 💚	compile	1m 11s	master passed
+1 💚	checkstyle	1m 1s	master passed
+1 💚	javadoc	1m 1s	master passed
+0 🆗	spotbugs	3m 59s	phoenix-core in master has 970 extant spotbugs warnings.
		_ Patch Compile Tests _
+1 💚	mvninstall	10m 11s	the patch passed
+1 💚	compile	1m 9s	the patch passed
+1 💚	javac	1m 9s	the patch passed
-1 ❌	checkstyle	1m 0s	phoenix-core: The patch generated 4 new + 1296 unchanged - 3 fixed = 1300 total (was 1299)
+1 💚	whitespace	0m 0s	The patch has no whitespace issues.
+1 💚	javadoc	0m 55s	the patch passed
+1 💚	spotbugs	4m 2s	the patch passed
		_ Other Tests _
-1 ❌	unit	174m 50s	phoenix-core in the patch failed.
+1 💚	asflicense	0m 28s	The patch does not generate ASF License warnings.
		225m 0s

Reason	Tests
Failed junit tests	phoenix.end2end.index.IndexUsageIT

Subsystem	Report/Notes
Docker	ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-928/4/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#928
Optional Tests	dupname asflicense javac javadoc unit spotbugs hbaseanti checkstyle compile
uname	Linux 0da729dfccb7 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev/phoenix-personality.sh
git revision	master / `8c81131`
Default Java	Private Build-1.8.0_242-8u242-b08-0ubuntu3~16.04-b08
checkstyle	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-928/4/artifact/yetus-general-check/output/diff-checkstyle-phoenix-core.txt
unit	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-928/4/artifact/yetus-general-check/output/patch-unit-phoenix-core.txt
Test Results	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-928/4/testReport/
Max. process+thread count	6823 (vs. ulimit of 30000)
modules	C: phoenix-core U: phoenix-core
Console output	https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-PreCommit-GitHub-PR/job/PR-928/4/console
versions	git=2.7.4 maven=3.3.9 spotbugs=4.1.3
Powered by	Apache Yetus 0.12.0 https://yetus.apache.org

This message was automatically generated.

ChinmaySKulkarni

+1. Thanks @virajjasani

yanxinyi

thanks for the patch

yanxinyi reviewed Oct 21, 2020

View reviewed changes

ChinmaySKulkarni requested changes Oct 21, 2020

View reviewed changes

yanxinyi reviewed Oct 21, 2020

View reviewed changes

PHOENIX-6192 : Use tenant connection to resolve tenant views in syncU…

dcebf51

…pdateCacheFreqAllIndexes()

virajjasani requested a review from ChinmaySKulkarni October 22, 2020 12:46

ChinmaySKulkarni approved these changes Oct 23, 2020

View reviewed changes

yanxinyi approved these changes Oct 23, 2020

View reviewed changes

virajjasani closed this Oct 25, 2020

virajjasani deleted the PHOENIX-6192-master branch October 25, 2020 17:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PHOENIX-6192 : Use tenant connection to resolve tenant views in syncUpdateCacheFreqAllIndexes() #928

PHOENIX-6192 : Use tenant connection to resolve tenant views in syncUpdateCacheFreqAllIndexes() #928

virajjasani commented Oct 19, 2020

stoty commented Oct 19, 2020

yanxinyi Oct 21, 2020

ChinmaySKulkarni Oct 21, 2020

ChinmaySKulkarni Oct 21, 2020

ChinmaySKulkarni Oct 21, 2020

ChinmaySKulkarni Oct 21, 2020

ChinmaySKulkarni Oct 21, 2020

ChinmaySKulkarni Oct 21, 2020

ChinmaySKulkarni Oct 21, 2020

ChinmaySKulkarni Oct 21, 2020

ChinmaySKulkarni Oct 21, 2020

ChinmaySKulkarni Oct 21, 2020

ChinmaySKulkarni Oct 21, 2020

yanxinyi Oct 21, 2020

yanxinyi commented Oct 21, 2020

stoty commented Oct 22, 2020

virajjasani commented Oct 22, 2020

stoty commented Oct 22, 2020

ChinmaySKulkarni left a comment

yanxinyi left a comment

PHOENIX-6192 : Use tenant connection to resolve tenant views in syncUpdateCacheFreqAllIndexes() #928

PHOENIX-6192 : Use tenant connection to resolve tenant views in syncUpdateCacheFreqAllIndexes() #928

Conversation

virajjasani commented Oct 19, 2020

stoty commented Oct 19, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yanxinyi commented Oct 21, 2020

stoty commented Oct 22, 2020

virajjasani commented Oct 22, 2020

stoty commented Oct 22, 2020

ChinmaySKulkarni left a comment

Choose a reason for hiding this comment

yanxinyi left a comment

Choose a reason for hiding this comment