Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LPS-121714 Service builder should remove duplicate unique indexes #9483

Closed
wants to merge 6 commits into from

Conversation

Preston-Crary
Copy link

Hi @shuyangzhou,

Removing duplicate finders looks a bit tricky since it's not always clear to me which one should be removed and timing seems bad for major version changes. So for now just cleaning up the indexes, some unique indexes got left behind from changing constraints, those needed to be removed manually first in 28becca. I'm not sure if we need upgrades for all of these or if we can wait until the automatic index cleanup is run by future upgrades. All these changes only work for unique indexes because otherwise there are issues with ranged comparators (unique has to be ==), see d267df9.

@liferay-continuous-integration
Copy link
Collaborator

To conserve resources, the PR Tester does not automatically run for every pull.

If your code changes were already tested in another pull, reference that pull in this pull so the test results can be analyzed.

If your pull was never tested, comment "ci:test" to run the PR Tester for this pull.

@Preston-Crary
Copy link
Author

ci:test:sf

@Preston-Crary
Copy link
Author

ci:test:relevant

@liferay-continuous-integration
Copy link
Collaborator

✔️ ci:test:sf - 1 out of 1 jobs passed in 4 minutes

Click here for more details.

Base Branch:

Branch Name: master
Branch GIT ID: a9159f9538b844a43e6d0d4a6f0633627aac4c27

Sender Branch:

Branch Name: LPS-121714
Branch GIT ID: d267df9a40c46f3e47a8438dad5e5417327ebabe

1 out of 1jobs PASSED
1 Successful Jobs:
For more details click here.

@liferay-continuous-integration
Copy link
Collaborator

@liferay-continuous-integration
Copy link
Collaborator

✔️ ci:test:stable - 14 out of 14 jobs passed

❌ ci:test:relevant - 87 out of 91 jobs passed in 3 hours 27 minutes

Click here for more details.

Base Branch:

Branch Name: master
Branch GIT ID: 3229e46de503ae94ab8fc8c7b0ccdbf552fe062e

Copied in Private Modules Branch:

Branch Name: master-private
Branch GIT ID: 13c59df02738cba7c7387b3490775720676498c7

ci:test:stable - 14 out of 14 jobs PASSED
14 Successful Jobs:
ci:test:relevant - 87 out of 91 jobs PASSED
87 Successful Jobs:
For more details click here.

Failures unique to this pull:

  1. test-portal-acceptance-pullrequest-batch(master)/modules-integration-mysql57-jdk8
    Job Results:

    6452 Tests Passed.
    5 Tests Failed.

    1. AXIS_VARIABLE=4,label_exp=!master #200409
      1. CommerceSubscriptionEntryTest.testOverrideCPInstanceSubscriptionInfoAllDisabled
        java.lang.NullPointerException: 
        [
        SCENARIO Add an order
        GIVEN One subscription disabled product with two SKUs
         BUT One of them overrides the subscription info
        WHEN Order is placed
        THEN No subscription entries are created]
        
        at com.liferay.commerce.subscription.test.CommerceSubscriptionEntryTest._testOverrideCPInstanceSubscriptionInfo(CommerceSubscriptionEntryTest.java:285)
        at com.liferay.commerce.subscription.test.CommerceSubscriptionEntryTest.testOverrideCPInstanceSubscriptionInfoAllDisabled(CommerceSubscriptionEntryTest.java:156)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.liferay.arquillian.extension.junit.bridge.server.TestExecutorRunnable$2.evaluate(TestExecutorRunnable.java:234)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.li...</code></pre></div></li><li><div><a href="https://test-1-18.liferay.com/job/test-portal-acceptance-pullrequest-batch(master)/AXIS_VARIABLE=4,label_exp=!master/200409//testReport/com.liferay.commerce.subscription.test/CommerceSubscriptionEntryTest/testOverrideCPInstanceSubscriptionInfoAllEnabled">CommerceSubscriptionEntryTest.testOverrideCPInstanceSubscriptionInfoAllEnabled</a><pre><code>java.lang.Exception: 
        

        [
        SCENARIO Add an order
        GIVEN One subscription enabled product with two SKUs
        BUT One of them overrides the subscription info
        WHEN Subscription entries are created
        THEN It should not be possible to add two subscription order items in the same order
        ]
        [
        Message: Unexpected exception, expected<com.liferay.commerce.exception.CommerceOrderValidatorException> but was<java.lang.NullPointerException>
        ]
        at com.liferay.arquillian.extension.junit.bridge.server.TestExecutorRunnable._processExpectedException(TestExecutorRunnable.java:298)
        at com.liferay.arquillian.extension.junit.bridge.server.TestExecutorRunnable.access$300(TestExecutorRunnable.java:66)
        at com.liferay.arquillian.extension.junit.bridge.server.TestExecutorRunnable$2.evaluate(TestExecutorRunnable.java:261)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(Ab...

      2. CommerceSubscriptionEntryTest.testOverrideCPInstanceSubscriptionInfoDefinitionEnabled
        java.lang.NullPointerException:
        [
        SCENARIO Add an order
        GIVEN One subscription enabled product with two SKUs
        BUT One of them overrides the subscription info
        WHEN A subscription entry is created
        THEN It should have right subscription info]

        at com.liferay.commerce.subscription.test.CommerceSubscriptionEntryTest._testOverrideCPInstanceSubscriptionInfo(CommerceSubscriptionEntryTest.java:285)
        at com.liferay.commerce.subscription.test.CommerceSubscriptionEntryTest.testOverrideCPInstanceSubscriptionInfoDefinitionEnabled(CommerceSubscriptionEntryTest.java:195)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.liferay.arquillian.extension.junit.bridge.server.TestExecutorRunnable$2.evaluate(TestExecutorRunnable.java:234)
        at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
        at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTestRule.java:99)
        at com.liferay.portal.kernel.test.rule.AbstractTestRule$2.evaluate(AbstractTest...</code></pre></div></li><li>...</li></ol></div></li></ol></div></li></ol><hr/><details><summary><strong>Failures in common with <a href="https://test-1-1.liferay.com/job/test-portal-acceptance-upstream-dxp(master)">acceptance upstream results</a> at b5a410ba533a7ae80112d850306bc6fd52732391:</strong></summary><ol><li><div><h5><a href="https://test-1-9.liferay.com/job/test-portal-acceptance-pullrequest-batch(master)/361525/">test-portal-acceptance-pullrequest-batch(master)/functional-tomcat90-mysql57-jdk8/37</a></h5><div><h6>Job Results:</h6><p>0 Tests Passed.<br/>1 Test Failed.</p></div><ol><li><div><a href="https://test-1-9.liferay.com/job/test-portal-acceptance-pullrequest-batch(master)/AXIS_VARIABLE=1,label_exp=!master/361525//consoleText">AXIS_VARIABLE=1,label_exp=!master #361525</a><pre><code>delete-liferay-home:
         [exec] mysql: [Warning] Using a password on the command line interface can be insecure.
         [exec] ERROR 2026 (HY000): SSL connection error: protocol version mismatch
        

        [stopwatch] [run.batch.test.action: -372.178 sec]
        [echo] The following error occurred while executing this line:
        [echo] /opt/dev/projects/github/liferay-portal/build-test-batch.xml:359: The following error occurred while executing this line:
        [echo] /opt/dev/projects/github/liferay-portal/build-test-batch.xml:1579: /opt/dev/projects/github/liferay-portal/build-test-batch.xml:1583: The following error occurred while executing this line:
        [echo] /opt/dev/projects/github/liferay-portal/build-test-tomcat.xml:50: The following error occurred while executing this line:
        [echo] /opt/dev/projects/github/liferay-portal/build-test-tomcat.xml:62: The following error occurred while executing this line:
        [echo] /opt/dev/projects/github/liferay-portal/build-test.xml:12821: The following error occurred while executing this line:
        [echo] /opt/dev/projects/github/liferay-portal/build-test.xml:11352: The following error occurred while executing this line:
        [echo] /opt/dev/projects/github/liferay-portal/build-common.xml:528: The following error occurred while executing this line:
        [echo] /opt/dev/projects/github/liferay-portal/build-common.xml:647: exec returned: 1
        [get] Getting: http://test-1-9/job/test-portal-acceptance-pullrequest-batch(master)/AXIS_VARIABLE=1,label_exp=!master/361525//consoleText
        [get] To: /opt/dev/projects/github/liferay-portal/20201006190936933.txt
        [delete] Deleting: /opt/dev/projects/github/liferay-portal/20201006190936933.txt
        [delete] Deleting: /opt/dev/projects/github/liferay-portal/null88386560.properties

    2. test-portal-acceptance-pullrequest-batch(master)/modules-integration-mysql57-jdk8
      Job Results:

      6452 Tests Passed.
      2 Tests Failed.

      1. AXIS_VARIABLE=16,label_exp=!master #200409
        1. PortalLogAssertorTest.testScanXMLLog
          junit.framework.AssertionFailedError:
          Unable to connect to a valid mail server. Please make sure one is properly configured: Couldn't connect to host, port: localhost, 25; timeout -1
          at com.liferay.portal.log.assertor.PortalLogAssertorTest.scanXMLLogFile(PortalLogAssertorTest.java:171)
          at com.liferay.portal.log.assertor.PortalLogAssertorTest$1.visitFile(PortalLogAssertorTest.java:98)
          at com.liferay.portal.log.assertor.PortalLogAssertorTest$1.visitFile(PortalLogAssertorTest.java:88)
          at java.nio.file.Files.walkFileTree(Files.java:2670)
          at java.nio.file.Files.walkFileTree(Files.java:2742)
          at com.liferay.portal.log.assertor.PortalLogAssertorTest.testScanXMLLog(PortalLogAssertorTest.java:86)
      2. AXIS_VARIABLE=4,label_exp=!master #200409
        1. PortalLogAssertorTest.testScanXMLLog
          junit.framework.AssertionFailedError:
          There is no base price list configured for the current catalog
          at com.liferay.portal.log.assertor.PortalLogAssertorTest.scanXMLLogFile(PortalLogAssertorTest.java:171)
          at com.liferay.portal.log.assertor.PortalLogAssertorTest$1.visitFile(PortalLogAssertorTest.java:98)
          at com.liferay.portal.log.assertor.PortalLogAssertorTest$1.visitFile(PortalLogAssertorTest.java:88)
          at java.nio.file.Files.walkFileTree(Files.java:2670)
          at java.nio.file.Files.walkFileTree(Files.java:2742)
          at com.liferay.portal.log.assertor.PortalLogAssertorTest.testScanXMLLog(PortalLogAssertorTest.java:86)

@liferay-continuous-integration
Copy link
Collaborator

@shuyangzhou
Copy link
Owner

ci:forward

@liferay-continuous-integration
Copy link
Collaborator

CI is automatically triggering the following test suites:

  •     ci:test:relevant
  •     ci:test:sf

The pull request will automatically be forwarded to the user brianchandotcom if the following test suites pass:

  •     ci:test:relevant
  •     ci:test:sf
  •     ci:test:stable

@liferay-continuous-integration
Copy link
Collaborator

Skipping previously passed test suites:
ci:test:sf

@liferay-continuous-integration
Copy link
Collaborator

✔️ ci:test:stable - 14 out of 14 jobs passed

✔️ ci:test:relevant - 89 out of 91 jobs passed in 2 hours 24 minutes

Click here for more details.

Base Branch:

Branch Name: master
Branch GIT ID: c294aae2b145b10f873a318b43b35e70115228b6

Copied in Private Modules Branch:

Branch Name: master-private
Branch GIT ID: 13c59df02738cba7c7387b3490775720676498c7

ci:test:stable - 14 out of 14 jobs PASSED
14 Successful Jobs:
ci:test:relevant - 88 out of 91 jobs PASSED
88 Successful Jobs:
For more details click here.

This pull contains no unique failures.


Failures in common with acceptance upstream results at b5a410b:

@liferay-continuous-integration
Copy link
Collaborator

All required test suite(s) passed.
Forwarding pullrequest to brianchandotcom.

@liferay-continuous-integration
Copy link
Collaborator

Pull request has been successfully forwarded to brianchandotcom#94826

@liferay-continuous-integration
Copy link
Collaborator

@marianoalvarosaiz
Copy link

@shuyangzhou, @Preston-Crary why did we implemented this? From a constraint point of view it makes sense, but from an index one they aren't redundant.

For example we've removed:
create unique index IX_9B44D13C on JournalArticleLocalization (companyId, articlePK, languageId[$COLUMN_LENGTH:75$], ctCollectionId); because we already have create unique index IX_5593D868 on JournalArticleLocalization (articlePK, languageId[$COLUMN_LENGTH:75$], ctCollectionId); but that implies not using the index when someone wants to filter only by companyId and articlePK.

/cc @achaparro

@Preston-Crary
Copy link
Author

@marianoalvarosaiz if an index is needed for companyId, articlePK - add it separately. It shouldn't be a unique index if that's the reason.

@marianoalvarosaiz
Copy link

marianoalvarosaiz commented Dec 3, 2020

@Preston-Crary the reason is that order in index matters, and removing a simple index like the following one:

create unique index IX_6B9E4A66 on DLOpenerFileEntryReference (referenceType[$COLUMN_LENGTH:75$], fileEntryId);

You're creating creating a performance regression in every method that uses: DLOpenerFileEntryReferencePersistenceImpl.findByR_F(String referenceType, long fileEntryId) because now we're performing a FULL SCAN with that finder.

This is just an example, but with this implementation we're open to generate performance regressions when defining finders that aren't going to be backed up by database indexes. And it makes sense that some of those indexes are defined as unique for API usability reasons, to avoid having to perform afterwards a get(0) from the resulted collection.

@Preston-Crary
Copy link
Author

You're creating creating a performance regression in every method that uses: DLOpenerFileEntryReferencePersistenceImpl.findByR_F(String referenceType, long fileEntryId) because now we're performing a FULL SCAN with that finder.

Here's the mysql explain for that query:

mysql> explain select dlOpenerFileEntryReferenceId from DLOpenerFileEntryReference where referenceType = 'abc' and fileEntryId = 11;
+----+-------------+----------------------------+------------+-------+---------------+-------------+---------+-------+------+----------+-------+
| id | select_type | table                      | partitions | type  | possible_keys | key         | key_len | ref   | rows | filtered | Extra |
+----+-------------+----------------------------+------------+-------+---------------+-------------+---------+-------+------+----------+-------+
|  1 | SIMPLE      | DLOpenerFileEntryReference | NULL       | const | IX_54148667   | IX_54148667 | 9       | const |    1 |   100.00 | NULL  |
+----+-------------+----------------------------+------------+-------+---------------+-------------+---------+-------+------+----------+-------

@marianoalvarosaiz
Copy link

marianoalvarosaiz commented Dec 15, 2020

@Preston-Crary Sorry for the delay, I was on vacation.

Unique index keys can be null. Try the following:

INSERT INTO DLOpenerFileEntryReference(dlOpenerFileEntryReferenceId, groupId, companyId, userId, userName, createDate, modifiedDate, referenceKey, referenceType, fileEntryId, type_)  VALUES 
	(1,1,1,1,'userName1',NULL,NULL,'1key','1type',1,1),
	(2,1,1,1,'userName1',NULL,NULL,'1key','1type',2,2),
	(3,1,1,1,'userName1',NULL,NULL,'1key','1type',NULL,3),
	(4,1,1,1,'userName1',NULL,NULL,'1key','1type',NULL,4),
	(5,1,1,1,'userName1',NULL,NULL,'1key','2type',NULL,4),
	(6,1,1,1,'userName1',NULL,NULL,'1key','3type',NULL,4);

Now execute the explain plan in mysql:

mysql> explain select dlOpenerFileEntryReferenceId from DLOpenerFileEntryReference where referenceType = '1type' and fileEntryId is null;
+----+-------------+----------------------------+------------+------+---------------+------+---------+------+------+----------+-------------+
| id | select_type | table                      | partitions | type | possible_keys | key  | key_len | ref  | rows | filtered | Extra       |
+----+-------------+----------------------------+------------+------+---------------+------+---------+------+------+----------+-------------+
|  1 | SIMPLE      | DLOpenerFileEntryReference | NULL       | ALL  | IX_54148667   | NULL | NULL    | NULL |    6 |    16.67 | Using where |
+----+-------------+----------------------------+------------+------+---------------+------+---------+------+------+----------+-------------+

I know that by API we don't allow nulls in DLOpenerFileEntryReferencePersistenceImpl.findByR_F(String referenceType, long fileEntryId) or even in the insertion, but it was just an example of what could happen:

Assume a custom development like the previous one with fileEntryId being a String. We could have the previous all access situation. We could have it also if the customer decides to use that filter in a custom sql.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants