Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IdentifierFactoryImpl.findByURI() is getting a full table scan instead of index scan #7918

Closed
chrismccracken opened this issue Jul 2, 2015 · 6 comments

Comments

@chrismccracken
Copy link
Contributor

At https://github.com/dotCMS/core/blob/master-3.2/src/com/dotmarketing/business/IdentifierFactoryImpl.java#L173, the lower() SQL function is causing the sql index to be unused for this query, which is very expensive on large identifier tables.

On postgres, an expressional index (http://www.postgresql.org/docs/9.3/static/indexes-expressional.html) such as this fixes it:
create index identifier_lcase_parent_path_asset_name_host_inode on identifier ( lower(parent_path), lower(asset_name), host_inode );

0.233 ms to 0.085 ms speedup, on demo. https://gist.github.com/chrismccracken/a563d419464838a41d82

@jtesser jtesser added this to the 3.2.4 milestone Jul 17, 2015
@maurizo
Copy link
Contributor

maurizo commented Jul 17, 2015

This approach will only work on Postgresql and Oracle. Neither MySQL, MSSQL or H2 allows expressions (for the LOWER function) as part of the index definition.

@brentgriffin
Copy link
Contributor

Seems that we may need to add a column that is auto set to be the lower(of parent_path + asset_name), create an index on this column, and then change the query to use the newly created column.

@jgambarios
Copy link
Contributor

@jtesser thoughts?

@jtesser
Copy link
Contributor

jtesser commented Jul 21, 2015

Yes I think we need to hold until 3.3 on this one

@jtesser jtesser modified the milestones: 3.3, 3.2.4 Jul 21, 2015
@jtesser jtesser closed this as completed Dec 3, 2015
@jtesser jtesser removed this from the 3.3 milestone Dec 3, 2015
@wezell wezell reopened this Mar 2, 2017
@jgambarios jgambarios added this to the Cody Next milestone Apr 10, 2017
@jcastro-dotcms jcastro-dotcms self-assigned this Apr 19, 2017
jcastro-dotcms added a commit that referenced this issue Apr 24, 2017
…entifier URLs are stored in lower-case. Adding Upgrade Task to lower-case Identifier URLs. Making sure that the Starter Importer is creating URLs in lower-case. Adding Javadoc.
@jcastro-dotcms
Copy link
Contributor

PR: #11385

jcastro-dotcms added a commit that referenced this issue Apr 24, 2017
jcastro-dotcms added a commit that referenced this issue Apr 24, 2017
jgambarios pushed a commit that referenced this issue Apr 24, 2017
#11385)

* #7918 : Removing LOWER function from SQL queries. Making sure that Identifier URLs are stored in lower-case. Adding Upgrade Task to lower-case Identifier URLs. Making sure that the Starter Importer is creating URLs in lower-case. Adding Javadoc.

* #7918 : Code review.

* #7918 : Solving merge conflict.
@bryanboza
Copy link
Member

Fixed, tested on master // postgres // FF

@wezell wezell closed this as completed May 1, 2017
agomez-dotcms added a commit that referenced this issue May 18, 2017
oarrietadotcms pushed a commit that referenced this issue May 18, 2017
#11623)

* #7918: Fixed functional-tests for RulesUnderPageAssetsFTest (which are
failing for Oracle and Postgres)

* #11196: Disabled failing functional-tests by request
agomez-dotcms added a commit that referenced this issue May 18, 2017
#11623)

* #7918: Fixed functional-tests for RulesUnderPageAssetsFTest (which are
failing for Oracle and Postgres)

* #11196: Disabled failing functional-tests by request
oarrietadotcms pushed a commit that referenced this issue May 18, 2017
#11623) (#11630)

* #7918: Fixed functional-tests for RulesUnderPageAssetsFTest (which are
failing for Oracle and Postgres)

* #11196: Disabled failing functional-tests by request
dsilvam pushed a commit that referenced this issue Jun 5, 2017
* Changed cache provider to CaffineCache (#11490)

(cherry picked from commit 0956e55)

* Updated to use the latest Hazelcast changes (#11491)

(cherry picked from commit d604680)

* Updated to use the latest Hazelcast changes

(cherry picked from commit 46c64a0)

* Issue 11124 fixes/enhances Hazelcast implementation (#11414) (#11493)

* #11124: Made manual-cluster config wire with hazel

* #11124: Fixed hazelcast util shutdown

* #11124: Not to send cache-invalidation-messages when a cache-chain is
totally distributed for a particular region

* #11414: Incorporated feedback after PR (blindly)

* #11414: Incorporated feedback after PR (blindly)

* #11414: Incorporated feedback after PR (blindly)

* #11414: Incorporated feedback after PR (blindly)

* #11124: Introduced requested performance optimization for not spreading
cache-invalidations repeatedly in distributed caches over cluster-nodes

* #11124: Introduced design improvement hazelcastutil (singleton, blindly)

* #11124 / #11468: Removed CLUSTER_AUTOWIRE property

* #11124/#11463: Fixed number of nodes on Hazelcast cluster

* #11124 / #11468: Removed CLUSTER_AUTOWIRE property

* #11124 / #11464: Cleaned-up cache-stats for Hazelcast client

* #11432 : Adding correct filtering criteria to avoid re-processing bundles that were sent correctly and without problems. Adding more logging to the process to better track down bundle status. Adding Javadoc. (#11498)

(cherry picked from commit 4d4d6c1)

* Update dependencies

* Issue 10711 make use of contentType.fields() instead of FieldsCache (#11503) (#11520)

* #10711: Use contentType.fields() insteadof FieldsCache

* #10711: Remove usage of legacy-field-transformer

* #11451 Send total number of notifications in getNotifications end point (#11523)

(cherry picked from commit 957bdc0)

* Issue 11516 non macros getting eaten (#11522)

* #11457 fixed

* #11516 fixing macros

(cherry picked from commit 2c72def)

* Reverting changes #11516

* Issue 11516 non macros getting eaten (#11522)

* #11457 fixed

* #11516 fixing macros

* #118=481: Uses Mockito to set properties accordinly. (#11532)

* Issue 11529 edit mode edit container (#11530)

* #11457 fixed

* #11529 reverted to the way it was

(cherry picked from commit 1c2477a)

* Issue 11524 custom field showing pencil on content edit (#11526)

* #11457 fixed

* #11524 will only show if running throguh the cmsfielter

(cherry picked from commit b2e1a9e)

* #11296 Undefined message when you use the viewing as feature (#11538)

(cherry picked from commit cb9db1f)

* Issues 11497 10173 merge into release 41 (#11539)

* Issue 11497 include field variablest mer (#11499)

* Changed cache provider to CaffineCache (#11490)

(cherry picked from commit 0956e55)

* #11497 sending field variables

* #11497 sending field variables

* #11497 reverting properties

* #11497 now returning ResponseEntityView

* #10173: Added REST API to let update of content-types (#11528)

* #10173: Added rest-api to let update of content-types

* #11497: Fixed field/field-variable resources tests

* #11497: Reverted fix for field resource tests

* #11533: Fixed Content Type resource and added its tests (#11537) (#11541)

* #11533: Fixed Content Type resource and its tests

* Issue 11481 r41 osgi util change (#11544)

* #11481: Yet another change on the logic of the OSGIUtil to fit the UnitTest

* #11481: UnitTest added to OSGUtil for base directory

* #10325: Added unit-test-for-immutable-type-adapter (#11548) (#11549)

* #10711: Prefer field.variables() instead of reloading (#11557) (#11561)

* #11071 : Adding new date format that includes the seconds parameter. Removing unnecessary comment lines. Adding/fixing Javadoc. (#11564)

(cherry picked from commit 52cd973)

* Issue 11196 tries to fix issue with Existing Database on AWS by adding branch into name (#11584)

* #11196: Setup database with branch name (#11578)

* #11196: Setup database with branch name (2)

* #11542 fixing velocity template issue (#11573)

* #11542 fixing velocity template issue

* #11542 applying revision changes

(cherry picked from commit 659d6d0)

* 4.1 language props updates (#11551)

* Add Enterprise Edition link to login screen

* Add Enterprise Edition link to login screen

* Add Enterprise Edition link to login screen

* Add Enterprise Edition link to login screen

* Add Enterprise Edition link to login screen

* Add Enterprise Edition link to login screen

* Add Enterprise Edition link to login screen

* Add Enterprise Edition link to login screen

* 4.1.0 Language props changes

Added all strings which exist in English props but not non-English props.  Note that some of these strings may have been missed in the props files from 4.0.0 changes.

* 4.1.0 Language props changes

Added all strings which exist in English props but not non-English props.  Note that some of these strings may have been missed in the props files from 4.0.0 changes.

* 4.1.0 Language props changes

Added all strings which exist in English props but not non-English props.  Note that some of these strings may have been missed in the props files from 4.0.0 changes.

* 4.1.0 Language props changes

Added all strings which exist in English props but not non-English props.  Note that some of these strings may have been missed in the props files from 4.0.0 changes.

* 4.1.0 Language props changes

Added all strings which exist in English props but not non-English props.  Note that some of these strings may have been missed in the props files from 4.0.0 changes.

* 4.1.0 Language props changes

Added all strings which exist in English props but not non-English props.  Note that some of these strings may have been missed in the props files from 4.0.0 changes.

* 4.1.0 Language props changes

Added all strings which exist in English props but not non-English props.  Note that some of these strings may have been missed in the props files from 4.0.0 changes.

* 4.1.0 Language props changes

Added all strings which exist in English props but not non-English props.  Note that some of these strings may have been missed in the props files from 4.0.0 changes.

* 4.1.0 Language props changes

Added all strings which exist in English props but not non-English props.  Note that some of these strings may have been missed in the props files from 4.0.0 changes.

(cherry picked from commit cebc058)

* #11196: Small change to trigger nightly build

* #11196: Disabled gradle-jvmargs in favor of gradle-opts (#11589) (#11590)

* #11196: Trying to fix functional-tests  in Jenkins (#11591) (#11593)

* #11196: Trying to fix functional-tests in Jenkins (2)

* #11563 Unable to disable the never checkbox when you use date time fields to expire content (#11595)

(cherry picked from commit 096594e)

* #11196: Trying to fix functional-tests for MSSQL and Oracle (2)

* #11196: Trying to fix functional-tests for MSSQL and Oracle (3)

* #11196: Trying to fix functional-tests for MSSQL and Oracle (4)

* Issue 11596 reindex (#11602) (#11603)

* #11596: Fallback to dbById() search

* #11596: Return SYSTEM dataType if the value is not found

* #11596: Commenting out the check on Field.java for now

* #11348 Fixes to create a Schedule Fix Task to do the ping pong by default for each session (#11598)

(cherry picked from commit 74db8db)

* #11588: Disable and Enable trigger on identifier when running the upgrade task (#11605)

* Issue 11567 filter structures by base type (#11570)

* #11457 fixed

* fixes #11567

(cherry picked from commit 95a8280)

* #11592 changes default h2 db settings for more concurrency (#11611)

* Issue 11588 other dbs (#11616) (#11617)

* #11588: fix Oracle invalid character

* #11588: fix MSSQL isolation before disabling trigger

* #11588: using Collections.singletonList(T element)

* #11071 : Adding more date/time formats that include the seconds parameter. (#11612)

(cherry picked from commit 56146bd)

* #11620

* Issue 11622 fix plugins compiling windows (#11624)

* #11622 fix compiling issues for Windows

* #11622 add maxparentlevels attrib to ant task

(cherry picked from commit 4883c08)

* #10233 Clean up stringbuf on each iteration and avoid writting wrong lines to the xml file (#11618)

(cherry picked from commit 81b40b5)

* Issue 11196 fixes functional-tests failing in Jenkins after re-enabled (#11623) (#11630)

* #7918: Fixed functional-tests for RulesUnderPageAssetsFTest (which are
failing for Oracle and Postgres)

* #11196: Disabled failing functional-tests by request

* #11633 for 4.1 (#11635)

* #11563 Make sure it works when the date field its required (#11636)

(cherry picked from commit f2c97f5)

* #11196: Touching file to generate a new commit for nightly build to be
generated

* #11571 Large amount of White Space added on Events (#11643)

(cherry picked from commit a7918a1)

* #8152 Added fixes for new ping pong approach (#11649)

(cherry picked from commit 780f8da)

* #11646 order the cache stats list (#11647)

* Issue 11644 content type field cache reindex (#11645)

* #11644 forces read from cache

* #11644 do not need this class

* #11648 language fallback now works better (#11651)

* #11566: we need to add the preview and user attribute to the session/… (#11641) (#11657)

* #11566: we need to add the preview and user attribute to the session/request

* #11566: from will: it should give the mock request a session and attributes

* #9429 PR for release-4.1 (#11560)

* 11650 simple fix to avoid extra hits on the db (#11658)

(cherry picked from commit 65ce680)

* #11196: Touching file to generate a new commit for nightly build to be
generated (2)

* #11196: Touching file to generate a new commit for nightly build to be
generated (3)

* Issue 11652 saving text areas (#11667)

* #11648 language fallback now works better

* #11652 removing field instanciation checks, moving data integrity to factory

* #11652 catching an NPE when user does not select a datatype

* #11652 deprecated old fieldAPI

* #11652

* #11196: Re-enabled gradle-jvmargs in favor local builds

* #11196: Re-enabled gradle-jvmargs in favor local builds

* Issue 11668 non defualt datatypes (#11671)

* #11648 language fallback now works better

* #11668 fixes field saving, file cleanup

* #11668 only delete db if field is in db

* #11668 prevent read-only or fixed fields from being deleted

* #11668 retain fixed/read only properties when editing a field

* #11668 handle null dbType

* #11668 handle null values

* #11668 personas should be a system object

* #11668 starter loading

* #11668 fixing the one failing test

* #11664: Log failed upgrade records and filter urlmaps (#11670) (#11676)

* Issue 11674 unit tests (#11675)

* #11457 fixed

* #11674 fixed unit tests

(cherry picked from commit 313ae03)

* #11542 solving dotparse issues (#11677)

* #11542 solving dotparse issues

* #11542

(cherry picked from commit 373733a)

* Issue 11668 datatypes again (#11678)

* #11648 language fallback now works better

* #11668 commiting with tests

* #11668 commiting with tests

* #11196/#11673: Setup AWS instance with dependencies-cache populated

* #11682 Random only positive numbers,accepted dataType of TagField.java is SYSTEM (#11683)

(cherry picked from commit 2dea3d4)

* Issue 11684 starter now loading (#11685)

* added unit test #11680

* fixing unit tests

* #11684 starter loading

* #11196/#11680: Fixed field-resource-tests failing (moved changes into (#11688)

their own branch)

(cherry picked from commit 6726239)

* Issue 11689 save non default datatypes from legacy ui (#11690)

* #11689 fixed

* #11689 added test cases

* #11689 made test cases more obvious

* #11679 fixes problem creating pages (#11693)

(cherry picked from commit ce58659)

* #11135 Avoid sending list items as query strings (#11669) (#11694)

* #11681 radio button inputs disabled when the user tries to edit an existing field (#11700)

(cherry picked from commit 942c42e)

* Issue 11696 snapshot (#11699)

* #11696 fixing snapshot deploy

* #11696 Removed git hash from the dotCMS core version

(cherry picked from commit d38da5e)

* Issue 11695 unable edit system legacy field (#11697)

* #11695 fixes data as it saves it

* #11695 fixing data now has unit test

* Issue 11638 time machine (#11692) (#11701)

* #11638: Using _show_working_ for Time Machine

* #11638: Comments on why use _show_working_

* #11542 (#11706)

(cherry picked from commit 0b4ff63)

* #11196/#11197: Adjusted Jenkins/AWS scripts for continuous build

* #11196/#11673: Adding date-stamp on build script for better profiling

* #11196/#11197: Added Jenkins/AWS scripts for continuous build

* Include missing mocks (#11714)

* Revert "Include missing mocks (#11714)" (#11715)

This reverts commit 9fc1ba9.

* #11717: By default we dont throw the required field Exception (#11718) (#11719)

* #11196/#11197: Setup AWS instance with dependencies-cache populated for
continuous builds

* Merge 11725 into 4.1 (#11728)

* #11725 Validate wrong varNamea and return proper response.

* #11725 Return NotFoundInDbException when field by var not found.

* Issue #11720 fixes NPE on empty required-fields provided to (#11731)

content-type-rest-api (#11729)

* #11720: Fixed NPE on empty required-fields provided to
content-type-rest-api

* #11720: Fixed NPE on empty required-fields provided to
content-type-rest-api (2)

# Conflicts:
#	dotCMS/src/main/java/com/dotcms/rest/api/v1/contenttype/ContentTypeResource.java

* #11704 added a condition to show a delete button (#11724)

* #11704 added a condition to show a delete button

* #11704 change request

(cherry picked from commit 6f584f2)

* #11722: Fixed for invalid HTTP response when creating content-type with (#11740) (#11742)

existing variable

* Issue 11723 fix param (#11739)

* #11695 fixes data as it saves it

* #11695 fixing data now has unit test

* #11723 set host id instead of param

* #11542 Fixing ParseErrorException (#11741)

(cherry picked from commit e067560)

* #11727: Fixed NPE on empty required-fields provided to (#11743) (#11744)

field-resource rest-api

* #11730: Prevent empty field-variable key/value in (#11745) (#11749)

field-variable-resource

* #11713 fixes pp error updating content types (#11736)

* #11713 fixes pp error updating content types

* #11713 improvements

* #11713 improvements

(cherry picked from commit b115de8)

* Updating commit reference for src/main/enterprise

* #10370: Fix the UI - Undeploy - option not showing when deploy folder is not named "load". (#11761)

* Issue 11713 problem deleting fields (#11763)

* #11713 fixing delete field error

* #11713 adding new integration test with the two new save methods

* Issue 11735 fix content type resource test (#11738)

* #11695 fixes data as it saves it

* #11695 fixing data now has unit test

* #11735 fixing ContentTypeResourceTest

* #11735 Fix resource resolution.

* Ignore failing test. (#11767) (#11768)

* #11735 Missing code for setting host. (#11771) (#11772)

* #11735 fixes www.host.com resolution (#11774)

* #11735 fixes www.host.com resolution

* #11735 permissions added back

* Updated the core web release version (#11778)

* #11713: Fix merge conflicts (#11788)

* #11776 updating content type mod_date after delete field and fieldvar… (#11781)

* #11776 updating content type mod_date after delete field and fieldvariables

* #11776 includig update contenttype mode date on delete field variable

* #11776 changing var name

* #11776 changing var name

* #11794 return a fresh object after saving (#11795)

* #11757: Fixed integrity-checker issue with empty results due to runs (#11803) (#11805)

over another endpoints (2)

* #11793 for 4.1 (#11807)

* Updating commit reference for src/main/enterprise

* Modify ee and lic to download release jars

* #11756: Merge release-4.1 into master
jgambarios added a commit that referenced this issue Apr 11, 2019
* #16115 reverting changes introduced in #7918

* #16115 save point

* #16115 merge with master

* #16115 refactoring to utillize a generated column to calc asset unicity

* #16115 code clean-up + humor codacy-bot

* #16115 using  custom host to test folder tree + cache invalidation

* #16115 improve test to skip adding a column if already exists

* #16115 h2.sql

* #16115 corrections per given feedback

* #16115 corrections per given feedback

* #16115  remove unnecessary toLower function call

* #16115  fixing test that fails because of a uppercased table named on mySQL
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

9 participants