Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GraphQL freezes when PP new content #19725

Closed
wezell opened this issue Dec 18, 2020 · 4 comments · Fixed by #19735
Closed

GraphQL freezes when PP new content #19725

wezell opened this issue Dec 18, 2020 · 4 comments · Fixed by #19735
Labels
LTS: Excluded Ticket that has been excluded from at least one LTS Merged QA : Approved QA : Passed Internal Release : 5.3.8.4 Included in LTS patch release 5.3.8.4 Release : 21.01 Type : Defect

Comments

@wezell
Copy link
Contributor

wezell commented Dec 18, 2020

When new content and content types are pushed, we invalidate the graphql schema numerous times. This causes graphql queries against the production environment to hang until the schema as been rebuilt, which can take a long time. I think we need a few things:

  1. not replace the existing schema until the new one has been built.
  2. run the schema cache flush in a commit listener so that it only gets rebuilt once
    ....

This is an example of the invalidations in the logs

[18/12/20 09:44:19:169 EST] INFO util.LogTimeUtil: Call for class: com.dotcms.graphql.business.GraphqlAPIImpl#generateSchema, duration:120023 millis
[18/12/20 09:44:19:226 EST] INFO util.LogTimeUtil: Call for class: com.dotcms.graphql.business.GraphqlAPIImpl#generateSchema, duration:120578 millis
[18/12/20 09:44:19:472 EST] INFO util.LogTimeUtil: Call for class: com.dotcms.graphql.business.GraphqlAPIImpl#generateSchema, duration:123986 millis
[18/12/20 09:44:19:620 EST] INFO util.LogTimeUtil: Call for class: com.dotcms.graphql.business.GraphqlAPIImpl#generateSchema, duration:121361 millis
[18/12/20 09:44:37:961 EST] INFO util.LogTimeUtil: Call for class: com.dotcms.graphql.business.GraphqlAPIImpl#generateSchema, duration:67960 millis
[18/12/20 09:44:42:874 EST] INFO util.LogTimeUtil: Call for class: com.dotcms.graphql.business.GraphqlAPIImpl#generateSchema, duration:72397 millis
@wezell wezell added this to the Falcon Current milestone Dec 18, 2020
@yolabingo
Copy link
Contributor

cloud support ticket https://dotcms.zendesk.com/agent/tickets/103210

wezell added a commit that referenced this issue Dec 22, 2020
@wezell
#19725 debounces graphql cache evictions and syncronizes the cache re…
e8fde8f 
…build
wezell added a commit that referenced this issue Dec 22, 2020
@wezell
#19725 more tests
cd7e3b1
wezell added a commit that referenced this issue Dec 22, 2020
@wezell
#19725 more tests
f924515
wezell added a commit that referenced this issue Dec 22, 2020
@wezell
#19725 better example
61ee967
wezell added a commit that referenced this issue Dec 22, 2020
@wezell
#19725 debounces graphql cache evictions and syncronizes the cache re…
2da8004 
…build
wezell added a commit that referenced this issue Dec 22, 2020
@wezell
#19725 more tests
6836518
wezell added a commit that referenced this issue Dec 22, 2020
@wezell
#19725 more tests
515c10d
wezell added a commit that referenced this issue Dec 22, 2020
@wezell
#19725 better example
e05c1f6
wezell added a commit that referenced this issue Dec 23, 2020
@wezell
#19725 more doc for debouncer
bf0c397
wezell added a commit that referenced this issue Dec 24, 2020
@wezell
#19725 use cached content types to build list
0eb5d1f
wezell added a commit that referenced this issue Dec 24, 2020
@wezell
#19725 fixing tests
5dfc0c0
wezell added a commit that referenced this issue Dec 24, 2020
@wezell
#19725 fixing final test
7cf308d
nollymar pushed a commit that referenced this issue Dec 29, 2020
#19725 Fixing failing tests
77c3a72
nollymar pushed a commit that referenced this issue Dec 29, 2020
@wezell
Issue 19725 graphql runaway cache evictions 20 12 (#19735)
65686d8 
* #19725 debounces graphql cache evictions and syncronizes the cache rebuild

* #19725 more tests

* #19725 more tests

* #19725 better example

* #19725 more doc for debouncer

* #19725 use cached content types to build list

* #19725 fixing tests

* #19725 fixing final test

* #19725 Fixing failing tests

Co-authored-by: Nollymar Longa <>
@nollymar nollymar modified the milestones: Falcon Current, Maintenance Sprint Dec 29, 2020
@nollymar nollymar linked a pull request Dec 29, 2020 that will close this issue
Issue 19725 graphql runaway cache evictions 20 12 #19735
Merged
@nollymar
Copy link
Contributor

Passed internal qa:

16:28:22.240  INFO  util.PushPublishLogger - class com.dotcms.enterprise.publishing.remote.handler.ContentTypeHandler : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Content Type publish success, ID: 91812c8b-0441-4139-8d4d-7423cfb0e979, Name: Destination
16:28:22.240  INFO  util.SecurityLogger - class com.dotmarketing.portlets.workflows.business.WorkflowAPIImpl : Saving schemes: [WorkflowScheme [id=d61a59e1-a49c-46f2-a929-db2b4bfa88b2, name=System Workflow, archived=false, mandatory=false]], to the content type: com.dotmarketing.portlets.structure.model.Structure@7c9e993b[name=Destination,description=Travel destinations landing pages,defaultStructure=false,reviewInterval=<null>,reviewerRole=<null>,pagedetail=<null>,structureType=5,fixed=false,system=false,velocityVarName=Destination,urlMapPattern=<null>,host=48190c8c-42c4-46af-8d1a-0cd5db894797,folder=SYSTEM_FOLDER,publishDateVar=<null>,expireDateVar=<null>,modDate=Tue Dec 29 15:54:45 EST 2020,iDate=Tue Aug 27 15:00:26 EST 2019,type=structure,owner=<null>,inode=91812c8b-0441-4139-8d4d-7423cfb0e979,identifier=91812c8b-0441-4139-8d4d-7423cfb0e979] -- ip:ukn,user:null
16:28:22.244  INFO  caffine.CaffineCache - ***	 Building Cache : workflowstepcache, size:1000,Concurrency:32
16:28:22.244  INFO  caffine.CaffineCache - ***	 Building Cache : workflowstepcache, size:1000,Concurrency:32
16:28:22.257  INFO  util.PushPublishLogger - class com.dotcms.enterprise.publishing.remote.handler.RelationshipHandler : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Relationship publish (new) success, Inode: ea9b4811-5eac-4c81-8dde-6ae25d0a590f, Name: null location
16:28:22.260  INFO  util.PushPublishLogger - class com.dotcms.enterprise.publishing.remote.handler.RelationshipHandler : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Relationship publish (new) success, Inode: 3f88bdaa-c53a-4176-9ac0-ea459df94351, Name: null activities
16:28:22.277  INFO  util.PushPublishLogger - class com.dotcms.enterprise.publishing.remote.handler.LanguageVariablesHandler : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Language File publish success, Name: cms_language_en.properties
16:28:22.277  INFO  util.PushPublishLogger - class com.dotcms.enterprise.publishing.remote.handler.LanguageVariablesHandler : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Language File publish success, Name: cms_language_en_US.properties
16:28:22.278  INFO  util.PushPublishLogger - class com.dotcms.enterprise.publishing.remote.handler.LanguageVariablesHandler : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Language File publish success, Name: cms_language_es.properties
16:28:22.278  INFO  util.PushPublishLogger - class com.dotcms.enterprise.publishing.remote.handler.LanguageVariablesHandler : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Language File publish success, Name: cms_language_es_ES.properties
16:28:22.279  INFO  util.PushPublishLogger - class com.dotcms.enterprise.publishing.remote.handler.LanguageVariablesHandler : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Language File publish success, Name: cms_language_fr.properties
16:28:22.279  INFO  util.PushPublishLogger - class com.dotcms.enterprise.publishing.remote.handler.LanguageVariablesHandler : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Language File publish success, Name: cms_language_fr_FR.properties
16:28:22.279  INFO  util.PushPublishLogger - class com.dotcms.enterprise.publishing.remote.handler.LanguageVariablesHandler : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Language File publish success, Name: cms_language_zh.properties
16:28:22.280  INFO  util.PushPublishLogger - class com.dotcms.enterprise.publishing.remote.handler.LanguageVariablesHandler : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Language File publish success, Name: cms_language_zh_CN.properties
16:28:22.305  INFO  caffine.CaffineCache - ***	 Building Cache : hostaliascache, size:1000,Concurrency:32
16:28:22.305  INFO  caffine.CaffineCache - ***	 Building Cache : hostaliascache, size:1000,Concurrency:32
16:28:22.311  INFO  util.PushPublishLogger - class com.dotcms.rest.PublishThread : [BundleID: 01ETR6DJC581F0D3TAXTFG4F8X.tar.gz] Finished bundle publish process
16:28:22.311  INFO  rest.PublishThread - Finished bundle publish process
16:28:22.340  INFO  caffine.CaffineCache - ***	 Building Cache : workflowstepcache, size:1000,Concurrency:32
16:28:22.342  INFO  caffine.CaffineCache - ***	 Building Cache : workflowstepcache, size:1000,Concurrency:32
16:28:22.422  INFO  caffine.CaffineCache - ***	 Building Cache : workflowtaskcache, size:10000,Concurrency:32
16:28:22.484  INFO  reindex.BulkProcessorListener - -----------
16:28:22.484  INFO  reindex.BulkProcessorListener - Reindexing Server #  : 1 of 1
16:28:22.484  INFO  reindex.BulkProcessorListener - Total Indexed        : 0
16:28:22.484  INFO  reindex.BulkProcessorListener - ReindexEntries found : 3
16:28:22.484  INFO  reindex.BulkProcessorListener - BulkRequests created : 6
16:28:22.484  INFO  reindex.BulkProcessorListener - -----------
16:28:25.408  INFO  caffine.CaffineCache - ***	 Building Cache : hostaliascache, size:1000,Concurrency:32
16:28:25.408  INFO  caffine.CaffineCache - ***	 Building Cache : hostaliascache, size:1000,Concurrency:32
16:28:25.411  INFO  caffine.CaffineCache - ***	 Building Cache : workflowstepcache, size:1000,Concurrency:32
16:28:25.412  INFO  caffine.CaffineCache - ***	 Building Cache : workflowstepcache, size:1000,Concurrency:32
16:28:27.410  INFO  concurrent.Debouncer - Debouncing : invalidateGraphSchema after 5000 MILLISECONDS
16:28:27.410  INFO  caffine.CaffineCache - ***	 Building Cache : graphqlschemacache, size:1000,Concurrency:32

@bryanboza
Copy link
Member

Fixed, tested after the last changes and now we are invalidating the schema just once, tested on release 21.01 // Postgres

@fmontes fmontes closed this as completed Jan 8, 2021
fmontes added a commit that referenced this issue Jan 11, 2021
@fmontes @dsilvam
@freddyDOTCMS @nollymar @wezell @fabrizzio-dotCMS @victoralfaro-dotcms @erickgonzalez @jdotcms
Merge Release 21.01 into master (#19786)
3d548cc 
* Update dotcmsReleaseVersion and coreWebReleasion version

* update release version

* #19660 Rename graphql-page-api types. Fix caching issues (#19662)

* #19660 Rename graphql-page-api types. Fix caching issues

* #19660 rename graphql-content-api types to avoid clashing

* #19660 code-review

* #19660 fix compiling issues

* #19660 fix compiling issues

* #19275 Fixing (#19698)

* #19275 Fixing

* removing console.log

* Refactoring

* Adding doc

* #19686 Returning fallback HTMLPageAsset by default if no page is found for a specific language (#19703)

* #19686 Returning fallback HTMLPageAsset by default if no page is found for a specific language

* #19686 New IT

Co-authored-by: Nollymar Longa <>

* #19709 EMA does not return the page HTML

* Updating starter to version 20201217

* #19728 null check before returning (#19733)

* #19728 null check before returning

* #19728 Implementing tests

Co-authored-by: Nollymar Longa <>

* #19636 defensive null check (#19721)

* Issue 19725 graphql runaway cache evictions 20 12 (#19735)

* #19725 debounces graphql cache evictions and syncronizes the cache rebuild

* #19725 more tests

* #19725 more tests

* #19725 better example

* #19725 more doc for debouncer

* #19725 use cached content types to build list

* #19725 fixing tests

* #19725 fixing final test

* #19725 Fixing failing tests

Co-authored-by: Nollymar Longa <>

* bump version

* Issue 19608 contentlets lose inherited permissions until cache is flushed (#19745)

* #19608 only cache found permissions 5.3.8

* #19608 adding tests

Co-authored-by: Will Ezell <will@dotcms.com>
Co-authored-by: fabrizzio-dotCMS <fabrizzio@dotCMS.com>

* Update coreWebReleaseVersion

* #19744 Unable to use the filter in the content types portlet

* Introducing new get-commit-message action (#19760)

* update starter v20210106 (#19767)

* fixing test

* Revert "fixing test"

This reverts commit 5d15416.

* #19275 Fixing error when remote host is unknow (#19766)

* #19275 Fixing error when remote host is unknow

* Fixing error

* fixing test

* upgrade the saml referecne for 21.01 (#19780)

Co-authored-by: Nollymar Longa <>
Co-authored-by: Daniel Silva <daniel.silva@dotcms.com>
Co-authored-by: Freddy Rodriguez <freddy0309@gmail.com>
Co-authored-by: Nollymar Longa <nollymar.longa@dotcms.com>
Co-authored-by: Will Ezell <will@dotcms.com>
Co-authored-by: fabrizzio-dotCMS <fabrizzio@dotCMS.com>
Co-authored-by: Victor Alfaro <victor.alfaro@dotcms.com>
Co-authored-by: erickgonzalez <erick.gonzalez@dotcms.com>
Co-authored-by: Jonathan <jonathan.sanchez@dotcms.com>
@jcastro-dotcms jcastro-dotcms added the LTS : Next Ticket that will be added to LTS label Jan 20, 2021
jcastro-dotcms added a commit that referenced this issue Feb 3, 2021
@jcastro-dotcms
Adding code changes for #19725 to LTS release 5.3.8.4
eba1956
@jcastro-dotcms jcastro-dotcms added LTS: Excluded Ticket that has been excluded from at least one LTS LTS: Released labels Feb 3, 2021
@jcastro-dotcms
Copy link
Contributor

The official code changes have been included in the dotCMS 5.3.8.4 LTS release only. The 5.2.8.4 LTS release is several commits behind, which made the code back-port not possible.

@john-thomas-dotcms john-thomas-dotcms added Release : 5.3.8.4 Included in LTS patch release 5.3.8.4 and removed LTS : Next Ticket that will be added to LTS LTS: Released labels Feb 10, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
LTS: Excluded Ticket that has been excluded from at least one LTS Merged QA : Approved QA : Passed Internal Release : 5.3.8.4 Included in LTS patch release 5.3.8.4 Release : 21.01 Type : Defect
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Issue 19725 graphql runaway cache evictions 20 12 dotCMS/core
8 participants
@yolabingo @fmontes @wezell @bryanboza @erickgonzalez @nollymar @jcastro-dotcms @john-thomas-dotcms