Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Reindex Logic #16277

Closed
wezell opened this issue Mar 22, 2019 · 0 comments

Comments

Projects
None yet
4 participants
@wezell
Copy link
Contributor

commented Mar 22, 2019

A bit of an epic for a number of issues surrounding the reindexation process, including but not limited to these issues:

#7950 - Handle server failures during reindex
#15421 - remove forced table locks
#15245 - random not switching failures
#3149 - make elasticsearch use bulk and make size configurable

wezell added a commit that referenced this issue Mar 22, 2019

wezell added a commit that referenced this issue Mar 23, 2019

wezell added a commit that referenced this issue Mar 23, 2019

wezell added a commit that referenced this issue Mar 23, 2019

wezell added a commit that referenced this issue Mar 23, 2019

wezell added a commit that referenced this issue Mar 23, 2019

wezell added a commit that referenced this issue Mar 23, 2019

wezell added a commit that referenced this issue Mar 25, 2019

wezell added a commit that referenced this issue Mar 25, 2019

wezell added a commit that referenced this issue Mar 25, 2019

wezell added a commit that referenced this issue Mar 25, 2019

wezell added a commit that referenced this issue Mar 25, 2019

wezell added a commit that referenced this issue Mar 26, 2019

wezell added a commit that referenced this issue Mar 26, 2019

@wezell wezell added this to the Rex Current milestone Mar 26, 2019

@nollymar nollymar self-assigned this Mar 26, 2019

wezell added a commit that referenced this issue Mar 26, 2019

wezell added a commit that referenced this issue Mar 27, 2019

wezell added a commit that referenced this issue Mar 27, 2019

wezell added a commit that referenced this issue Mar 27, 2019

wezell added a commit that referenced this issue Mar 27, 2019

wezell added a commit that referenced this issue Mar 27, 2019

wezell added a commit that referenced this issue Mar 27, 2019

@nollymar nollymar added the Epic label Mar 28, 2019

wezell added a commit that referenced this issue Apr 11, 2019

nollymar added a commit that referenced this issue Apr 12, 2019

#16277: On this commit:
- Avoid sending reindex request when a contentlet is going to be destroyed
- Fixing failing test (HTMLPageAssetRenderedTest)

nollymar added a commit that referenced this issue Apr 15, 2019

nollymar added a commit that referenced this issue Apr 15, 2019

nollymar added a commit that referenced this issue Apr 15, 2019

nollymar added a commit that referenced this issue Apr 15, 2019

nollymar added a commit that referenced this issue Apr 16, 2019

nollymar added a commit that referenced this issue Apr 17, 2019

dsilvam added a commit that referenced this issue Apr 17, 2019

Issue 16277 update reindex thread (#16303)
* #16236 initial pull caused performance issues, which this branch resolves

* #16236 only add rollback if we are in a transaction

* #16236 fixing test

* #16236 more cleanup

* #16236 fixing ChainableCacheAdministratorImpl

* #16236 more cleanup

* #16277 initial commit and cleanup

* dotCMS/src/main/java/.

* #16277 bulks up to 500

* #16277 adding failure cause

* #16277 adding tests

* #16277 refactored

* #16277 refactored

* #16277 repointing updated

* #16277 adding testing

* #16277 added findInDb method to contentAPI

* #16277 added integration test

* #16277 more tests and cleanup

* #16277 caught loop exception

* #16277 adding better logging, sleep and switching

* #16277 logging

* #16277 reindex relationship ids in the catchall field

* #16277 reindex relationship ids in the catchall field

* #16277 fixing reindex switch in cluster

* #16277 formatting

* #16277 making reindexing happen at startup

* #16277 fixing tests

* #16277 more database closes

* #16277 more clean

* #16277 closing db connection leak

* #16277 working on final test

* #16277 fixing tests, langauge cache was being mutated

* #16277 Cleaning up tests

* #16277 adding elapsed time to reindex screen

* #16277 Improving performance using BulkProcessor for full reindex process

* #16277 Rolling back last change because does not improve perfomance

* #16277 Adding validation to avoid calling deleteReindexEntry when there is no entries to delete

* #16277 Removing unused class

* #16277 Applying codacy suggestions

* #16277 Applying codacy suggestions

* #16277 Fixing race condition that ran a switchover before the reindex process started

* #16277 Rolling back last change

* #16277 Ignoring tests temporarely

* #16277 Fixing IT

* #16277 Ignoring tests temporarely

* #16277 Rolling back INDEX_OPERATIONS_TIMEOUT_IN_MS change temporarely

* #16277 Ignoring tests temporarely

* #16277 Fixing IT

* Connection should be closed only if it is new

* Uncommenting tests

* #16277 Fixing IT

* #16277 Removing unused import

* #16277 Removing test

* #16277 Fixing IT

* #16277 Fixing IT

* #16277 Preventing race conditions on IT

* #16277 Preventing race conditions on IT

* #16277 Fixing race condition on IT

* #16277 Fixing race condition on ITs

* #16277 Remove WrapInTransaction from several methods to avoid reindex deadlocks

* #16277 Remove WrapInTransaction from several methods to avoid reindex deadlocks

* #16277 adding transaction isolation level for mysql

* #16277 removing the FOR UPDATE from mysql stored proc, and not switching over if reindex less than 20 sec old

* #16277 removing the FOR UPDATE from mysql stored proc, and not switching over if reindex less than 20 sec old

* #16277 selecting reindex by mod number of servers

* #16277 cleanup

* #16277 removing impl using interface

* #16277 fixing clause

* #16277 always updateing reindex count

* #16277 bulk reindex structure/permissions changes

* Updating commit reference for src/main/enterprise

* #16277 Updating ee jar dependency and reducing bulk size for reindex

* #16277 Adding query compatibility for mod or % according to the database used

* #16277 Validating null itemResponse for failing bulk response

* #16277 deleting reindex records in smaller batches

* #16277 Adding query compatibility for LIMIT according to the database used

* #16277 just for testing

* #16277 Now, deletes from reindex journal are executed in batch

* #16277 fixing EsmappingAPITest

* Ignoring tests temporarily

* #16277: Increasing mysql aws instance size

* #16277 Waiting more time before closing bulk processor

* #16277 don't add a possibily dirty contentlet to cache

* #16227 testing BulkProcessor rebuild

* #16277 rebuild bulk

* #16277 Reducing the await close timeout

* #16277 Rolling back thread dump logging

* #16277: On this commit:
- Avoid sending reindex request when a contentlet is going to be destroyed
- Fixing failing test (HTMLPageAssetRenderedTest)

* #16277 Fixing failing test RulesUnderPageAssetsFTest.testFireRuleUnderLivePage

* #16277 Fixing ReindexAPITest.test_failed_records

* #16277 Fixing ReindexAPITest.test_failed_records

* #16277 Removing no longer valid test: ReindexAPITest.test_old_records_get_re_queued

* #16277: Uncommenting tests in ReindexThreadTest

* #16277 Fixing misspelled word

* #16277 Fixing ReindexAPITest.test_failed_records

* #16287: Fixing race condition on ContentletAPITest.addRemoveContentFromIndex

* #16277 Avoiding race conditions

* #16277 Rolling back ee version

@dsilvam dsilvam added the WF : Merged label Apr 17, 2019

nollymar added a commit that referenced this issue Apr 23, 2019

dsilvam added a commit that referenced this issue Apr 23, 2019

Fix for mysql tests (#16436)
* Fixing tests

* Troubleshooting for tests

* Troubleshooting for tests

* Setting index policy to avoiding race conditions

* #16277 Adding pagination to batch delete to avoid deadlocks in MySQL

nollymar added a commit that referenced this issue Apr 23, 2019

nollymar added a commit that referenced this issue Apr 23, 2019

dsilvam added a commit that referenced this issue Apr 23, 2019

Fix for mysql deadlock (#16440)
* Setting index policy to avoiding race conditions

* #16277 Adding pagination to batch delete to avoid deadlocks in MySQL

* #16277 Adding pagination to batch delete to avoid deadlocks in MySQL

@dsilvam dsilvam modified the milestones: 042319_REX, Rex Current Apr 26, 2019

@wezell wezell closed this May 9, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.