Use a dedicated thread pool for searchable snapshot cache prewarming #59313

tlrx · 2020-07-09T15:38:48Z

Since #58728 writing operations on searchable snapshot directory cache files are executed in an asynchronous manner using a dedicated thread pool. The thread pool used is searchable_snapshots which has been created to execute prewarming tasks.

Reusing the same thread pool wasn't a good idea as it can lead to deadlock situations. One of these situation arose in a test failure where the thread pool was full of prewarming tasks, all waiting for a cache file to be accessible, while the cache file was being evicted by the cache service. But such an eviction can only be processed when all read/write operations on the cache file are completed and in this case the deadlock occurred because the cache file was actively being read by a concurrent search which also won the privilege to write the range of bytes in cache... and this writing operation could never have been completed
because of the prewarming tasks making no progress and filling up the thread pool.

This pull request renames the searchable_snapshots thread pool to searchable_snapshots_cache_fetch_async. Assertions are added to assert that cache writes are executed using this thread pool and to assert that read on cached index inputs are executed using a different thread pool to avoid potential deadlock situations.

This pull request also adds a searchable_snapshots_cache_prewarming that is used to execute prewarming tasks. It also converts the existing cache prewarming test into a more complte integration test that creates multiple searchable snapshot indices concurrently with randomized thread pool sizes, and verifies that all files have been correctly prewarmed.

After this pull request other improvements could be done:

limit the number of concurrent shards that are actively prewarming
make the prewarming of each file part fully asynchronous

elasticmachine · 2020-07-09T15:39:08Z

Pinging @elastic/es-distributed (:Distributed/Snapshot/Restore)

DaveCTurner

LGTM

tlrx · 2020-07-15T07:58:13Z

Thanks David!

…lastic#59313) Since elastic#58728 writing operations on searchable snapshot directory cache files are executed in an asynchronous manner using a dedicated thread pool. The thread pool used is searchable_snapshots which has been created to execute prewarming tasks. Reusing the same thread pool wasn't a good idea as it can lead to deadlock situations. One of these situation arose in a test failure where the thread pool was full of prewarming tasks, all waiting for a cache file to be accessible, while the cache file was being evicted by the cache service. But such an eviction can only be processed when all read/write operations on the cache file are completed and in this case the deadlock occurred because the cache file was actively being read by a concurrent search which also won the privilege to write the range of bytes in cache... and this writing operation could never have been completed because of the prewarming tasks making no progress and filling up the thread pool. This commit renames the searchable_snapshots thread pool to searchable_snapshots_cache_fetch_async. Assertions are added to assert that cache writes are executed using this thread pool and to assert that read on cached index inputs are executed using a different thread pool to avoid potential deadlock situations. This commit also adds a searchable_snapshots_cache_prewarming that is used to execute prewarming tasks. It also converts the existing cache prewarming test into a more complte integration test that creates multiple searchable snapshot indices concurrently with randomized thread pool sizes, and verifies that all files have been correctly prewarmed.

…59313) (#59590) Since #58728 writing operations on searchable snapshot directory cache files are executed in an asynchronous manner using a dedicated thread pool. The thread pool used is searchable_snapshots which has been created to execute prewarming tasks. Reusing the same thread pool wasn't a good idea as it can lead to deadlock situations. One of these situation arose in a test failure where the thread pool was full of prewarming tasks, all waiting for a cache file to be accessible, while the cache file was being evicted by the cache service. But such an eviction can only be processed when all read/write operations on the cache file are completed and in this case the deadlock occurred because the cache file was actively being read by a concurrent search which also won the privilege to write the range of bytes in cache... and this writing operation could never have been completed because of the prewarming tasks making no progress and filling up the thread pool. This commit renames the searchable_snapshots thread pool to searchable_snapshots_cache_fetch_async. Assertions are added to assert that cache writes are executed using this thread pool and to assert that read on cached index inputs are executed using a different thread pool to avoid potential deadlock situations. This commit also adds a searchable_snapshots_cache_prewarming that is used to execute prewarming tasks. It also converts the existing cache prewarming test into a more complte integration test that creates multiple searchable snapshot indices concurrently with randomized thread pool sizes, and verifies that all files have been correctly prewarmed.

…59313) (#59595) Since #58728 writing operations on searchable snapshot directory cache files are executed in an asynchronous manner using a dedicated thread pool. The thread pool used is searchable_snapshots which has been created to execute prewarming tasks. Reusing the same thread pool wasn't a good idea as it can lead to deadlock situations. One of these situation arose in a test failure where the thread pool was full of prewarming tasks, all waiting for a cache file to be accessible, while the cache file was being evicted by the cache service. But such an eviction can only be processed when all read/write operations on the cache file are completed and in this case the deadlock occurred because the cache file was actively being read by a concurrent search which also won the privilege to write the range of bytes in cache... and this writing operation could never have been completed because of the prewarming tasks making no progress and filling up the thread pool. This commit renames the searchable_snapshots thread pool to searchable_snapshots_cache_fetch_async. Assertions are added to assert that cache writes are executed using this thread pool and to assert that read on cached index inputs are executed using a different thread pool to avoid potential deadlock situations. This commit also adds a searchable_snapshots_cache_prewarming that is used to execute prewarming tasks. It also converts the existing cache prewarming test into a more complte integration test that creates multiple searchable snapshot indices concurrently with randomized thread pool sizes, and verifies that all files have been correctly prewarmed.

…lastic#59313) Since elastic#58728 writing operations on searchable snapshot directory cache files are executed in an asynchronous manner using a dedicated thread pool. The thread pool used is searchable_snapshots which has been created to execute prewarming tasks. Reusing the same thread pool wasn't a good idea as it can lead to deadlock situations. One of these situation arose in a test failure where the thread pool was full of prewarming tasks, all waiting for a cache file to be accessible, while the cache file was being evicted by the cache service. But such an eviction can only be processed when all read/write operations on the cache file are completed and in this case the deadlock occurred because the cache file was actively being read by a concurrent search which also won the privilege to write the range of bytes in cache... and this writing operation could never have been completed because of the prewarming tasks making no progress and filling up the thread pool. This commit renames the searchable_snapshots thread pool to searchable_snapshots_cache_fetch_async. Assertions are added to assert that cache writes are executed using this thread pool and to assert that read on cached index inputs are executed using a different thread pool to avoid potential deadlock situations. This commit also adds a searchable_snapshots_cache_prewarming that is used to execute prewarming tasks. It also converts the existing cache prewarming test into a more complte integration test that creates multiple searchable snapshot indices concurrently with randomized thread pool sizes, and verifies that all files have been correctly prewarmed.

The number of prewarming workers to spawn was taken from the wrong thread pool info, fixed to size number of workers after the prewarming thread pool. Relates elastic#59313

The number of prewarming workers to spawn was taken from the wrong thread pool info, fixed to size number of workers after the prewarming thread pool. Relates #59313

Use dedicated thread pool for cache prewarming

7489500

tlrx added :Distributed/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v7.8.2 v7.9.0 v8.0.0 labels Jul 9, 2020

elasticmachine added the Team:Distributed Meta label for distributed team label Jul 9, 2020

tlrx added >enhancement and removed Team:Distributed Meta label for distributed team labels Jul 9, 2020

tlrx requested review from DaveCTurner and ywelsch July 9, 2020 16:31

tlrx added 2 commits July 10, 2020 10:08

Revert changes in SearchableSnapshotsIntegTests

dc06fbd

Merge branch 'master' into use-dedicated-thread-pool-for-prewarming

766982e

DaveCTurner approved these changes Jul 14, 2020

View reviewed changes

tlrx merged commit 3ab0445 into elastic:master Jul 15, 2020

tlrx deleted the use-dedicated-thread-pool-for-prewarming branch July 15, 2020 07:58

tlrx mentioned this pull request Jul 15, 2020

Use a dedicated thread pool for searchable snapshot cache prewarming (#59313) #59590

Merged

This was referenced Jul 15, 2020

Use a dedicated thread pool for searchable snapshot cache prewarming (#59313) #59595

Merged

Test failure: org.elasticsearch.xpack.searchablesnapshots.rest.FsSearchableSnapshotsIT #58961

Closed

henningandersen mentioned this pull request Oct 20, 2020

Searchable snapshot number of prewarmers #63919

Merged

jakelandis removed the v8.0.0 label Jul 26, 2021

jakelandis added the v8.0.0-alpha1 label Jul 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use a dedicated thread pool for searchable snapshot cache prewarming #59313

Use a dedicated thread pool for searchable snapshot cache prewarming #59313

tlrx commented Jul 9, 2020

elasticmachine commented Jul 9, 2020

DaveCTurner left a comment

tlrx commented Jul 15, 2020

Use a dedicated thread pool for searchable snapshot cache prewarming #59313

Use a dedicated thread pool for searchable snapshot cache prewarming #59313

Conversation

tlrx commented Jul 9, 2020

elasticmachine commented Jul 9, 2020

DaveCTurner left a comment

Choose a reason for hiding this comment

tlrx commented Jul 15, 2020