Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize Thread Usage while Loading Resources #1923

Merged
merged 1 commit into from
Jul 23, 2024

Conversation

alexanderkiel
Copy link
Member

Before this change, the KV Resource Store used it's thread pool for both loading and parsing/conforming the resources. However loading is I/O but parsing/conforming is CPU bound. So it is better to use the KV Resource Store Executor only for the loading, the I/O part.

With this change the KV Resource Store uses do-async to change to a common ForkJoinPool thread. This change has also the advantage that functions that are applied after the returned future do not have to change the thread pool because the futures are already on the common ForkJoinPool.

Removed all pool switching async functions. Tested the thread pool used.

@alexanderkiel alexanderkiel added this to the v0.29.0 milestone Jul 23, 2024
@alexanderkiel alexanderkiel self-assigned this Jul 23, 2024
Copy link

codecov bot commented Jul 23, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.81%. Comparing base (e3eb6c7) to head (50f2289).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1923      +/-   ##
==========================================
+ Coverage   94.77%   94.81%   +0.03%     
==========================================
  Files         319      318       -1     
  Lines       19359    19347      -12     
  Branches      478      474       -4     
==========================================
- Hits        18348    18344       -4     
+ Misses        533      529       -4     
+ Partials      478      474       -4     
Files Coverage Δ
.../db-resource-store/src/blaze/db/resource_store.clj 100.00% <ø> (ø)
...-resource-store/src/blaze/db/resource_store/kv.clj 100.00% <100.00%> (ø)
modules/db/src/blaze/db/api.clj 95.20% <100.00%> (ø)
modules/db/src/blaze/db/impl/db.clj 97.67% <100.00%> (ø)
modules/db/src/blaze/db/node.clj 94.53% <100.00%> (ø)
modules/db/src/blaze/db/node/resource_indexer.clj 96.42% <100.00%> (-0.04%) ⬇️
...dules/interaction/src/blaze/interaction/delete.clj 100.00% <100.00%> (ø)
modules/interaction/src/blaze/interaction/read.clj 100.00% <100.00%> (ø)
.../interaction/src/blaze/interaction/transaction.clj 98.76% <100.00%> (ø)
...nc-interaction/src/blaze/job/async_interaction.clj 97.80% <100.00%> (ø)
... and 8 more

... and 2 files with indirect coverage changes

@alexanderkiel alexanderkiel force-pushed the resource-store-threads branch 2 times, most recently from d6f989e to f3bb39f Compare July 23, 2024 16:28
Before this change, the KV Resource Store used it's thread pool for both
loading and parsing/conforming the resources. However loading is I/O but
parsing/conforming is CPU bound. So it is better to use the KV Resource
Store Executor only for the loading, the I/O part.

With this change the KV Resource Store uses do-async to change to a
common ForkJoinPool thread. This change has also the advantage that
functions that are applied after the returned future do not have to
change the thread pool because the futures are already on the common
ForkJoinPool.

The performance improves by a little bit because the threads are now
directly usable to lead the next resource instead having to parse/
conform the current one.

Removed all pool switching async functions. Tested the thread pool used.
@alexanderkiel alexanderkiel added this pull request to the merge queue Jul 23, 2024
Merged via the queue into main with commit 123d0a9 Jul 23, 2024
126 checks passed
@alexanderkiel alexanderkiel deleted the resource-store-threads branch July 23, 2024 19:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant