Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't unnecessarily shuffle items before config validation #10008

Merged
merged 1 commit into from
Mar 7, 2024

Conversation

Al2Klimov
Copy link
Member

@Al2Klimov Al2Klimov commented Feb 19, 2024

Before ae693cb (#9577) we've repeatedly looped over all items in parallel like this:

while not types.done:
  for t in types:
    if not t.done and t.dependencies.done:
      with parallel(all_items, CONCURRENCY) as some_items:
        for i in some_items:
          if i.type is t:
            i.commit()

I.e. all items got distributed over CONCURRENCY threads, but not always equally. E.g. it was the hosts' turn, but only two threads got hosts and did all the work. The others didn't do actual work (due to the lack of hosts in their queue) which reduced the performance. c721c30 (#6581) fixed it by shuffling all_items first. ae693cb (#9577) made the latter unnecessary by replacing the above algorithm with this:

while not types.done:
  for t in types:
    if not t.done and t.dependencies.done:
      with parallel(all_items[t], CONCURRENCY) as some_items:
        for i in some_items:
          i.commit()

I.e. parallel() gets only items of type t, so all threads get e.g. hosts.

Before ae693cb (#9577) we've repeatedly looped over all items in parallel like this:

while not types.done:
  for t in types:
    if not t.done and t.dependencies.done:
      with parallel(all_items, CONCURRENCY) as some_items:
        for i in some_items:
          if i.type is t:
            i.commit()

I.e. all items got distributed over CONCURRENCY threads, but not always equally. E.g. it was the hosts' turn, but only two threads got hosts and did all the work. The others didn't do actual work (due to the lack of hosts in their queue) which reduced the performance. c721c30 (#6581) fixed it by shuffling all_items first. ae693cb (#9577) made the latter unnecessary by replacing the above algorithm with this:

while not types.done:
  for t in types:
    if not t.done and t.dependencies.done:
      with parallel(all_items[t], CONCURRENCY) as some_items:
        for i in some_items:
          if i.type is t:
            i.commit()

I.e. parallel() gets only items of type t, so all threads get e.g. hosts.
@Al2Klimov Al2Klimov added area/configuration DSL, parser, compiler, error handling core/quality Improve code, libraries, algorithms, inline docs labels Feb 19, 2024
@cla-bot cla-bot bot added the cla/signed label Feb 19, 2024
@icinga-probot icinga-probot bot added the bug Something isn't working label Feb 19, 2024
@Al2Klimov Al2Klimov added this to the 2.15.0 milestone Mar 4, 2024
@julianbrost julianbrost merged commit 097ba00 into master Mar 7, 2024
25 checks passed
@julianbrost julianbrost deleted the Al2Klimov-patch-12 branch March 7, 2024 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/configuration DSL, parser, compiler, error handling bug Something isn't working cla/signed core/quality Improve code, libraries, algorithms, inline docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants