Skip to content

SOLR-15760: Improve the default distributed facet overrequest function/heuristic#385

Closed
magibney wants to merge 1 commit intoapache:mainfrom
magibney:SOLR-15760
Closed

SOLR-15760: Improve the default distributed facet overrequest function/heuristic#385
magibney wants to merge 1 commit intoapache:mainfrom
magibney:SOLR-15760

Conversation

@magibney
Copy link
Contributor

@magibney magibney commented Nov 2, 2021

See: SOLR-15760

For logical consistency, distributed facet overrequest should make no distinction between offset and limit; instead, distributed overrequest should be calculated as a function of the sum of offset+limit, boosting relatively heavily when few values are requested, and decaying asymptotically to f(x)=x for larger numbers of requested values.

…n/heuristic

For logical consistency, distributed facet overrequest should
make no distinction between offset and limit; instead, distributed
overrequest should be calculated as a function of the sum of
offset+limit, boosting relatively heavily when few values are
requested, and decaying asymptotically to `f(x)=x` for larger
numbers of requested values.
@magibney
Copy link
Contributor Author

magibney commented Nov 5, 2021

The condition on small offset was added in 1f77776, and I think in a pinch this could stand to simply be removed (to restore the initial unconditional linear overrequest boost according to the overrequest function f(x)=1.1*x + 4, initially introduced* in 7b5df8a).

*EDIT: I misidentified the initial introduction of the f(x)=1.1*x + 4; it's been refactored around a few times, but was already present in the commit that introduced JSON Facets: 3dc5ed3

@dsmiley, could you take a quick look at this? I think there's some sense to the assertion that larger numbers of requested values actually need less overrequest (though I still don't think the distinction between offset and limit is relevant). Intuitively I'd also think that perhaps we'd want to boost extremely low gross-limit requests by more than 4 (the proposal currently in this PR has a floor of 12 for shard requests).

Ultimately I think there a three options that might make sense:

  1. revert the overrequest function change from 1f77776, going back to the original, unconditional linear boost
  2. change to a function something like what's proposed in this PR, which decays for larger gross limits
  3. switch to a static additive boost (+12 or something) that would help small requests, and "decay" (relative to the overall requested size) for higher gross limits.

@dsmiley
Copy link
Contributor

dsmiley commented Nov 5, 2021

I like it -- especially treating offset+limit the same as limit, and for monotonically increasing the over request to a point. but I'm not the ideal reviewer. @yonik , @joel-bernstein and @hossman come to mind.

@github-actions
Copy link

This PR had no visible activity in the past 60 days, labeling it as stale. Any new activity will remove the stale label. To attract more reviewers, please tag someone or notify the dev@solr.apache.org mailing list. Thank you for your contribution!

@github-actions github-actions bot added the stale PR not updated in 60 days label Feb 23, 2024
@github-actions
Copy link

github-actions bot commented Oct 6, 2024

This PR is now closed due to 60 days of inactivity after being marked as stale. Re-opening this PR is still possible, in which case it will be marked as active again.

@github-actions github-actions bot added the closed-stale Closed after being stale for 60 days label Oct 6, 2024
@github-actions github-actions bot closed this Oct 6, 2024
@dsmiley
Copy link
Contributor

dsmiley commented Oct 6, 2024

WDYT of re-opening & merging this one @magibney ? Seems straight-forward change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

closed-stale Closed after being stale for 60 days stale PR not updated in 60 days

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants