Optimize list subtraction (A -- B) and make it yield on large inputs #1998
Conversation
jhogberg
force-pushed the
jhogberg:john/erts/minusminus_trapping/OTP-15371
branch
2 times, most recently
from
5dd398f
to
655ef4d
| product of the length of its operands, which was extremely slow on long | ||
| lists.</p> | ||
|
|
||
| <p>These days the run-time complexity is "n log n" and the operation will |
bjorng
Oct 24, 2018
Contributor
"These days". I suggest making it more specific, e.g. from "OTP 22".
"These days". I suggest making it more specific, e.g. from "OTP 22".
| lists.</p> | ||
|
|
||
| <p>These days the run-time complexity is "n log n" and the operation will | ||
| complete quickly even on very long lists. In fact, it's faster and uses |
bjorng
Oct 24, 2018
Contributor
"it's": OTP's documentation guidelines says that contraction should be avoided.
"it's": OTP's documentation guidelines says that contraction should be avoided.
jhogberg
Oct 24, 2018
Author
Contributor
Thanks, fixed!
Thanks, fixed!
jhogberg
force-pushed the
jhogberg:john/erts/minusminus_trapping/OTP-15371
branch
6 times, most recently
from
4f4b5da
to
2643f64
The first stage wasn't bounded by reductions, and it bumped far more reductions than it should have due to a logic bug.
jhogberg
force-pushed the
jhogberg:john/erts/minusminus_trapping/OTP-15371
branch
from
2643f64
to
b719424
This greatly increases the performance of '--'/2 which does a lot of term comparisons.
jhogberg
force-pushed the
jhogberg:john/erts/minusminus_trapping/OTP-15371
branch
3 times, most recently
from
44e6530
to
127e78f
The removal set now uses a red-black tree instead of an array on large inputs, decreasing runtime complexity from `n*n` to `n*log(n)`. It will also exit early when there are no more items left in the removal set, drastically improving performance and memory use when the items to be removed are present near the head of the list. This got a lot more complicated than before as the overhead of always using a red-black tree was unacceptable when either of the inputs were small, but this compromise has okay-to-decent performance regardless of input size. Co-authored-by: Dmytro Lytovchenko <dmytro.lytovchenko@erlang-solutions.com>
jhogberg
force-pushed the
jhogberg:john/erts/minusminus_trapping/OTP-15371
branch
from
127e78f
to
eb9ee88
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
@kvakvs had to pause his work on #1993 so this PR takes over where he left off. I've refactored it to make state handling easier to follow, and started using a tree instead of an array for the removal set, decreasing run-time complexity from
n*nton*log(n).Other than that it's pretty much the same, and it should still be relatively easy to backport.