Optimize list subtraction (A -- B) and make it yield on large inputs #1998
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jhogberg
force-pushed
the
john/erts/minusminus_trapping/OTP-15371
branch
2 times, most recently
from
5dd398f
to
655ef4d
Compare
bjorng
reviewed
Oct 24, 2018
| product of the length of its operands, which was extremely slow on long | ||
| lists.</p> | ||
|
|
||
| <p>These days the run-time complexity is "n log n" and the operation will |
There was a problem hiding this comment.
"These days". I suggest making it more specific, e.g. from "OTP 22".
bjorng
reviewed
Oct 24, 2018
| lists.</p> | ||
|
|
||
| <p>These days the run-time complexity is "n log n" and the operation will | ||
| complete quickly even on very long lists. In fact, it's faster and uses |
There was a problem hiding this comment.
"it's": OTP's documentation guidelines says that contraction should be avoided.
jhogberg
force-pushed
the
john/erts/minusminus_trapping/OTP-15371
branch
6 times, most recently
from
4f4b5da
to
2643f64
Compare
The first stage wasn't bounded by reductions, and it bumped far more reductions than it should have due to a logic bug.
jhogberg
force-pushed
the
john/erts/minusminus_trapping/OTP-15371
branch
from
2643f64
to
b719424
Compare
This greatly increases the performance of '--'/2 which does a lot of term comparisons.
jhogberg
force-pushed
the
john/erts/minusminus_trapping/OTP-15371
branch
3 times, most recently
from
44e6530
to
127e78f
Compare
The removal set now uses a red-black tree instead of an array on large inputs, decreasing runtime complexity from `n*n` to `n*log(n)`. It will also exit early when there are no more items left in the removal set, drastically improving performance and memory use when the items to be removed are present near the head of the list. This got a lot more complicated than before as the overhead of always using a red-black tree was unacceptable when either of the inputs were small, but this compromise has okay-to-decent performance regardless of input size. Co-authored-by: Dmytro Lytovchenko <dmytro.lytovchenko@erlang-solutions.com>
jhogberg
force-pushed
the
john/erts/minusminus_trapping/OTP-15371
branch
from
127e78f
to
eb9ee88
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
@kvakvs had to pause his work on #1993 so this PR takes over where he left off. I've refactored it to make state handling easier to follow, and started using a tree instead of an array for the removal set, decreasing run-time complexity from
n*nton*log(n).Other than that it's pretty much the same, and it should still be relatively easy to backport.