Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize list subtraction (A -- B) and make it yield on large inputs #1998

Merged

Conversation

jhogberg
Copy link
Contributor

@kvakvs had to pause his work on #1993 so this PR takes over where he left off. I've refactored it to make state handling easier to follow, and started using a tree instead of an array for the removal set, decreasing run-time complexity from n*n to n*log(n).

Other than that it's pretty much the same, and it should still be relatively easy to backport.

@jhogberg jhogberg added team:VM Assigned to OTP team VM enhancement labels Oct 23, 2018
@jhogberg jhogberg self-assigned this Oct 23, 2018
@jhogberg jhogberg force-pushed the john/erts/minusminus_trapping/OTP-15371 branch 2 times, most recently from 5dd398f to 655ef4d Compare October 24, 2018 09:39
product of the length of its operands, which was extremely slow on long
lists.</p>

<p>These days the run-time complexity is "n log n" and the operation will
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"These days". I suggest making it more specific, e.g. from "OTP 22".

lists.</p>

<p>These days the run-time complexity is "n log n" and the operation will
complete quickly even on very long lists. In fact, it's faster and uses
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"it's": OTP's documentation guidelines says that contraction should be avoided.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, fixed!

@jhogberg jhogberg force-pushed the john/erts/minusminus_trapping/OTP-15371 branch 6 times, most recently from 4f4b5da to 2643f64 Compare October 24, 2018 13:10
jhogberg and others added 2 commits October 25, 2018 07:47
The first stage wasn't bounded by reductions, and it bumped far
more reductions than it should have due to a logic bug.
@jhogberg jhogberg force-pushed the john/erts/minusminus_trapping/OTP-15371 branch from 2643f64 to b719424 Compare October 25, 2018 05:52
This greatly increases the performance of '--'/2 which does a lot
of term comparisons.
@jhogberg jhogberg force-pushed the john/erts/minusminus_trapping/OTP-15371 branch 3 times, most recently from 44e6530 to 127e78f Compare October 26, 2018 13:20
The removal set now uses a red-black tree instead of an array on
large inputs, decreasing runtime complexity from `n*n` to
`n*log(n)`. It will also exit early when there are no more items
left in the removal set, drastically improving performance and
memory use when the items to be removed are present near the head
of the list.

This got a lot more complicated than before as the overhead of
always using a red-black tree was unacceptable when either of the
inputs were small, but this compromise has okay-to-decent
performance regardless of input size.

Co-authored-by: Dmytro Lytovchenko <dmytro.lytovchenko@erlang-solutions.com>
@jhogberg jhogberg force-pushed the john/erts/minusminus_trapping/OTP-15371 branch from 127e78f to eb9ee88 Compare October 30, 2018 07:04
@jhogberg jhogberg merged commit eb9ee88 into erlang:maint Nov 2, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement team:VM Assigned to OTP team VM
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants