This repository has been archived by the owner on May 18, 2023. It is now read-only.
Better implementation of SparseArray. #120
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…hich it really isn't), but more imitating Array itself in pattern.
…ing scalastyle error exposed by Travis.
…the array to the map function.
… the default isn't reused for each element.
|
Note to self: Check if anything but the reduce function actually cares about the default being a zero, as opposed to the most common value. Check if anything is using the reduce function. If the answer to both those questions is "no", we can get rid of this assumption by eliminating the reduce function. |
…ings with it that are non-standard-seq or materializing, and no one needs it.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I tried for a while to fix this while staying within the scala collections framework.
The problem is, SparseArrays really aren't normal collections - it's the attempt to view them as such that causes a lot of the unneeded materialization.
When that wouldn't work, I looked at how Array itself was implemented. It isn't part of the scala collections framework (of course), but has wrappers to help it fit therein.
I have not made said wrappers - they are a lot of extraneous boilerplate code with no current use, as far as I could tell.
Note: I have changed the signature for creating a sparse array. I don't think anyone but me is currently using it outside spark itself (since it was only recently exposed at all, and that was for me), but it's possible there are ripple effects in some examples and salt modules.
Fixes #119