Dimension Independent Matrix Square Using MapReduce #336

wants to merge 5 commits into


None yet
3 participants

rezazadeh commented Apr 6, 2014

Provide Implementation of DIMSUM for squaring a Tall and Skinny Matrix as described in: http://arxiv.org/abs/1304.1467

Also, refactor Matrix squaring into a MatrixAlgebra object.

Scaling this up now until it breaks. This PR is currently only a request for comments. Will update with scaling results once ready.

rezazadeh changed the title from Dimension Independent Similarity Computation to Dimension Independent Matrix Square Using MapReduce Apr 6, 2014

Merged build triggered.

Merged build started.

Merged build finished.

Merged build triggered.

Merged build started.

Merged build finished. All automated tests passed.

All automated tests passed.
Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13830/

@andrewor14 andrewor14 pushed a commit to andrewor14/spark that referenced this pull request Apr 7, 2014

@pwendell pwendell Merge pull request #336 from liancheng/akka-remote-lookup
Get rid of `Either[ActorRef, ActorSelection]'

In this pull request, instead of returning an `Either[ActorRef, ActorSelection]`, `registerOrLookup` identifies the remote actor blockingly to obtain an `ActorRef`, or throws an exception if the remote actor doesn't exist or the lookup times out (configured by `spark.akka.lookupTimeout`).  This function is only called when an `SparkEnv` is constructed (instantiating driver or executor), so the blocking call is considered acceptable.  Executor side `ActorSelection`s/`ActorRef`s to driver side `MapOutputTrackerMasterActor` and `BlockManagerMasterActor` are affected by this pull request.

`ActorSelection` is dangerous and should be used with care.  It's only absolutely safe to send messages via an `ActorSelection` when the remote actor is stateless, so that actor incarnation is irrelevant.  But as pointed by @ScrapCodes in the comments below, executor exits immediately once the connection to the driver lost, `ActorSelection`s are not harmful in this scenario.  So this pull request is mostly a code style patch.

mateiz commented Aug 27, 2014

@rezazadeh are you still updating this PR or have you sent a new one for DimSum now? If this one is stale, it would be great if you could close it.


rezazadeh commented Aug 27, 2014

Moved to #1778

rezazadeh closed this Aug 27, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment