-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
P1: Code stubs for parallel matrix-vector multiplication #20
Comments
This is 1 of the 4 initial Spark primitives we need for dictionary learning: matrix-vector multiplication. It can be accomplished through steps:
|
@magsol |
The second one is easy too; we can broadcast the vector v and the list of On Tue, Dec 15, 2015 at 6:59 PM MOJTABAFA notifications@github.com wrote:
|
@magsol import numpy as np
lines = sc.textFile("/home/targol/Desktop/s.txt")
dataArrays = lines.map(lambda s:s.split("\t"))
data = raw_data.flatMap(lambda s: s.split("\t")).collect() there we can remove /t from textfile but the problem is the file will be changed into list of strings. I mean the rows would be string and we can not change the screen lists to float. also we have tested the dataArrays = data.map(lambda s : np.asarray(s))
dataArrays.collect()
dataArrays.first() so in this case we can change the lists to arrays. hence we can access the array elements for example the first element could be as : >>> dataArrays[2]
'0.728215' but we cannot do that on all elements even with following codes : def parseVector(line):
return np.array([float(x) for x in line.split(',')])
a = parseVector(dataArrays.first())
a = parseVector(dataArrays.first())
a
def parse(data):
a = parseVector(dataArrays.first())
floatData = dataArrays.map(lambda s : parseVector(s))
.
.
.
float = dataArrays.map(lambda x : x.foreach(lambda s: float(s)))
float
float.collect always the Error is as follows : "It appears that you are attempting to broadcast an RDD or reference an RDD from an "
Exception: It appears that you are attempting to broadcast an RDD or reference an RDD from an action or transformation. RDD transformations and actions can only be invoked by the driver, not inside of other transformations; for example, rdd1.map(lambda x: rdd2.values.count() * x) is invalid because the values transformation and count action cannot be performed inside of the rdd1.map transformation. For more information, see SPARK-5063. even we tried to exactly code the page 219 of spark book which you already gave me : but even this code again generated the previous Error. So I confused alot again today. Please help us and let us know how should we can continue. |
Ok, lots of issues here to clear up. I'm glad you're working through this; keep it coming! BranchingFirst of all, I think it'd be very useful if you were able to commit the code you're using. However, we don't want to "pollute" the repository with experimental code that you're just tinkering with. Fortunately, git has a solution: branching. Effectively, you create a "branch" in the revision tree, leaving the "master" branch (default branch you've been working on this whole time) separate from your "experimental" branch, where you can go wild with whatever edits you want. That way, I can still track your progress and pull down your code, but it won't nuke our "master" branch that has our stable code. I'd highly recommend getting an experimental branch set up that we can tinker with. Once we have a solution, you can either implement it directly in the master branch, or "merge" the changes from one branch into another. Datatype conversionYou're almost there, just missing a crucial step that is in the book. dataArrays = lines.map(lambda s:s.split("\t")) That does indeed split each line up on the tab And that's where my next big point comes in: RDD APIs. RDDsThe error you're getting is related to how distributed computing works. Whenever you invoke a method on an RDD, e.g. The code in the book is correct; I just think you're not matching up your braces correctly. Here's what the Python code would look like: raw_data = sc.textFile('/path/to/csv/data') # creates an RDD of strings
data = raw_data.filter(lambda x: x.startswith("#")).map(lambda x: map(float, x.split(","))) Another point of confusion here may be the two What that second (in our case, we also want to take one extra step and convert the list to a NumPy array) |
I had a few challenges during installing thunder package on the temporary spark server and I faced with a few error mostly related to some math dependents even after installing anaconda package. We worked with two different version of python2 and 3 and each of which came up with unknown errors. Any help will be highly appreciated. |
@magsol
import numpy as np
def floater(x):
for i in range(len(x)):
if(x[i]!= ''):
x[i]= float(x[i])
else:
del x[i]
return (x)
#importing the data
lines = sc.textFile("/home/targol/Desktop/s.txt")
# floating the all the matrix data
dataArrays = lines.map(lambda s:floater(s.split("\t")))
#changing the list of float data to np array
dataArrays = lines.map(lambda s:np.asarray(floater(s.split("\t")))) So now our matrix is a float matrix which is ready to be processed.
### whitening process
S = dataArrays.first() -dataArrays.first().mean()
S = S / sla.norm(S, axis = 0)
Actually I totally get confused. please help us. by the way still our program is not so large but I'll put them in a new branch very soon. |
More thorough answers later, but a few quick points--
No, this code is extremely inefficient. You have an entire extraneous RDD.map just to convert to numpy arrays when you could do that at the end of the previous map. Also, the explicit for loop is going to slow down the program drastically. The implementation in the book was fine and will do what you need it to do. Why not use that?
This won't work because S is a row-distributed RDD. RDD.first returns the first element of the RDD, in this case row 0. So you're subtracting off the mean of row 0 from row 0, and that's it. Everything else is unchanged. Use the thunder library and the RowMatrix data structure. I'll help with that.
My answer to this is the same as the previous: use thunder, then we don't have to worry about indexing ourselves (thunder will handle it). I'll help you get that up and running. |
I'm not at my computer so I can't access the pip.txt file you posted, but I do know thunder requires Python 2, so that may be one source of errors. iPhone'd
|
I've created a new branch In the meantime, check out the thunder documentation for installation and see if that addresses your problems. |
@milad181 Yes, the issue is you're missing a fortran compiler (which SciPy needs in order to build the underlying LAPACK / ARPACK libraries it relies on). I'd personally recommend downloading Anaconda and using that instead. Check out the Dockerfile I committed in quinn-branch for the steps needed. |
…, vector-matrix multiplication), and P4 (#46, deflation). Not yet tested, but this is what an initial proof of concept looks like.
We would need the code for parallel matrix-vector multiplication and the corresponding test cases to see whether we could get a performance boost from doing so, which would support the development of the parallel version of r1DL and sccDL.
The text was updated successfully, but these errors were encountered: