Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move DDF to SparkSQL #26

Merged
merged 65 commits into from
Nov 18, 2014
Merged

Move DDF to SparkSQL #26

merged 65 commits into from
Nov 18, 2014

Conversation

Huandao0812
Copy link
Contributor

No description provided.

Huandao0812 and others added 30 commits October 14, 2014 10:22
@Huandao0812
DDF on SparkSQL-1.1.0
28a97f3
@Huandao0812
get RDD[Row] from cacheTable SchemaRDD
4f39061
@Huandao0812
use RDD[Row] instead of SchemaRDD to do transformation
c4cb14a
@Huandao0812
sample use RDD[Row]
56811f7
@Huandao0812
set SparkSQL compression use HiveContext.setConf
f078d34
@Huandao0812
clean up, instantiate new DDF using manager.newDDF
9c029db
@Huandao0812
fix bug in CrossValidation.scala
d314e37
@Huandao0812
debugging, add test to get RDD[LabeledPoint]
81d637c
@Huandao0812
add test to RepHandlerSuite
b8f0be7 
to test for getting RDD[LabeledPoint]
@Huandao0812
fix MLlibIntegrationSuite
57aa533 
fix typo in RepresentationHandlerSuite
@Huandao0812
add log message in applyModel
47ab000
@Huandao0812
debugging
1ce1f3f
@Huandao0812
debugging
7941684
@Huandao0812
[wip] debugging
f4b7270
@Huandao0812
add prediction for Vector
3a1df67
@Huandao0812
add new SparkModel
b409c2c
@Huandao0812
add method to instantiate new Model dynamically
1daa549
@Huandao0812
convert Array[java.lang.Double] to Array[Double]
ca036bb
@Huandao0812
use io.spark.ddf.ml.Model.java
e25a0f9
@Huandao0812
fix MLSupporter.newModel
82ba909
@Huandao0812
fix spark.ddf.ml.model.predict
ab6464d
@Huandao0812
Refactor MLlibIntegrationSuite
50b7921
@Huandao0812
bug, don't shutdown manager after test Kmeans
576abfe
@Huandao0812
fix FiveNum
6f78d2d
@Huandao0812
fix VectorQuantile
3951520
add log message
a253b5b
@Huandao0812
git FiveNum
a474d51
@Huandao0812
add log message
5e35886
@Huandao0812
fix getVectorHistogram
45bc20d
@Huandao0812
don't drop table in MutabilityHandler
0bc290c
@Huandao0812
Copy link
Contributor Author

I also create a tag spark-0.9.2 for the current master

@Huandao0812 Huandao0812 changed the title Ddf sparksql 1.1.0 Move DDF to SparkSQL Nov 7, 2014
@khangich khangich self-assigned this Nov 7, 2014
@khangich
Copy link
Contributor

khangich commented Nov 7, 2014

There are some failed tests: https://gist.github.com/khangich/380f3a7420615963b982

@Huandao0812 please take a look and see if we can resolve any.

@nhanitvn
Copy link
Contributor

nhanitvn commented Nov 7, 2014

ok to test

@Huandao0812
Copy link
Contributor Author

All known issue:
ALSTest.TestALS dues to change in MLlib side
other three tests due to current spark 1.2.0-SNAPSHOT don't support quantile that take array in put yet.

@Huandao0812
Copy link
Contributor Author

ok to test

@khangich
Copy link
Contributor

khangich commented Nov 7, 2014

@Huandao0812 if so can you make it merge-able ?

@Huandao0812
Merge branch 'master' into ddf-sparksql-1.1.0
ecedccf 
Conflicts:
	bin/run-example
	pom.xml
	project/RootBuild.scala
@Huandao0812
Copy link
Contributor Author

merged master into this

@khangich
Copy link
Contributor

@binhmop @Huandao0812 We have ddf 1.0 compatible with Spark 0.9.2, this branch should be new DDF version 1.1. Please review.

@Huandao0812
Copy link
Contributor Author

should we merge this soon?

@piccolbo
Copy link
Contributor

if we have two skip some tests and temporarily break summary and fivenum as mentioned in meeting, plz create issue about that and reference this one so that we don't forget about it. Thanks,

@binhmop binhmop assigned binhmop and unassigned khangich Nov 17, 2014
@piccolbo piccolbo mentioned this pull request Nov 17, 2014
Open
@binhmop
Copy link
Member

binhmop commented Nov 17, 2014

ok to test

binhmop added a commit that referenced this pull request Nov 18, 2014
@binhmop
Merge pull request #26 from ddf-project/ddf-sparksql-1.1.0
1f181d0 
DDF implementation on SparkSQL
@binhmop binhmop merged commit 1f181d0 into master Nov 18, 2014
@piccolbo piccolbo mentioned this pull request Nov 18, 2014
Closed
@ljzzju ljzzju mentioned this pull request Nov 21, 2014
Merged
@hai-adatao hai-adatao deleted the ddf-sparksql-1.1.0 branch April 13, 2016 10:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@binhmop binhmop

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@Huandao0812 @khangich @nhanitvn @piccolbo @binhmop @qinxinwei @ljzzju @pzzs