Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Spark Integration and Custom Types #19

Merged
merged 35 commits into from
Oct 30, 2016
Merged

Spark Integration and Custom Types #19

merged 35 commits into from
Oct 30, 2016

Conversation

hollinwilkins
Copy link
Member

This branch allows for MLeap transformers to execute on Spark DataFrames
This branch allows for the use of custom types in a LeapFrame as well as in the MLeap transformers.

Further changes:

  1. Load the registry from reference.conf files instead of manually building them
  2. Load custom types for both Bundle.ML and MLeap Runtime from reference.conf files

@hollinwilkins
Initial implementation of UDFs with TransformBuilder.
fa30144
@hollinwilkins
Move UDF execution to be row level.
cfb659a
@hollinwilkins
Merge branch 'master' into feature/udfs
34792a0
@hollinwilkins
Refining UDFs and adding in unit tests.
b2372c6
@hollinwilkins
Finish refactoring and testing user defined functions.
3499d00
@hollinwilkins
Get rid of unused import.
adbd58c
@hollinwilkins
Converted to UDF transformers except vector assembler.
b36302e
@hollinwilkins
Use UDFs for all transform builders.
874c8d8
@hollinwilkins
Update comment for UDF.
ea21977
@hollinwilkins
Get rid of unneeded LeapFrame value builders.
61319dc
@hollinwilkins
Change ArraySelector to a MultipleFieldSelector.
307ec06
@hollinwilkins
Rename back to ArraySelector.
1a3b2b2
@hollinwilkins
AnyType should not inherit BasicType.
694fd5d
@hollinwilkins
Allow MLeap transformers to transform Spark DataFrames efficiently.
1ab0a05
@hollinwilkins
Implement reverse type reflection.
ff1d4ee
@hollinwilkins
Better type conversions to Spark.
baf9c7b
@hollinwilkins
Fix type converters, add mleap array expression.
6ca2837
@hollinwilkins
Mark spark-mllib-local as a provided dependency.
59999d9
@hollinwilkins
Fix OneHotEncoder Vector types.
1970066
@hollinwilkins
Move SparkSupport around.
8fd5841
@hollinwilkins
Fix several issues with serialization.
78003f3
@hollinwilkins
Get rid of reverse reflection.
7867124
@hollinwilkins
First pass at custom types.
b650d36
@hollinwilkins
Bundle.ML uses configs for generating registry now.
fba2fe9
@hollinwilkins
Improved support classes with contexts.
9e2e417
@hollinwilkins
Finish custom types with some tests and reflection fixes.
c099355
@hollinwilkins
Get rid of Mleap/Spark Registry, no longer needed.
a976ef0
@hollinwilkins
Move a few files around.
5f8f416
@hollinwilkins
Merge branch 'master' into feature/custom-types
81fac37
@hollinwilkins
Fix reference.conf for MLeap runtime.
aa8d286
@prianna
Copy link
Member

prianna commented Oct 27, 2016

Reviewing today

@hollinwilkins hollinwilkins merged commit f9ac8e1 into master Oct 30, 2016
@hollinwilkins hollinwilkins deleted the feature/custom-types branch October 30, 2016 19:23
This was referenced Oct 30, 2016
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hollinwilkins @prianna