Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement Spark library #353

Open
dafreels opened this issue Feb 17, 2023 · 1 comment
Open

Implement Spark library #353

dafreels opened this issue Feb 17, 2023 · 1 comment
Assignees
@djfreels @dafreels
Labels
2.0.0

Comments

@dafreels
Copy link
Collaborator

No description provided.

@dafreels dafreels added the 2.0.0 label Feb 17, 2023
dafreels added a commit to dafreels/metalus that referenced this issue Feb 17, 2023
@dafreels
Acxiom#349 Updated code based on PR feedback.
44150fa 
Acxiom#353 Beginning work on metalus-spark
@dafreels
Copy link
Collaborator Author

dafreels commented Feb 21, 2023 •
edited

  • Get project running stand-alone
  • Provide a step that allows starting a job from a non-spark pipeline
  • Enable context sharing between calling pipeline and spark pipeline
    • SessionContext can be used to restore state one the Spark job starts.
    • SessionContext can be used to restore the Spark state once complete for use by the rest of the Pipeline

dafreels added a commit to dafreels/metalus that referenced this issue Feb 22, 2023
@dafreels
Acxiom#353 Spark project now builds. Added proper tests.
7c61d61
dafreels added a commit to dafreels/metalus that referenced this issue Feb 22, 2023
@dafreels
Acxiom#353 Migrated SparkConfigurationSteps to spark project.
8ce48cb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@djfreels djfreels

@dafreels dafreels

Labels
2.0.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@djfreels @dafreels