Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Metalus 2 #345

Open
dafreels opened this issue Oct 31, 2022 · 0 comments
Open

Metalus 2 #345

dafreels opened this issue Oct 31, 2022 · 0 comments
Assignees
@djfreels @dafreels
Labels
2.0.0

Comments

@dafreels
Copy link
Collaborator

This ticket will be used to track version 2 of the Metalus engine.

  • Migrate Spark integration into a separate module
  • Implement state handler to allow applications to be paused/resumed
    • Provide proper status reporting mechanism
  • Replace driver classes with launcher trait which implements the basic features
    • Launchers will need to be runnable from the command line or from spark-submit call
  • Make JSON4S integration not conflict with the different versions of Spark
  • Research HDFS file system libraries to minimize the number of cloud libraries required
  • Shade module jars to simplify usage together
  • Simplify the pipeline parameters
  • Update documentation to make it easier to find features
@dafreels dafreels added the 2.0.0 label Oct 31, 2022
dafreels added a commit that referenced this issue Jan 17, 2023
@dafreels
#345 Initial code changes for new 2.0 version
8935681
dafreels added a commit that referenced this issue Jan 18, 2023
@dafreels
#345 Repackaged classes under the metalus
400cb54
dafreels added a commit that referenced this issue Jan 18, 2023
@dafreels
#345 Forced all JSON parsing through a single object
5d0cebb
dafreels added a commit that referenced this issue Jan 18, 2023
@dafreels
#345 Added support for scala 2.12 and 2.13
d6a8f50
dafreels added a commit that referenced this issue Jan 19, 2023
@dafreels
#345 Switched from log4j to slf4j
a002581
dafreels added a commit that referenced this issue Jan 20, 2023
@dafreels
#345 Migrated FileManagerSteps to core
0b3bc06
dafreels added a commit that referenced this issue Jan 20, 2023
@dafreels
#345 Migrated CredentialSteps to core
f58e084
dafreels added a commit that referenced this issue Jan 20, 2023
@dafreels
#345 Migrated ApiSteps, LoggingSteps and StringSteps to core
ade5bc4
dafreels added a commit that referenced this issue Jan 20, 2023
@dafreels
#345 Migrated copy file pipeline to core
1a6f705
dafreels added a commit that referenced this issue Jan 27, 2023
@dafreels
#345 Refactored EngineMeta to store the command in the command attrib…
06f4cef 
…ute instead of the spark attribute.
dafreels added a commit to dafreels/metalus that referenced this issue Feb 10, 2023
@dafreels
Acxiom#345 Added ability for StepGroup pipelineId attribute to contai…
150ec31 
…n a mapping in addition to the id. Remove some unused classes and performed additional cleanup.
dafreels added a commit to dafreels/metalus that referenced this issue Feb 22, 2023
@dafreels
Acxiom#345 Moved JSONSteps to core
96e71ef
dafreels added a commit to dafreels/metalus that referenced this issue Feb 22, 2023
@dafreels
Acxiom#345 Moved FlowUtilsSteps to core
b6326b4
dafreels added a commit to dafreels/metalus that referenced this issue Feb 22, 2023
@dafreels
Acxiom#345 Removed StreamingUtils from core
527d23d
dafreels added a commit to dafreels/metalus that referenced this issue Feb 23, 2023
@dafreels
Acxiom#345 Created CommandSteps
1a09700
dafreels added a commit that referenced this issue Mar 3, 2023
@dafreels
Merge pull request #357 from dafreels/metalus_2_0_0
3150e1d 
#345 Created CommandSteps
dafreels added a commit to dafreels/metalus that referenced this issue Mar 16, 2023
@dafreels
Acxiom#368 Added parameters to the copy pipeline and a source verific…
35c0a17 
…ation section

Acxiom#345 Updated Connectors traits to have a connectorType of either FILE or DATA
dafreels added a commit to dafreels/metalus that referenced this issue Mar 20, 2023
@dafreels
Acxiom#345 Updated readme files and fixed aws-spark pom.
1c4ad8c
dafreels added a commit to dafreels/metalus that referenced this issue Mar 20, 2023
@dafreels
Acxiom#345 Added a simple step that handles reading the first line of…
5705db3 
… a file and creating a simple Schema.
dafreels added a commit to dafreels/metalus that referenced this issue Mar 22, 2023
@dafreels
Acxiom#345 Added DataStream trait and reader/writer traits. Implement…
ba66e5e 
…ed a CSVDataROwReader.
dafreels added a commit to dafreels/metalus that referenced this issue Mar 22, 2023
@dafreels
Acxiom#345 Updated readHeader step to use CSV parser
da3795f
dafreels added a commit to dafreels/metalus that referenced this issue Mar 23, 2023
@dafreels
Acxiom#345 Fix for FileManager serialization and default DriverSetup …
13a3cf2 
…issues.
dafreels added a commit to dafreels/metalus that referenced this issue Mar 23, 2023
@dafreels
Acxiom#345 Additional bug fixes
c45b41b
dafreels added a commit to dafreels/metalus that referenced this issue Mar 23, 2023
@dafreels
Acxiom#345 Added FlowUtils.readStream step to allow windowing data fr…
5b169da 
…om a source to a pipeline.
dafreels added a commit to dafreels/metalus that referenced this issue Mar 23, 2023
@dafreels
Acxiom#345 Enabled CSV file reading/writing from any FileConnector
6ba9a4f
dafreels added a commit to dafreels/metalus that referenced this issue Mar 27, 2023
@dafreels
Acxiom#345 JDBC DataRow(Reader|Writer)s are now working.
a0ea9da
dafreels added a commit to dafreels/metalus that referenced this issue Mar 27, 2023
@dafreels
Acxiom#345 Updated JDBCDataReference to include the createConnection(…
262168f 
…) again.
dafreels added a commit that referenced this issue Apr 18, 2023
@dafreels
Merge pull request #390 from dafreels/metalus_2_0_0
ac3f787 
#345 Added ProcessUtils for starting and registering processes.
dafreels added a commit to dafreels/metalus that referenced this issue Apr 19, 2023
@dafreels
Acxiom#345 Updated agent to close jdbc connection cleanly. Updated sl…
6dc5aaa 
…f4j to version 1.7.36 to remove conflicts with play and spark. Fixed agent package structure.
dafreels added a commit that referenced this issue Apr 19, 2023
@dafreels
Merge pull request #391 from dafreels/metalus_2_0_0
89477f9 
#345 Agent work
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on build scripts
dde218d
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on build scripts
4a2642f
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on build scripts
ca53501
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on build scripts
e4a0271
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on build scripts
ed524cb
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on build scripts
2ed762d
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on build scripts
cbe4491
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Updated test to fix build.
7fdedcc
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Updated test to fix build.
b64c389
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Updated test to fix build.
5ba2a78
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on build scripts
803f896
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on build scripts
0af10b4
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on build scripts
8c739b6
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on spark build issues
abd97ec
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Added aws-spark build
b6a8e92
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Added scala version 2.13
cc73075
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Updated cache mechanism
b7b0651
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Added java distribution for builds
f10db8e
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Added java distribution for builds
19688fb
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Updated java version
4cc95bc
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on shared artifacts
6fe1876
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Working on shared artifacts
27d4547
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Updating build to include spark aws
9f4c5cd
dafreels added a commit to dafreels/metalus that referenced this issue Apr 20, 2023
@dafreels
Acxiom#345 Fixed typo
d856e65
dafreels added a commit to dafreels/metalus that referenced this issue Apr 24, 2023
@dafreels
Acxiom#345 Fixed issue with connector provider code causing aws-spark…
ad54621 
… build to fail.
dafreels added a commit to dafreels/metalus that referenced this issue Apr 24, 2023
@dafreels
Acxiom#345 Updated pr build file for 2.0
2871610
dafreels added a commit to dafreels/metalus that referenced this issue Apr 25, 2023
@dafreels
Acxiom#345 Added endpoint to generate classpath
202f4a0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@djfreels djfreels

@dafreels dafreels

Labels
2.0.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@djfreels @dafreels