-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
chore(glue-alpha): refactor spark jobs code-related props handling #33567
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #33567 +/- ##
=======================================
Coverage 82.37% 82.37%
=======================================
Files 120 120
Lines 6933 6933
Branches 1169 1169
=======================================
Hits 5711 5711
Misses 1119 1119
Partials 103 103
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
After the second commit, I had to export Another thought I'm having is that there's a lot of duplicative logic in the different spark job classes, and I wonder if it can be simplified by either (1) Extracting common protected methods into |
@GavinZZ this is a PR where I address my comments in #33238 (comment) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wow, this is awesome! Thank you for the refactoring and typo fixing!
Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
AWS CodeBuild CI Report
Powered by github-codebuild-logs, available on the AWS Serverless Application Repository |
Thank you for contributing! Your pull request will be updated from main and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork). |
Comments on closed issues and PRs are hard for our team to see. |
Issue # (if applicable)
Related to #33356
Reason for this change
address the feedback section of extraJars, extraFiles, extraPythonFiles, extraJarsFirst + refactor to reduce duplicative code across the different spark job classes
Description of changes
extraPythonFiles
,extraFiles
,extraJars
, andextraJarsFirst
available in pyspark jobsextraFiles
,extraJars
, andextraJarsFirst
available in scala spark jobsSparkJob
class andSparkJobProps
to reduce duplicate logic across different spark job classessetupSparkCodeArguments
protected
method along with an interfaceSparkExtraCodeProps
to handle setting upCode
-related argumentssparkUI
propspark-ui-utils.ts
intospark-job.ts
and retire itDescribe any new or updated permissions being added
N/A
Description of how you validated changes
updated unit tests
Checklist
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license