[GOBBLIN-1439] Handle flow configs that cause compilation errors from being added and blocking scheduler#3273
Closed
Will-Lo wants to merge 5 commits intoapache:masterfrom
Closed
Conversation
…multihop flow compilation) and prevent compilation errors from blocking flows from being scheduled
Will-Lo
changed the title
Codecov Report
@@ Coverage Diff @@
## master #3273 +/- ##
============================================
- Coverage 46.47% 8.99% -37.48%
+ Complexity 9996 1739 -8257
============================================
Files 2036 2037 +1
Lines 79200 79237 +37
Branches 8834 8838 +4
============================================
- Hits 36809 7130 -29679
- Misses 38971 71406 +32435
+ Partials 3420 701 -2719 Continue to review full report at Codecov.
|
arjun4084346
reviewed
May 1, 2021
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_catalog/FlowCatalog.java
Outdated
Show resolved
Hide resolved
Contributor
|
a minor comment. lgtm otherwise |
aplex
reviewed
May 3, 2021
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/api/MockSpecCatalogListener.java
Outdated
Show resolved
Hide resolved
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_catalog/FlowCatalog.java
Outdated
Show resolved
Hide resolved
...c/test/java/org/apache/gobblin/service/modules/scheduler/GobblinServiceJobSchedulerTest.java
Outdated
Show resolved
Hide resolved
…or reduced flakiness
aplex
approved these changes
May 5, 2021
sv2000
requested changes
May 5, 2021
gobblin-runtime/src/main/java/org/apache/gobblin/runtime/spec_catalog/FlowCatalog.java
Outdated
Show resolved
Hide resolved
jhsenjaliya
pushed a commit
to jhsenjaliya/incubator-gobblin
that referenced
this pull request
Jun 7, 2021
… being added and blocking scheduler Closes apache#3273 from Will-Lo/handle-flowspec- compilation-errors
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
@aplex @arjun4084346 @jack-moseley @umustafi Please take a look
JIRA
Description
FlowSpecs that fail to compile due to unexpected exceptions (e.g. NullPointerException) can cause the Job Scheduler to silently fail adding jobs.
To fix the error we should first reject users from adding flowConfigs that cause compilation errors. Also, if encountering a flow that fails to compile, log it and continue scheduling the rest of the flows instead of silently failing with unscheduled jobs.
Tests
Added unit tests mocking listener compilation errors
Commits