Add Auto-Disable Failing Connections feature #11099
Conversation
This is removing the attemptId from the create attempt activity to replace it with the attemptNumber. This will be modified in the workflow in a later commit.
github-actions
bot
added
area/platform
| final String JobStatusSelect = "SELECT status FROM jobs "; | ||
| return jobDatabase.query(ctx -> ctx | ||
| .fetch(JobStatusSelect + "WHERE " + | ||
| "CAST(scope AS VARCHAR) = ? AND " + |
There was a problem hiding this comment.
Why do we need to cast as a VARCHAR? scope = '?' should be working (same with the config type).
There was a problem hiding this comment.
I thought that was the convention since most of the other queries casted the scope, but just ran the unit tests without it and had no problems, so i'll remove it.
There was a problem hiding this comment.
I misspoke earlier, looks like the convention is not to cast the scope. But ConfigType is normally casted.
| * connection to prevent wasting resources | ||
| */ | ||
| @ActivityMethod | ||
| boolean autoDisableFailingConnection(AutoDisableConnectionActivityInput input); |
There was a problem hiding this comment.
Temporal recommends to use a POJO for the input/output. will help with adding element in the input or the output. This shouldn't require versioning if if we add something in the POJO. Even if it is likely that we will use one it might help to avoid potential issues if we forget it.
There was a problem hiding this comment.
done, added an output object
What
Adds feature to automatically disable failing connections. This addresses issue 9715. When this feature is turned on, it will disable a connection after
MAX_FAILED_JOBS_IN_A_ROW_BEFORE_CONNECTION_DISABLEconsecutive failures (currently defaults to 100 jobs) or afterMAX_DAYS_OF_ONLY_FAILED_JOBS_BEFORE_CONNECTION_DISABLEdays of only failed jobs (currently defaults to 14 days).todo:
Notify user when connection has been auto-disable.Will be followed up through this issue.listJobStatusWithConnectionis not filtering by timestamp in unit testsHow
When the
AUTO_DISABLE_FAILING_CONNECTIONSfeature flag is turned on, after a job failure, theDisableActivitywill check if either of the above criteria has been met. If so, it will set the connection status toINACTIVE.Recommended reading order
airbyte-scheduler/persistence/src/main/java/io/airbyte/scheduler/persistence/DefaultJobPersistence.javaairbyte-workers/src/main/java/io/airbyte/workers/temporal/scheduling/activities/AutoDisableConnectionActivityImpl.java