control-service: data job synchronizer error handling #2742
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mivanov1988
changed the title
mivanov1988
force-pushed
the
person/miroslavi/data-job-synchronizer-error-handling
branch
from
8964d74
to
4f9ef0e
Compare
Why As part of the VEP-2272, we have to introduce a process to synchronize the data jobs from the database to the Kubernetes. The initial implementation depends on the data_job_deployment table for both reading and writing operations. As the deployment process operates asynchronously with unpredictable durations, users may believe that their deployment has been completed while it may not have. To address this issue, we'll be introducing a new table called "desired_data_job_deployment" and renaming the current one to "actual_data_job_deployment". The read operations will read from "actual_data_job_deployment" while the write operations will write to "desired_data_job_deployment". What We've implemented new deployment tables and modified the deployment logic to operate in conjunction with these new tables. Just so you know, this is just the second phase of implementation. The following enhancements are planned for future PRs: We will annotate the method DataJobsSynchronizer.synchronizeDataJobs() with @scheduled. Improved exception handling will be integrated. More tests will be included in subsequent updates. ThreadPool configuration will be tuned and exposed through the application.properties. Testing Done Integration tests Signed-off-by: Miroslav Ivanov miroslavi@vmware.com
mivanov1988
force-pushed
the
person/miroslavi/data-job-synchronizer-error-handling
branch
from
3094e98
to
5f90aa1
Compare
Why As part of VEP-2272, we need to introduce a process for synchronizing data jobs from the database to Kubernetes. In the event of a data job deployment failure (due to user or platform errors), the current implementation attempts to deploy the data job during every synchronization cycle. What We have added a data job deployment status field to the desired_data_job_deployment table. This status is used to determine whether the deployment failed in the previous cycle and should be skipped in the current one. Testing done Unit tests Signed-off-by: Miroslav Ivanov miroslavi@vmware.com
mivanov1988
force-pushed
the
person/miroslavi/data-job-synchronizer-error-handling
branch
from
1c40bb8
to
ab3c4b2
Compare
mrMoZ1
approved these changes
Oct 3, 2023
mivanov1988
deleted the
person/miroslavi/data-job-synchronizer-error-handling
branch
antoniivanov
reviewed
Oct 4, 2023
| if (DeploymentStatus.USER_ERROR.equals(desiredJobDeployment.getStatus()) | ||
| || DeploymentStatus.PLATFORM_ERROR.equals(desiredJobDeployment.getStatus())) { | ||
| log.debug( | ||
| "Skipping the data job [job_name={}] deployment due to the previously failed deployment" |
There was a problem hiding this comment.
Is this to prevent re-tries?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why
As part of VEP-2272, we need to introduce a process for synchronizing data jobs from the database to Kubernetes. In the event of a data job deployment failure (due to user or platform errors), the current implementation attempts to deploy the data job during every synchronization cycle.
What
We have added a data job deployment status field to the desired_data_job_deployment table. This status is used to determine whether the deployment failed in the previous cycle and should be skipped in the current one. The failed deployment status can be reset by invocation of updateDeployment API event without a job code change (it will be implemented as part of https://github.com/vmware/versatile-data-kit/pull/2731/files).
This is just the third phase of implementation. The following enhancements are planned for future PRs:
Testing done
Unit tests
Signed-off-by: Miroslav Ivanov miroslavi@vmware.com