Issue 7239 - Check Kubernetes pods/jobs are deleted after a bulk import#7272
Conversation
patchwork01
added
the
needs-reviewer
| String executionArn = stateMachineArn.replace(":stateMachine:", ":execution:") + ":" + executionName; | ||
| DescribeExecutionResponse response = sfnClient.describeExecution(req -> req.executionArn(executionArn)); | ||
| LOGGER.info("Found execution for job {}: {}", jobId, response); | ||
| LOGGER.info("Error: {}", response.error()); |
There was a problem hiding this comment.
Should errors be logged at info level? Will an error always be present?
There was a problem hiding this comment.
This is a system test driver, I think it's important to track the progress of a test. I've added a null check for the error.
| PodList list = k8sProvider.getClient(properties).pods() | ||
| .inNamespace(properties.get(BULK_IMPORT_EKS_NAMESPACE)) | ||
| .list(); | ||
| LOGGER.info("Found pods in Spark namespace: {}", list); |
There was a problem hiding this comment.
This looks like debugging information
There was a problem hiding this comment.
This is a system test driver, I think it's important to track the progress of a test.
| JobList list = k8sProvider.getClient(properties).batch().v1().jobs() | ||
| .inNamespace(properties.get(BULK_IMPORT_EKS_NAMESPACE)) | ||
| .list(); | ||
| LOGGER.info("Found jobs in Spark namespace: {}", list); |
There was a problem hiding this comment.
This looks like debugging information
There was a problem hiding this comment.
This is a system test driver, I think it's important to track the progress of a test.
| public static String generateToken(String clusterName, Region region, AwsCredentialsProvider credentialsProvider) { | ||
| SdkHttpRequest request = SdkHttpRequest.builder() | ||
| .method(SdkHttpMethod.GET) | ||
| .uri(URI.create("https://sts." + region.id() + ".amazonaws.com/")) |
There was a problem hiding this comment.
Hardcoded DNS suffix will cause issues in a different AWS partition. Please can this be configured as in other places in codebase!
m09526
removed
the
needs-reviewer
rtjd6554
force-pushed
the
7239-check-k8s-pods-deleted
branch
from
678cd43 to
ac15338
Compare
3 participants
Make sure you have checked all steps below.
Issue
Feature". Note that before an issue is finished, you can still make a pull request by raising a separate issue
for your progress.
Tests
Documentation
separate issue for that below.