helm chart v2 cannot connect to external minio #84284

bWorksApp · 2026-08-12T03:52:03Z

bWorksApp
Aug 12, 2026

Hi all,
I'm trying to deploy airbyte v2 by helm chart on K8S
helm -n staging-infra upgrade --install airbyte airbyte/airbyte --version 2.1.1 -f airbyte-values.yaml
this is my values yaml

cat airbyte-values.yaml
# cài airbyte với postgres external
postgresql:
  enabled: false

minio:
  enabled: false

global:

  extraEnv:
    - name: AWS_ENDPOINT_URL_S3
      value: "http://minio.staging-infra:9000"
    - name: S3_PATH_STYLE_ACCESS
      value: "true"
    - name: AWS_ENDPOINT_URL
      value: "http://minio:9000"
    - name: AWS_S3_ENDPOINT          # ⬅️ Thêm
      value: "http://minio:9000"
    - name: S3_ENDPOINT              # ⬅️ Thêm
      value: "http://minio:9000"

  # url của airbyte , trùng với ingress bên dưới
  airbyteUrl: "https://xxx"

  # Khi kích sync, airbyte-workload-launcher sẽ tạo một Pod (Job) tạm , chạy connector, sau đó pod tự động xóa
  workloads:
    namespace: "staging-infra"   # Namespace chứa các pod sync tạm

  database:
#    type: internal
    type: external                       # Báo cho Airbyte dùng DB external
    host: "10.20.31.4"         # 🔁 Thay bằng host DB của bạn
    port: "5432"                         # 🔁 Thay bằng port DB của bạn
    name: "airbyte"                  # 🔁 Tên database
    user: "airbyte"                         # khi airbyte init cần user có quyền trên schema public
    password: "xxx"

  storage:
    secretName: ""
    type: s3                          # Dùng MinIO (hoặc s3 nếu tương thích S3)
    bucket:
      log: airbyte
      state: airbyte
      workloadOutput: airbyte
      auditLogging: airbyte
      activityPayload: airbyte
    s3:
      region: "us-east-1"
      endpoint: "http://minio:9000" # minio đã được deploy riêng bằng helm chart khác , cùng namespace với airbyte
      forcePathStyle: true
      authenticationType: "credentials"
      accessKeyId: "xxx"            # 🔁 Access Key
      secretAccessKey: "xxx"            # 🔁 Secret Key

# Cấu hình Ingress (nếu muốn expose ra ngoài)
ingress:
  enabled: true
  className: "nginx"
  hosts:
    - host: xxx         # 🔁 Domain của bạn
      paths:
        - path: /
          pathType: Prefix
          backend: server
  tls:
    - secretName: xxx
      hosts:
        - xxx

temporal:
  extraEnv:
    - name: SQL_TLS_ENABLED
      value: "false"
    - name: POSTGRES_TLS_ENABLED
      value: "false"
    - name: SKIP_DB_CREATE
      value: "true"
  database:
    engine: postgres12
    host: "10.20.31.4"
    port: "5432"
    user: "airbyte"
    password: "xxx"

# ⭐ ĐẢM BẢO SERVER NHẬN ĐÚNG BIẾN (override nếu cần)
server:
  #env:
  extraEnv:
    - name: AWS_ENDPOINT_URL_S3
      value: "http://minio.staging-infra:9000"
    - name: S3_PATH_STYLE_ACCESS
      value: "true"
    - name: AWS_ENDPOINT_URL
      value: "http://minio:9000"
    - name: AWS_S3_ENDPOINT          # ⬅️ Thêm
      value: "http://minio:9000"
    - name: S3_ENDPOINT              # ⬅️ Thêm
      value: "http://minio:9000"

workload-launcher:
  #env:
  extraEnv:
    - name: AWS_ENDPOINT_URL_S3
      value: "http://minio.staging-infra:9000"
    - name: S3_PATH_STYLE_ACCESS
      value: "true"
    - name: AWS_ENDPOINT_URL
      value: "http://minio:9000"
    - name: AWS_S3_ENDPOINT          # ⬅️ Thêm
      value: "http://minio:9000"
    - name: S3_ENDPOINT              # ⬅️ Thêm
      value: "http://minio:9000"

When I access to airbyte UI add connector to sharepoint I get error "Failed to save Microsoft SharePoint due to the following error: Unable to persist the job output, check the document store credentials."

this is log of airbyte server pod

2026-08-12T03:32:25.728592492Z default-eventLoopGroup-2-6 ERROR Log4j API could not find a logging provider.
2026-08-12 03:32:27,084 [io-executor-thread-14] WARN    i.a.c.e.EntitlementServiceImpl(hasEnterpriseConnectorEntitlements-Y0BDzM4):276 - Connector entitlement not available. actorDefinitionId=8d22fb25-a6e8-40e5-9a8b-0e057cc0bb86 organizationId=OrganizationId(value=00000000-0000-0000-0000-000000000000)
2026-08-12 03:32:27,169 [io-executor-thread-12] INFO    i.a.c.s.RemoteDefinitionsProvider(getRemoteConnectorRegistry):205 - Fetched latest remote definitions (-1058305454)
2026-08-12 03:32:28,045 [io-executor-thread-16] INFO    i.a.c.s.RemoteDefinitionsProvider(fetchDocFromUrl):294 - Successfully fetched live documentation from https://raw.githubusercontent.com/airbytehq/airbyte/master/docs/integrations/sources/microsoft-sharepoint.md
2026-08-12 03:35:03,093 [io-executor-thread-38] ERROR   i.a.c.s.e.h.UncaughtExceptionHandler(handle):37 - Uncaught exception [errorId: bf94a920-aa89-454b-9d96-66f316c41542]: class KnownExceptionInfo {
    message: Internal Server Error: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: Q11VR7M1DNJ53WSH, Extended Request ID: Cw5dR/lqWNGQ7tY56fj9118ZQkwBP7ns1VPLrk5/gwSxfNK09cmDIS5CLKnFbXJ2EWOBKx5sQF15S9avomR/6xaXse3G0rth) (SDK Attempt Count: 1)
    exceptionClassName: null
    exceptionStack: [software.amazon.awssdk.services.s3.model.S3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: Q11VR7M1DNJ53WSH, Extended Request ID: Cw5dR/lqWNGQ7tY56fj9118ZQkwBP7ns1VPLrk5/gwSxfNK09cmDIS5CLKnFbXJ2EWOBKx5sQF15S9avomR/6xaXse3G0rth) (SDK Attempt Count: 1),       at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:104),        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:58),         at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168),     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73),    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36),    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53),  at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82),         at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50),        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32),       at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26),  at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173),        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80),  at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182),    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74),   at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45),       at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53),    at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:10191),    at io.airbyte.commons.storage.AbstractS3StorageClient.write(StorageClient.kt:466),      at io.airbyte.server.apis.controllers.WorkloadOutputController.writeWorkloadOutput(WorkloadOutputController.kt:54),     at io.airbyte.server.apis.controllers.$WorkloadOutputController$Definition$Exec.dispatch(Unknown Source),       at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461),      at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4454),         at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272),     at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38),  at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:466),   at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:443),   at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92),     at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232),   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144),   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642),   at java.base/java.lang.Thread.run(Thread.java:1583)]
    rootCauseExceptionClassName: null
    rootCauseExceptionStack: []
    details: null
    errorId: null
}
2026-08-12 03:35:06,345 [io-executor-thread-40] ERROR   i.a.c.s.e.h.UncaughtExceptionHandler(handle):37 - Uncaught exception [errorId: 2bfc06e3-7937-42c0-97cd-96fce2ca5d3f]: class KnownExceptionInfo {
    message: Internal Server Error: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: SWNGJJEAZ1WJZ909, Extended Request ID: KCsbfoPRIZamZ7gVZSJ2JfQ8qKQgr41kT/93BeyDTXmOtB5ZMmFN4Brr/hquBFCK0QDBGvzca9R3l0J8lUmE/KlPLWEyv0k0) (SDK Attempt Count: 1)
    exceptionClassName: null
    exceptionStack: [software.amazon.awssdk.services.s3.model.S3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: SWNGJJEAZ1WJZ909, Extended Request ID: KCsbfoPRIZamZ7gVZSJ2JfQ8qKQgr41kT/93BeyDTXmOtB5ZMmFN4Brr/hquBFCK0QDBGvzca9R3l0J8lUmE/KlPLWEyv0k0) (SDK Attempt Count: 1),       at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:104),        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:58),         at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168),     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73),    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36),    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53),  at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82),         at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50),        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32),       at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26),  at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173),        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80),  at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182),    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74),   at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45),       at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53),    at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:10191),    at io.airbyte.commons.storage.AbstractS3StorageClient.write(StorageClient.kt:466),      at io.airbyte.server.apis.controllers.WorkloadOutputController.writeWorkloadOutput(WorkloadOutputController.kt:54),     at io.airbyte.server.apis.controllers.$WorkloadOutputController$Definition$Exec.dispatch(Unknown Source),       at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461),      at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4454),         at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272),     at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38),  at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:466),   at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:443),   at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92),     at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232),   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144),   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642),   at java.base/java.lang.Thread.run(Thread.java:1583)]
    rootCauseExceptionClassName: null
    rootCauseExceptionStack: []
    details: null
    errorId: null
}
2026-08-12 03:35:09,406 [io-executor-thread-43] ERROR   i.a.c.s.e.h.UncaughtExceptionHandler(handle):37 - Uncaught exception [errorId: 9c89aeb4-b7a2-40cc-9391-0f565ff4e073]: class KnownExceptionInfo {
    message: Internal Server Error: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: Y2KPP0XAE0PV701V, Extended Request ID: WfmkHLlmSeXnzy3IV30TrZ4t1GGKrH9rI3ZspSzr6ATWmqHF86rkrcxJQpesr0RH4XIbKQvXW4d2XEdzmxtK3cJiclNbDzSg) (SDK Attempt Count: 1)
    exceptionClassName: null
    exceptionStack: [software.amazon.awssdk.services.s3.model.S3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: Y2KPP0XAE0PV701V, Extended Request ID: WfmkHLlmSeXnzy3IV30TrZ4t1GGKrH9rI3ZspSzr6ATWmqHF86rkrcxJQpesr0RH4XIbKQvXW4d2XEdzmxtK3cJiclNbDzSg) (SDK Attempt Count: 1),       at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:104),        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:58),         at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168),     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73),    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36),    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53),  at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82),         at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50),        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32),       at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26),  at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173),        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80),  at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182),    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74),   at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45),       at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53),    at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:10191),    at io.airbyte.commons.storage.AbstractS3StorageClient.write(StorageClient.kt:466),      at io.airbyte.server.apis.controllers.WorkloadOutputController.writeWorkloadOutput(WorkloadOutputController.kt:54),     at io.airbyte.server.apis.controllers.$WorkloadOutputController$Definition$Exec.dispatch(Unknown Source),       at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461),      at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4454),         at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272),     at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38),  at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:466),   at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:443),   at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92),     at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232),   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144),   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642),   at java.base/java.lang.Thread.run(Thread.java:1583)]
    rootCauseExceptionClassName: null
    rootCauseExceptionStack: []
    details: null
    errorId: null
}
2026-08-12 03:35:12,440 [io-executor-thread-46] ERROR   i.a.c.s.e.h.UncaughtExceptionHandler(handle):37 - Uncaught exception [errorId: b0abb78d-8c35-4909-9d8d-a2f42344464a]: class KnownExceptionInfo {
    message: Internal Server Error: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: HPAX0TQASA5VDYCP, Extended Request ID: VNG2EoX0LsHFUc5iR2MLLibaQEjoPbafq/1XmDf0cmTlpE4kXu9tkZE7Sv/1Tko8UMYLy44Ss89IjDv+kWzuYXmEecFpkXAB) (SDK Attempt Count: 1)
    exceptionClassName: null
    exceptionStack: [software.amazon.awssdk.services.s3.model.S3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: HPAX0TQASA5VDYCP, Extended Request ID: VNG2EoX0LsHFUc5iR2MLLibaQEjoPbafq/1XmDf0cmTlpE4kXu9tkZE7Sv/1Tko8UMYLy44Ss89IjDv+kWzuYXmEecFpkXAB) (SDK Attempt Count: 1),       at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:104),        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:58),         at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168),     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73),    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36),    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53),  at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82),         at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50),        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32),       at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26),  at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173),        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80),  at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182),    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74),   at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45),       at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53),    at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:10191),    at io.airbyte.commons.storage.AbstractS3StorageClient.write(StorageClient.kt:466),      at io.airbyte.server.apis.controllers.WorkloadOutputController.writeWorkloadOutput(WorkloadOutputController.kt:54),     at io.airbyte.server.apis.controllers.$WorkloadOutputController$Definition$Exec.dispatch(Unknown Source),       at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461),      at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4454),         at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272),     at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38),  at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:466),   at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:443),   at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92),     at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232),   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144),   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642),   at java.base/java.lang.Thread.run(Thread.java:1583)]
    rootCauseExceptionClassName: null
    rootCauseExceptionStack: []
    details: null
    errorId: null
}
2026-08-12 03:35:15,558 [io-executor-thread-48] ERROR   i.a.c.s.e.h.UncaughtExceptionHandler(handle):37 - Uncaught exception [errorId: f0b2227a-9d46-43e4-a7e6-d0a833067477]: class KnownExceptionInfo {
    message: Internal Server Error: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: 4E5QA8BGZ02Q087F, Extended Request ID: OoSgs5yTWzkuR0aRJs3ZicOd5XyV1EDuYJaU1clbrHX2aFkxrbxzD5D13dRwql9yw5PHsK1J0j11qGJTxX0IKdg4OTj3yg17) (SDK Attempt Count: 1)
    exceptionClassName: null
    exceptionStack: [software.amazon.awssdk.services.s3.model.S3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: 4E5QA8BGZ02Q087F, Extended Request ID: OoSgs5yTWzkuR0aRJs3ZicOd5XyV1EDuYJaU1clbrHX2aFkxrbxzD5D13dRwql9yw5PHsK1J0j11qGJTxX0IKdg4OTj3yg17) (SDK Attempt Count: 1),       at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:104),        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:58),         at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168),     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73),    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36),    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53),  at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82),         at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50),        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32),       at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26),  at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173),        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80),  at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182),    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74),   at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45),       at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53),    at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:10191),    at io.airbyte.commons.storage.AbstractS3StorageClient.write(StorageClient.kt:466),      at io.airbyte.server.apis.controllers.WorkloadOutputController.writeWorkloadOutput(WorkloadOutputController.kt:54),     at io.airbyte.server.apis.controllers.$WorkloadOutputController$Definition$Exec.dispatch(Unknown Source),       at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461),      at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4454),         at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272),     at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38),  at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:466),   at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:443),   at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92),     at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232),   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144),   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642),   at java.base/java.lang.Thread.run(Thread.java:1583)]
    rootCauseExceptionClassName: null
    rootCauseExceptionStack: []
    details: null
    errorId: null
}
2026-08-12 03:35:18,503 [io-executor-thread-50] ERROR   i.a.c.s.e.h.UncaughtExceptionHandler(handle):37 - Uncaught exception [errorId: d351e437-2e87-453f-8043-6c297a89ddad]: class KnownExceptionInfo {
    message: Internal Server Error: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: PMRJGZABPDY2WKX4, Extended Request ID: AfKteXBuJ/UxAjnddmGW6K5rPXvyqvzomDgoAfZWqEyDOryRkorgOT/4oh5PmbIT+BAiaGm7JH3JI7SsjwXGUBiCiJLAgpRz) (SDK Attempt Count: 1)
    exceptionClassName: null
    exceptionStack: [software.amazon.awssdk.services.s3.model.S3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: PMRJGZABPDY2WKX4, Extended Request ID: AfKteXBuJ/UxAjnddmGW6K5rPXvyqvzomDgoAfZWqEyDOryRkorgOT/4oh5PmbIT+BAiaGm7JH3JI7SsjwXGUBiCiJLAgpRz) (SDK Attempt Count: 1),       at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:104),        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:58),         at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168),     at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73),    at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36),    at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53),  at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82),         at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50),        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32),       at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),     at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206),    at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37),  at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26),  at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103),   at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173),        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80),  at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182),    at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74),   at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45),       at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53),    at software.amazon.awssdk.services.s3.DefaultS3Client.putObject(DefaultS3Client.java:10191),    at io.airbyte.commons.storage.AbstractS3StorageClient.write(StorageClient.kt:466),      at io.airbyte.server.apis.controllers.WorkloadOutputController.writeWorkloadOutput(WorkloadOutputController.kt:54),     at io.airbyte.server.apis.controllers.$WorkloadOutputController$Definition$Exec.dispatch(Unknown Source),       at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461),      at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4454),         at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272),     at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38),  at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:466),   at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:443),   at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92),     at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232),   at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144),   at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642),   at java.base/java.lang.Thread.run(Thread.java:1583)]
    rootCauseExceptionClassName: null
    rootCauseExceptionStack: []
    details: null
    errorId: null
}
2026-08-12 03:35:20,620 [io-executor-thread-52] WARN    i.a.s.s.CommandService(getJobLogs):812 - Unable to find logs for command:a07bcc67-dae1-434d-91d2-e7788bf15037 workloadId:59353119-f0f2-4e5a-a8ba-15d887bc34f6_44db5af7-ae89-45ee-b9ed-d538c1fedbbc_0_check
software.amazon.awssdk.services.s3.model.S3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: M7REWTKXK8B69FKX, Extended Request ID: 4+RK6AqAQSfRgD+2syzRvTHGUbVf36n8IyqEPRtnlLtRV86nmXOaNRYmEM6kDIJfa1+jLUoCaEWCCx8Uh0nQNWMpuXFL5TyC) (SDK Attempt Count: 1)
        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:104)
        at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:58)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.RetryableStageHelper.retryPolicyDisallowedRetryException(RetryableStageHelper.java:168)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:73)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:36)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:53)
        at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:35)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:82)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:62)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:43)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:50)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:32)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
        at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
        at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182)
        at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74)
        at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
        at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53)
        at software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:7327)
        at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:154)
        at software.amazon.awssdk.services.s3.paginators.ListObjectsV2Iterable$ListObjectsV2ResponseFetcher.nextPage(ListObjectsV2Iterable.java:145)
        at software.amazon.awssdk.core.pagination.sync.PaginatedResponsesIterator.next(PaginatedResponsesIterator.java:58)
        at io.airbyte.commons.storage.AbstractS3StorageClient.list(StorageClient.kt:602)
        at io.airbyte.commons.logging.LogClient.getLogs(LogClient.kt:63)
        at io.airbyte.commons.logging.LogClientManager.getLogs(LogClientManager.kt:48)
        at io.airbyte.server.services.CommandService.getJobLogs(CommandService.kt:804)
        at io.airbyte.server.services.CommandService.getCheckJobOutput(CommandService.kt:835)
        at io.airbyte.server.apis.controllers.CommandApiController.getCheckCommandOutput(CommandApiController.kt:105)
        at io.airbyte.server.apis.controllers.$CommandApiController$Definition$Exec.dispatch(Unknown Source)
        at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invokeUnsafe(AbstractExecutableMethodsDefinition.java:461)
        at io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle.invokeUnsafe(DefaultBeanContext.java:4454)
        at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:272)
        at io.micronaut.web.router.DefaultUriRouteMatch.execute(DefaultUriRouteMatch.java:38)
        at io.micronaut.http.server.RouteExecutor.executeRouteAndConvertBody(RouteExecutor.java:466)
        at io.micronaut.http.server.RouteExecutor.lambda$callRoute$5(RouteExecutor.java:443)
        at io.micronaut.core.execution.ExecutionFlow.lambda$async$0(ExecutionFlow.java:92)
        at io.micronaut.core.propagation.PropagatedContext.lambda$wrap$3(PropagatedContext.java:232)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

airbyte server pod tried to connect to aws s3.
How can I fix this issue , please give me some advice , thank you very much.

Airbyte Support Bot (airbyte-support-bot) · 2026-08-12T03:53:35Z

Airbyte Support Bot (airbyte-support-bot)
Aug 12, 2026
Maintainer

Hi bWorksApp, thank you for the detailed write-up — the values file and server log make this much easier to look at.

The 403 InvalidAccessKeyId in the stack trace comes back from whatever S3-compatible endpoint the server actually reached when writing workload output, so the first thing worth narrowing down is whether the request went to your MinIO or somewhere else, and whether the key pair the pods received matches the one MinIO knows.

Things to check

Endpoint consistency. Your values mix two hostnames: http://minio:9000 (under global.storage.s3.endpoint and most extraEnv entries) and http://minio.staging-infra:9000 (under AWS_ENDPOINT_URL_S3). If the MinIO service name in that namespace is not literally minio, one of those will not resolve. Using the fully qualified name for both, e.g. http://<minio-service>.<namespace>.svc.cluster.local:9000, removes the ambiguity.
Drop the manual extraEnv overrides first. Setting AWS_ENDPOINT_URL, AWS_S3_ENDPOINT, S3_ENDPOINT, and AWS_ENDPOINT_URL_S3 by hand alongside global.storage.s3.* can end up fighting with what the chart renders. Try a run with only the global.storage block configured, and confirm from the rendered manifests (helm template ... | grep -i endpoint) what the server container actually receives.
Provide the credentials via a Kubernetes secret. In chart v2 the documented pattern is a secret referenced by global.storage.secretName with accessKeyIdSecretKey / secretAccessKeySecretKey, rather than an empty secretName: "" with inline values. See Configuring external storage. After deploying, check that the value actually landed in the pod (compare only the length or first characters — please don't paste the key itself).
Verify the key pair against MinIO directly. From a debug pod in the same namespace, run an mc alias set / mc ls against the same endpoint with the same access key. If that also returns InvalidAccessKeyId, the key doesn't exist in MinIO (a common cause is MinIO having been re-deployed with a fresh root credential, or the key belonging to a different MinIO instance).
Bucket existence. The buckets in your values (airbyte for log/state/workloadOutput/activityPayload) must already exist in MinIO — the chart does not create them for an external instance.

If those checks don't resolve it, could you share:

The output of helm template (or kubectl get deploy airbyte-server -o yaml) limited to the storage-related environment variables, with all secret values removed;
Whether an mc client using the same credentials and endpoint from inside the cluster succeeds;
The exact chart version and whether this deployment was upgraded from a v1 chart or installed fresh.

Please make sure to mask or remove any sensitive information (access keys, secret keys, passwords, connection strings, hostnames you'd rather not publish) before sharing any output.

Need more help? Join Airbyte Community Slack for peer support, or if you're a Cloud customer, open a support ticket referencing this URL.

0 replies

bWorksApp · 2026-08-12T08:19:48Z

bWorksApp
Aug 12, 2026
Author

I fixed this issue , this is my values yaml

# use postgresql external
postgresql:
  enabled: false
# use minio external
minio:
  enabled: false

global:
  airbyteUrl: "https://xxx"

  workloads:
    namespace: "staging-infra"

  database:
    type: external
    host: "10.20.31.4"
    port: "5432"
    name: "airbyte"
    user: "airbyte"
    password: "xxx"

  storage:
    type: minio
    bucket:
      log: airbyte
      state: airbyte
      workloadOutput: airbyte
      auditLogging: airbyte
      activityPayload: airbyte
    minio:
      region: "us-east-1"
      endpoint: "http://minio:9000"
      forcePathStyle: true
      authenticationType: "credentials"
      accessKeyId: "xxx"
      secretAccessKey: "xxx"

  extraEnv:
    - name: AWS_ACCESS_KEY_ID
      value: "xxx"
    - name: AWS_SECRET_ACCESS_KEY
      value: "xxx"

    - name: AWS_ENDPOINT_URL_S3
      value: "http://minio:9000"
    - name: AWS_ENDPOINT_URL
      value: "http://minio:9000"
    - name: AWS_S3_ENDPOINT
      value: "http://minio:9000"
    - name: S3_ENDPOINT
      value: "http://minio:9000"

    - name: S3_PATH_STYLE_ACCESS
      value: "true"

    - name: AWS_EC2_METADATA_DISABLED
      value: "true"

    - name: STORAGE_TYPE
      value: "minio"

ingress:
  enabled: true
  className: "nginx"
  hosts:
    - host: xxx
      paths:
        - path: /
          pathType: Prefix
          backend: server
  tls:
    - secretName: xxx
      hosts:
        - xxx

temporal:
  extraEnv:
    - name: SQL_TLS_ENABLED
      value: "false"
    - name: POSTGRES_TLS_ENABLED
      value: "false"
    - name: SKIP_DB_CREATE
      value: "true"
  database:
    engine: postgres12
    host: "10.20.31.4"
    port: "5432"
    user: "airbyte"
    password: "xxx"

Please add more example to documentation

0 replies

Navigation Menu

Repository navigation

helm chart v2 cannot connect to external minio #84284

Uh oh!

{{title}}

Uh oh!

Replies: 2 comments

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

helm chart v2 cannot connect to external minio #84284

Uh oh!

bWorksApp Aug 12, 2026

Replies: 2 comments

Uh oh!

Airbyte Support Bot (airbyte-support-bot) Aug 12, 2026 Maintainer

Uh oh!

bWorksApp Aug 12, 2026 Author

bWorksApp
Aug 12, 2026

Airbyte Support Bot (airbyte-support-bot)
Aug 12, 2026
Maintainer

bWorksApp
Aug 12, 2026
Author