Skip to content

Auto-discovery fails in US2 for PolicyAwarenessService #845

Open
Open
Auto-discovery fails in US2 for PolicyAwarenessService#845
Labels
bugSomething isn't working
@kotieyal

Description

@kotieyal
kotieyal
opened on Apr 30, 2026

Describe the Problem/Bug

Tenant Id: 41a4899d-4ae7-431b-87cb-bb5dfb913a88

Module: Deployment

Landscape: Prod-US

Execution Id:

Deployment Id:

Resource Group: 

Description:
The auto discovery of model is failing in US2.
We are using the SAP AI SDK for Java to auto-discover orchestration deployments. Our code uses AiCoreService.withBaseDestination() to connect to the AI Core API and call .getInferenceDestination(resourceGroup).forScenario("orchestration") to automatically discover available orchestration deployments.

Auto-discovery works successfully in EU2 and Integration environments.

The exception in US2 is :

ex-0000000014 recoverable I/O exception (org.bouncycastle.tls.TlsFatalAlert) caught when sending request to https://api.ai.prod.us-east-1.aws.ml.hana.ondemand.com:443;request will be automatically re-executed in 0 MILLISECONDS (exec count 2)
  AIOrchestratorService: AiCoreService-based client creation failed: Failed to load deployments for resource group default. Falling back to manual mode with configured deployment ID: d04d295d0342532b
  AIOrchestratorService: Creating OrchestrationClient with URL: https://api.ai.prod.us-east-1.aws.ml.hana.ondemand.com/v2/inference/deployments/d04d295d0342532b
  AIOrchestratorService: Configuring proxy: proxy.service.cnqr.tech:3128

Please guide us on this issue as it is causing issues with document upload in our Production

Steps to Reproduce

  1. Enter Policy Trip Tip: https://us2.concursolutions.com/nui/policyassistance/tips
  2. Go to a specific Expense Group
  3. Click on "View Tips" (Btn that generate tips from LLM)
  4. The process failure on legit document. the failure is on Content filtering.

Expected Behavior

The LLM request should work like it is woking in integration and EU2and return suggests tips based on LLM

Screenshots

No response

Used Versions

Java and Gradle version via ./gradlew --version:

Gradle 8.14

Build time: 2025-04-25 09:29:08 UTC
Revision: 34c560e3be961658a6fbcd7170ec2443a228b109

Kotlin: 2.0.21
Groovy: 3.0.24
Ant: Apache Ant(TM) version 1.10.15 compiled on August 25 2024
Launcher JVM: 17.0.18 (Homebrew 17.0.18+0)
Daemon JVM: /opt/homebrew/Cellar/openjdk@17/17.0.18/libexec/openjdk.jdk/Contents/Home (no JDK specified, using current Java home)
OS: Mac OS X 26.4.1 aarch64

SAP Cloud SDK for AI version: 1.15.0

  • Dependency: com.sap.ai.sdk:orchestration:1.15.0

Spring Boot version: 3.5.12 (configured in gradle.properties, resolved to 3.5.13 in dependencies)

  • Property: springBootVersion=3.5.12
  • Spring Framework: 6.2.15 / 6.2.17

Note: This is a Gradle project, not Maven. The project uses Gradle 8.14.

The complete dependency tree is available in the file I generated. The key dependencies include:

  • SAP AI SDK: com.sap.ai.sdk:orchestration:1.15.0
    • com.sap.ai.sdk:core:1.15.0
    • SAP Cloud SDK: 5.25.0
  • Spring Boot: 3.5.13
    • Spring Framework: 6.2.15 - 6.2.17
    • Spring Security: 6.5.9
    • Spring Data JPA: 3.5.10
    • Spring Cloud Config: 4.3.2
  • Concur MuService: 9.54.0
    • Authentication, API, Web, Exception modules
  • Netty: 4.2.12.Final
  • Jackson: 2.21.1 / 2.21.2
  • Hibernate: 6.6.45.Final
  • MySQL Connector: 9.3.0

Code Examples

https://github.concur.com/PAS/PolicyAwarenessService

Stack Trace

https://lp-search.us2.concur.global/app/r/s/SXODo

This is link to Kibana for a single request.

Log File

those are all the errors:

https://lp-search.us2.concur.global/app/r/s/6mdpq

// Your code here

{
  "_index": "comm2:log-2-6w-003246",
  "_id": "VYHs3p0B2lVherMKbm55",
  "_version": 1,
  "_size": 1607,
  "_source": {
    "company_uuid": "321a748e-9e7b-4487-a198-19c7ca0f28c7",
    "kubernetes": {
      "cluster": "pp-hh48glg488gu",
      "container_name": "policyawarenessservice",
      "pod": "policyawarenessservice-58c6bb8f49-gc8gz",
      "namespace": "pas",
      "account": "spend",
      "labels": {
        "version": "1.0.0"
      }
    },
    "server": "10.22.221.157",
    "description": "AI Response has failed Llama Guard content filter with error code : 500 and error message : Request failed with status 404 (Not Found): The response Content-Type is not JSON for PolicyDocId : e1497246-3063-4b28-8553-124f090014a9",
    "@tsm": {
      "pp_ts": "2026-04-30T15:05:24.562223953Z",
      "es_ts": "2026-04-30T15:05:29.932662740Z",
      "lgs_ts": "2026-04-30T15:05:20.648658746Z",
      "es_d": 593754,
      "lgs_d": 584470
    },
    "user_uuid": "c4f974cb-a799-41b4-b9a9-1364f290b006",
    "endpoint": "/pas/v4/companies/321a748e-9e7b-4487-a198-19c7ca0f28c7/groups/0D637CA450A5485E9C79D2E1C42BBBB8/aigeneratedtips",
    "request_obj": {
      "route": "stable",
      "method": "GET",
      "query": "version=v1"
    },
    "@s3i": {
      "file_name": "k8s/es/2026/04/30/spend/ip-10-22-214-141.us-west-2.compute.internal/14/EGjZXtJ5.log"
    },
    "@by": [
      "s3i",
      "lp-comm2fro8-essli-04r2-i-0868d4e99b5a0efb6"
    ],
    "host": "ip-10-22-214-141.us-west-2.compute.internal",
    "client": "165.85.58.124",
    "roletype": "pas",
    "number_of_events": 1,
    "application_version": "CURRENT",
    "level": "ERROR",
    "thread": "XNIO-1 task-3",
    "TestRequest": "false",
    "tags": [
      "kubernetes",
      "containerd"
    ],
    "environment": "PROD",
    "@env": "awsus2",
    "@timestamp": "2026-04-30T14:55:36.178635071Z",
    "response_obj": {
      "status_code": 200
    },
    "application": "policyawarenessservice",
    "correlation_id": "5a8e6646-db5e-40fe-b456-40ddc757ae44",
    "request_id": "5a8e6646-db5e-40fe-b456-40ddc757ae44"
  },
  "fields": {
    "@tsm.es_ts": [
      "2026-04-30T15:05:29.932Z"
    ],
    "company_uuid": [
      "321a748e-9e7b-4487-a198-19c7ca0f28c7"
    ],
    "server": [
      "10.22.221.157"
    ],
    "kubernetes.container_name.analyzed": [
      "policyawarenessservice"
    ],
    "kubernetes.pod": [
      "policyawarenessservice-58c6bb8f49-gc8gz"
    ],
    "response_obj.status_code": [
      200
    ],
    "description": [
      "AI Response has failed Llama Guard content filter with error code : 500 and error message : Request failed with status 404 (Not Found): The response Content-Type is not JSON for PolicyDocId : e1497246-3063-4b28-8553-124f090014a9"
    ],
    "kubernetes.account": [
      "spend"
    ],
    "request_obj.query": [
      "version=v1"
    ],
    "endpoint": [
      "/pas/v4/companies/321a748e-9e7b-4487-a198-19c7ca0f28c7/groups/0d637ca450a5485e9c79d2e1c42bbbb8/aigeneratedtips"
    ],
    "user_uuid": [
      "c4f974cb-a799-41b4-b9a9-1364f290b006"
    ],
    "@tsm.pp_ts": [
      "2026-04-30T15:05:24.562Z"
    ],
    "request_obj.route": [
      "stable"
    ],
    "server.analyzed": [
      "10.22.221.157"
    ],
    "level.analyzed": [
      "ERROR"
    ],
    "@by": [
      "s3i",
      "lp-comm2fro8-essli-04r2-i-0868d4e99b5a0efb6"
    ],
    "kubernetes.namespace": [
      "pas"
    ],
    "host": [
      "ip-10-22-214-141.us-west-2.compute.internal"
    ],
    "client": [
      "165.85.58.124"
    ],
    "roletype": [
      "pas"
    ],
    "client.analyzed": [
      "165.85.58.124"
    ],
    "number_of_events": [
      1
    ],
    "application_version": [
      "current"
    ],
    "host.analyzed": [
      "ip-10-22-214-141.us-west-2.compute.internal"
    ],
    "request_obj.method": [
      "get"
    ],
    "level": [
      "error"
    ],
    "kubernetes.cluster": [
      "pp-hh48glg488gu"
    ],
    "kubernetes.container_name": [
      "policyawarenessservice"
    ],
    "kubernetes.namespace.analyzed": [
      "pas"
    ],
    "thread": [
      "xnio-1 task-3"
    ],
    "tags": [
      "kubernetes",
      "containerd"
    ],
    "TestRequest": [
      "false"
    ],
    "environment": [
      "prod"
    ],
    "@env": [
      "awsus2"
    ],
    "@timestamp": [
      "2026-04-30T14:55:36.178635071Z"
    ],
    "@tsm.lgs_d": [
      584470
    ],
    "@s3i.file_name": [
      "k8s/es/2026/04/30/spend/ip-10-22-214-141.us-west-2.compute.internal/14/EGjZXtJ5.log"
    ],
    "application": [
      "policyawarenessservice"
    ],
    "endpoint.analyzed": [
      "/pas/v4/companies/321a748e-9e7b-4487-a198-19c7ca0f28c7/groups/0D637CA450A5485E9C79D2E1C42BBBB8/aigeneratedtips"
    ],
    "application.analyzed": [
      "policyawarenessservice"
    ],
    "correlation_id": [
      "5a8e6646-db5e-40fe-b456-40ddc757ae44"
    ],
    "@tsm.es_d": [
      593754
    ],
    "@tsm.lgs_ts": [
      "2026-04-30T15:05:20.648Z"
    ],
    "request_id": [
      "5a8e6646-db5e-40fe-b456-40ddc757ae44"
    ],
    "kubernetes.labels.version": [
      "1.0.0"
    ]
  }
}

Affected Development Phase

Getting Started

Impact

No Impact

Timeline

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions