VertexAiRagRetrieval fails in google cloud run

[bijeshcnair]

Problem Description:
I am encountering a persistent UNIMPLEMENTED error when my Google ADK Java agent, deployed on Cloud Run, attempts to query a Vertex AI RAG Corpus using VertexRagServiceClient.retrieveContexts. The full error from Cloud Run logs is:

com.google.api.gax.rpc.UnimplementedException: io.grpc.StatusRuntimeException: UNIMPLEMENTED: Operation is not implemented, or supported, or enabled.
    at com.google.adk.tools.retrieval.VertexAiRagRetrieval.lambda$runAsync$0(VertexAiRagRetrieval.java:142)
    at io.reactivex.rxjava3.internal.operators.single.SingleFromCallable.subscribeActual(SingleFromCallable.java:43)
    ... (rest of stack trace relevant to the VertexRagServiceClient call)

This error occurs specifically when the VertexAiRagRetrieval.runAsync method is called to fetch context for a given query (e.g., "How to reset my password?").

Expected Behavior:
The ADK agent should successfully query the Vertex AI RAG Corpus and retrieve relevant documentation, similar to its behavior when running locally.

Steps to Reproduce (High-Level):

A Google ADK Java agent (using google-adk-dev version 0.2.0 and google-cloud-aiplatform client library 3.57.0) is deployed on Google Cloud Run. The agent includes a VertexAiRagRetrieval tool configured to query a specific Vertex AI RAG Corpus.

A user interacts with the agent (e.g., via the ADK Web UI) with a query that triggers the VertexAiRagRetrieval tool.

The UNIMPLEMENTED error is observed in the Cloud Run logs for the agent service.

Environment Details:

Google Cloud Project ID: my-play-ground

Google ADK Java Version: 0.2.0 (from google-adk-dev Maven dependency)

google-cloud-aiplatform Java Client Version: 3.57.0 (from pom.xml)

Cloud Run Service Name: my-support-assistant

Cloud Run Service Account: adk-agent-identity@my-play-ground.iam.gserviceaccount.com

Cloud Run Region: europe-west4 (also tested in us-central1 with same results)

Vertex AI RAG Corpus Full Resource Name: projects/my-play-ground/locations/europe-west4/ragCorpora/ (Replace with your actual corpus ID)

Java Version (in container): E.g., openjdk:17-slim-buster

Troubleshooting Attempts & Observations (Crucial Details):

Local Execution: The exact same Java ADK agent code, running locally on my machine, successfully queries the RAG Corpus without any UNIMPLEMENTED errors. This strongly rules out general code bugs, client-side dependency conflicts, or a malformed RAG Corpus ID/request.

Vertex AI API Enablement: The "Vertex AI API" is confirmed to be enabled in the my-play-ground project.

Vertex AI RAG Feature Support: Vertex AI RAG is officially supported in both europe-west4 and us-central1 regions as per Google Cloud documentation.

Service Account Permissions (On Agent's SA):

The adk-agent-identity service account has been granted the Vertex AI User role (roles/aiplatform.user) at the project level.

Additionally, the Vertex AI Reader role (roles/aiplatform.viewer) has been explicitly granted to adk-agent-identity specifically on the RAG Corpus resource (projects/my-play-ground/locations/europe-west4/ragCorpora/) using:

gcloud projects add-iam-policy-binding my-play-ground \
    --member="serviceAccount:adk-agent-identity@my-play-ground.iam.gserviceaccount.com" \
    --role="roles/aiplatform.viewer" \
    --condition="expression=resource.name == 'projects/my-play-ground/locations/europe-west4/ragCorpora/<corupuraid>',title=rag-corpus-viewer-access,description=Allow adk-agent-identity to read specific RAG corpus" \
    --project=my-play-ground

Despite these comprehensive permissions, the UNIMPLEMENTED error persists on Cloud Run.

ADK Java Code

import com.google.cloud.aiplatform.v1.RetrieveContextsRequest.VertexRagStore.RagResource;
import com.google.cloud.aiplatform.v1.VertexRagServiceClient;
import java.io.IOException;
import java.util.Collections; // For ImmutableList if you're not using Guava

// Assuming you have your VertexAiRagRetrieval class and other imports in place

public class CreateVertexAiRagRetrievalInstance {

    public static void main(String[] args) {
        String projectId = "my-play-ground";
        String location = "europe-west4";
        String ragCorpusId = "2305843009213693952"; // The ID you provided

        // 1. Create an instance of VertexRagServiceClient
        VertexRagServiceClient vertexRagServiceClient = null;
        try {
            vertexRagServiceClient = VertexRagServiceClient.create(); // Uses Application Default Credentials
            System.out.println("VertexRagServiceClient created successfully.");

            // 2. Create a RagResource object
            RagResource ragResource = RagResource.newBuilder()
                    .setRagCorpus(
                        String.format("projects/%s/locations/%s/ragCorpora/%s", projectId, location, ragCorpusId)
                    )
                    // If you have specific RagFiles within the corpus, you can add them here:
                    // .addRagFiles("projects/P/locations/L/ragCorpora/C/ragFiles/F")
                    .build();

            // 3. Create the VertexAiRagRetrieval instance
            String toolName = "my-rag-tool";
            String toolDescription = "A tool to retrieve information from the my playground RAG corpus.";
            String parent = String.format("projects/%s/locations/%s", projectId, location); // Parent for the API call

            // You can pass a list of RagResources. For a single corpus, just put it in a list.
            java.util.List<RagResource> ragResourcesList = Collections.singletonList(ragResource);

            // Optional: Define a vector distance threshold if needed
            Double vectorDistanceThreshold = 0.5; // Example threshold, adjust as per your needs, or pass null

            VertexAiRagRetrieval ragRetrievalTool = new VertexAiRagRetrieval(
                toolName,
                toolDescription,
                vertexRagServiceClient,
                parent,
                ragResourcesList,
                vectorDistanceThreshold
            );

            System.out.println("VertexAiRagRetrieval instance created successfully: " + ragRetrievalTool.getName());

        } catch (IOException e) {
            System.err.println("Error creating VertexRagServiceClient or VertexAiRagRetrieval: " + e.getMessage());
            e.printStackTrace();
        } finally {
            if (vertexRagServiceClient != null) {
                vertexRagServiceClient.close(); // Important: Close the client when done
            }
        }
    }

}

Suspected Root Cause:
Given that all standard permissions and environmental checks pass, and local execution works, it seems like there might be a very specific internal configuration, a beta/preview feature enablement, or a subtle runtime environment difference within the Cloud Run context that causes the UNIMPLEMENTED error when the VertexRagServiceClient attempts to call retrieveContexts. This points to an issue on the service provider side related to how the RAG feature is provisioned or accessed in this specific deployment scenario.

error-log.txt

[bijeshcnair]

I've managed to resolve the issue(togehther with gemini cli, one heck of a tool) and wanted to share the solution for anyone who might encounter the same problem.

Problem Summary

My agent, using the VertexAiRagRetrieval tool from the ADK, was working perfectly in my local development environment. However, after deploying it to Google Cloud Run, it consistently failed with the
following error whenever the RAG tool was invoked:

1 com.google.api.gax.rpc.UnimplementedException: io.grpc.StatusRuntimeException: UNIMPLEMENTED: Operation is not implemented, or supported, or enabled.

This happened even after ensuring the Vertex AI API (aiplatform.googleapis.com) was enabled and that the service account had the correct IAM permissions (roles/aiplatform.user).

Root Cause

The problem was not with permissions or disabled APIs, but with how the VertexRagServiceClient was being initialized.

When running on Cloud Run, the default client created by VertexRagServiceClient.create() does not automatically infer the correct regional endpoint. It likely defaults to a global endpoint where the specific
RAG retrieval operation is not available, leading to the UNIMPLEMENTED error.

Solution

The fix is to explicitly configure the VertexRagServiceClient with the correct regional endpoint. Instead of creating the client with the default settings, you need to build a VertexRagServiceSettings object
that specifies the endpoint for your target region.

Here is the code change that resolved the issue:

Original Code (Fails on Cloud Run):

   1 import com.google.cloud.aiplatform.v1.VertexRagServiceClient;
   2 
   3 // ...
   4 
   5 // This initialization works locally but fails on Cloud Run
   6 VertexRagServiceClient vertexRagServiceClient = VertexRagServiceClient.create();

Corrected Code (Works on Cloud Run):

```

1 import com.google.cloud.aiplatform.v1.VertexRagServiceClient;
2 import com.google.cloud.aiplatform.v1.VertexRagServiceSettings;
3
4 // ...
5
6 // Define the location for your service
7 String location = "europe-west4"; // Your target region
8
9 // Construct the regional endpoint
10 String endpoint = String.format("%s-aiplatform.googleapis.com:443", location);
11
12 // Create service settings with the explicit endpoint
13 VertexRagServiceSettings vertexRagServiceSettings =
14 VertexRagServiceSettings.newBuilder().setEndpoint(endpoint).build();
15
16 // Create the client with the new settings
17 VertexRagServiceClient vertexRagServiceClient =
18 VertexRagServiceClient.create(vertexRagServiceSettings);

  By explicitly setting the endpoint, the client correctly targets the regional service where the RAG operation is implemented.


  I hope this helps others who run into this subtle but critical difference between local and Cloud Run environments.

[dd723]

Thank you, this was the exact answer i was looking for to resolve my issue.