Developer: add generative-ai-app-builder client REST api developer config - ideally in spring boot #562

obriensystems · 2023-10-16T15:11:40Z

see HttpClient in #645

https://github.com/ObrienlabsDev/rest-client-java/tree/main

Client request: generative-ai-app-bulder (Vertex AI search and conversation) REST API usage from containerized spring boot app
Review use of ADC with the client https://cloud.google.com/docs/authentication/provide-credentials-adc
ADC is used by https://github.com/cloud-quickstart/gcp-landing-zone/blob/main/gcp-landing-zone-deploy/src/main/java/zone/gcp/landing/Cli.java#L74

verify firewall as well - 443/8443 for https and https://www.gstatic.com/ipranges/goog.json for IPs

GoogleCredentials credential =
            GoogleCredentials.getApplicationDefault()
                .createScoped(Collections.singleton(IamScopes.CLOUD_PLATFORM));

Require both local dev (STS eclipse) and containerized auth/configs/API(open API/swagger) setup

https://cloud.google.com/generative-ai-app-builder/docs/locations#specify_a_multi-region_for_your_data_store

see #559

The text was updated successfully, but these errors were encountered:

obriensystems · 2023-10-16T15:41:05Z

Macbook pro 16 M1 running springsource tool suite 4.13.1 - JDK 17.0.5

Add stub endpoint to spring boot app
obrienlabs/magellan@a2d45f7

Follow
https://cloud.google.com/docs/authentication/provide-credentials-adc#local-key

Setup a service account on the genai project, set the GOOGLE_APPLICATION_CREDENTIALS env variable

or via gcloud https://cloud.google.com/iam/docs/keys-create-delete#creating

First reauthenticate gcloud locally

michaelobrien@mbp7 magellan % gcloud auth login

https://cloud.google.com/sdk/auth_success

michaelobrien@mbp7 magellan % gcloud config set project gen-ai-old
WARNING: Your active project does not match the quota project in your local Application Default Credentials file. This might result in unexpected quota issues.

To update your Application Default Credentials quota project, use the `gcloud auth application-default set-quota-project` command.
Updated property [core/project].

set quota project as i was using ADC https://cloud.google.com/docs/authentication/provide-credentials-adc

michaelobrien@mbp7 magellan % gcloud auth application-default set-quota-project gen-ai-old
API [cloudresourcemanager.googleapis.com] not enabled on project [gen-ai-old]. Would you like to enable and retry (this will take a few minutes)? (y/N)?  y

Enabling service [cloudresourcemanager.googleapis.com] on project [gen-ai-old]...
Operation "operations/acat.p2-310840400174-3ad62c19-2186-4147-9445-4ac0817a0483" finished successfully.

Credentials saved to file: [/Users/michaelobrien/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

Quota project "gen-ai-old" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.

To revert your CLI to the previously installed version, you may run:
  $ gcloud components update --version 419.0.0

michaelobrien@mbp7 magellan % gcloud components update

Create Service Account

https://cloud.google.com/iam/docs/service-accounts-create

michaelobrien@mbp7 magellan % gcloud iam service-accounts create gen-ai-old-sa --description="gen-ai-old-sa" --display-name="gen-ai-old-sa"                       
Created service account [gen-ai-old-sa].

Add roles to SA

gcloud projects add-iam-policy-binding gen-ai-old --member="serviceAccount:gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com" --role="roles/owner"

Create Service Account Key

https://cloud.google.com/iam/docs/keys-create-delete#creating

michaelobrien@mbp7 magellan % gcloud iam service-accounts keys create ~/keys/gcp-gen-ai-old-sa.json --iam-account=gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com
created key [02f9b5c04a76987e10f1e67467b326be53122c7e] of type [json] as [/Users/michaelobrien/keys/gcp-gen-ai-old-sa.json] for [gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com]


michaelobrien@mbp7 magellan % cat ~/keys/gcp-gen-ai-old-sa.json 
{
  "type": "service_account",
  "project_id": "gen-ai-old",
  "private_key_id": "02f9b.....2c7e",
  "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkq.....YFedw=\n-----END PRIVATE KEY-----\n",
  "client_email": "gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com",
  "client_id": "108...90",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/gen-ai-old-sa%40gen-ai-old.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

Export key

michaelobrien@mbp7 magellan % export GOOGLE_APPLICATION_CREDENTIALS="~/keys/gcp-gen-ai-old-sa.json"

Add service account impersonation

https://cloud.google.com/docs/authentication/use-service-account-impersonation

IAM is enabled on the project

serviceAccountTokenCreator is on your user roles

suspect I need to add to the SA list - my user - I do
https://console.cloud.google.com/iam-admin/serviceaccounts/details/108536977121828646390/permissions?project=gen-ai-old

check access for user

michaelobrien@mbp7 magellan % gcloud storage buckets list

creation_time: 2023-10-16T16:02:24+0000
default_storage_class: STANDARD
location: US
location_type: multi-region
metageneration: 1
name: gen-ai-old
public_access_prevention: enforced
rpo: DEFAULT
storage_url: gs://gen-ai-old/
uniform_bucket_level_access: true
update_time: 2023-10-16T16:02:24+0000

check SA

michaelobrien@mbp7 magellan % gcloud storage buckets list --impersonate-service-account=gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com
WARNING: This command is using service account impersonation. All API calls will be executed as [gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com].
WARNING: This command is using service account impersonation. All API calls will be executed as [gen-ai-old-sa@gen-ai-old.iam.gserviceaccount.com].
---
creation_time: 2023-10-16T16:02:24+0000
default_storage_class: STANDARD
location: US
location_type: multi-region
metageneration: 1
name: gen-ai-old
public_access_prevention: enforced
rpo: DEFAULT
storage_url: gs://gen-ai-old/
uniform_bucket_level_access: true
update_time: 2023-10-16T16:02:24+0000

obriensystems · 2023-10-16T16:48:48Z

Use ADC credentials via SA

https://cloud.google.com/docs/authentication/client-libraries
https://cloud.google.com/docs/authentication/client-libraries#java

set maven
https://cloud.google.com/storage/docs/reference/libraries

add to pom.xml as usual

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>libraries-bom</artifactId>
      <version>26.24.0</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<dependencies>
  <dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-storage</artifactId>
  </dependency>

Run a GCP GCS example


@Service
public class ApplicationService implements ApplicationServiceLocal {

	@Override
	public String health() {
		return "OK";
	}

	@Override
	public String gcp() {
		try {
			authenticateImplicitWithAdc("gen-ai-old");
		} catch (IOException io) {
			System.out.println(io.getMessage());
		}
		return "gcp";
	}
	
	// https://cloud.google.com/docs/authentication/client-libraries#java
	private void authenticateImplicitWithAdc(String project) throws IOException {

		    // *NOTE*: Replace the client created below with the client required for your application.
		    // Note that the credentials are not specified when constructing the client.
		    // Hence, the client library will look for credentials using ADC.
		    //
		    // Initialize client that will be used to send requests. This client only needs to be created
		    // once, and can be reused for multiple requests.
		    Storage storage = StorageOptions.newBuilder().setProjectId(project).build().getService();

		    System.out.println("Buckets:");
		    Page<Bucket> buckets = storage.list();
		    for (Bucket bucket : buckets.iterateAll()) {
		      System.out.println(bucket.toString());
		    }
		    System.out.println("Listed all storage buckets.");
		  }
	
	  
	@Override
	public String forward() {
		// TODO Auto-generated method stub
		return "OK";
	}
}


2023-10-16 12:46:34.327 DEBUG 61449 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : GET "/nbi/app/gcp", parameters={}
2023-10-16 12:46:34.328 DEBUG 61449 --- [nio-8080-exec-8] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ApplicationServiceController#getGCP()
Buckets:
Bucket{name=gen-ai-old}
Listed all storage buckets.
2023-10-16 12:46:34.992 DEBUG 61449 --- [nio-8080-exec-8] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2023-10-16 12:46:34.992 DEBUG 61449 --- [nio-8080-exec-8] m.m.a.RequestResponseBodyMethodProcessor : Writing ["gcp"]
2023-10-16 12:46:34.993 DEBUG 61449 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
2023-10-16 12:46:35.013 DEBUG 61449 --- [nio-8080-exec-4] o.s.web.servlet.DispatcherServlet        : GET "/nbi/webjars/springfox-swagger-ui/fonts/titillium-web-v6-latin-600.woff2", parameters={}
2023-10-16 12:46:35.014 DEBUG 61449 --- [nio-8080-exec-4] o.s.w.s.handler.SimpleUrlHandlerMapping  : Mapped to ResourceHttpRequestHandler ["classpath:/META-INF/resources/webjars/"]
2023-10-16 12:46:35.016 DEBUG 61449 --- [nio-8080-exec-4] o.s.web.servlet.DispatcherServlet        : Completed 200 OK

obriensystems · 2023-10-16T16:54:45Z

Working ADC GCP for GCS example from above
obrienlabs/magellan@554164f

obriensystems · 2023-10-18T15:55:34Z

Adding alternate direct File classloader (no ENV variable) - requested by customer (tomcat war) - but this example is spring boot jar with embedded tomcat

obrienlabs/magellan@5264199


	private String authenticateFileWithAdc(String project) throws IOException {

		String lastBucket = "none";

	    // *NOTE*: Replace the client created below with the client required for your application.
	    // Note that the credentials are not specified when constructing the client.
	    // Hence, the client library will look for credentials using ADC.
	    //
	    // Initialize client that will be used to send requests. This client only needs to be created
	    // once, and can be reused for multiple requests.
		
	    // move inside the resource path - for jav/war deployment
	    //File credentialsPath = new File("~/keys/gcp-gen-ai-old-sa.json");
	    File credentialsPath = ResourceUtils.getFile("classpath:gcp-gen-ai-old-sa.json");

	    // Load credentials from JSON key file. If you can't set the GOOGLE_APPLICATION_CREDENTIALS
	    // environment variable, you can explicitly load the credentials file to construct the
	    // credentials.
	    GoogleCredentials credentials;
	    try (FileInputStream serviceAccountStream = new FileInputStream(credentialsPath)) {
	      credentials = ServiceAccountCredentials.fromStream(serviceAccountStream);
	    }
	    Storage storage = StorageOptions
	    		.newBuilder()
	    		.setCredentials(credentials)
	    		.setProjectId(project).build().getService();

	    logger.info("Buckets:");
	    Page<Bucket> buckets = storage.list();
	    for (Bucket bucket : buckets.iterateAll()) {
	    	logger.info(bucket.toString());
	      lastBucket = bucket.toString();
	    }
	    logger.info("Listed all storage buckets.");
	    return lastBucket;
	}
	
	023-10-18 12:08:31.579 DEBUG 73958 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : GET "/nbi/app/gcpViaFile", parameters={}
2023-10-18 12:08:31.580 DEBUG 73958 --- [nio-8080-exec-8] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to global.packet.magellan.controller.ApplicationServiceController#getGCPViaFile()
2023-10-18 12:08:31.651  INFO 73958 --- [nio-8080-exec-8] g.p.magellan.service.ApplicationService  : Buckets:
2023-10-18 12:08:32.173  INFO 73958 --- [nio-8080-exec-8] g.p.magellan.service.ApplicationService  : Bucket{name=gen-ai-old}
2023-10-18 12:08:32.173  INFO 73958 --- [nio-8080-exec-8] g.p.magellan.service.ApplicationService  : Listed all storage buckets.
2023-10-18 12:08:32.174 DEBUG 73958 --- [nio-8080-exec-8] m.m.a.RequestResponseBodyMethodProcessor : Using 'text/plain', given [*/*] and supported [text/plain, */*, text/plain, */*, application/json, application/*+json, application/json, application/*+json, application/x-jackson-smile, application/cbor]
2023-10-18 12:08:32.174 DEBUG 73958 --- [nio-8080-exec-8] m.m.a.RequestResponseBodyMethodProcessor : Writing ["Bucket{name=gen-ai-old}"]
2023-10-18 12:08:32.174 DEBUG 73958 --- [nio-8080-exec-8] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
michaelobrien@mbp7 magellan % git add magellan-nbi/src/main/java/
michaelobrien@mbp7 magellan % git status
On branch master
Your branch is up to date with 'origin/master'.

Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
	modified:   magellan-nbi/pom.xml
	modified:   magellan-nbi/src/main/java/global/packet/magellan/controller/ApplicationServiceController.java
	modified:   magellan-nbi/src/main/java/global/packet/magellan/service/ApplicationService.java
	modified:   magellan-nbi/src/main/java/global/packet/magellan/service/ApplicationServiceLocal.java

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	KEY_FILE
	magellan-nbi/src/main/resources/gcp-gen-ai-old-sa.json

https://cloud.google.com/bigquery/docs/authentication/service-account-file
https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/AuthSnippets.java

obriensystems · 2023-10-31T14:23:57Z

Revisit curl for bigquery
following/editing
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#local-to-gcp-api-calls-over-public-ips

Follow for bigquery dataset creation
#494 (comment)
using
https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/files/12587461/201611185_gps_distinct_rollerblad.csv

start
https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev

michaelobrien@mbp7 magellan % export GOOGLE_CLOUD_PROJECT=bigquery-ol 
michaelobrien@mbp7 magellan % gcloud config set project $GOOGLE_CLOUD_PROJECT 
michaelobrien@mbp7 magellan % gcloud auth application-default login
Your browser has been opened to visit:

    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=764086051850-6qr4p6gpi6hn506pt8ejuq83di341hur.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8085%2F&scope=openid+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fsqlservice.login&state=2F1qPHOCSnylEphtdlusVqnCyRn3n1&access_type=offline&code_challenge=hcc2MVIAbZs0m2v99f3eArtCU1DUFx86CIsayULiA84&code_challenge_method=S256


Credentials saved to file: [/Users/michaelobrien/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

Quota project "bigquery-ol" was added to ADC which can be used by Google client libraries for billing and quota. Note that some services may still bill the project owning the resource.


michaelobrien@mbp7 magellan % cat /Users/michaelobrien/.config/gcloud/application_default_credentials.json                                                  
{
  "client_id": "76408605...pt8ejuq83di341hur.apps.googleusercontent.com",
  "client_secret": "d-F....D0Ty",
  "quota_project_id": "bigquery-ol",
  "refresh_token": "1//0...pPG-yUrK12H2dYwxz....D7RJo0q0M0",
  "type": "authorized_user"
}%

ADD service account impersonation

https://cloud.google.com/docs/authentication/use-service-account-impersonation

create service account
https://console.cloud.google.com/iam-admin/serviceaccounts?project=bigquery-ol

impersonate at the ADC level for SDKs

michaelobrien@mbp7 magellan % SERVICE_ACCT_EMAIL=bigquery-ol-sa@bigquery-ol.iam.gserviceaccount.com                                                                                    
michaelobrien@mbp7 magellan % gcloud auth application-default login --impersonate-service-account $SERVICE_ACCT_EMAIL
Your browser has been opened to visit:

    https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=764086...hFvqqXKWzq8u4&code_challenge_method=S256


Credentials saved to file: [/Users/michaelobrien/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).

check before

michaelobrien@mbp7 magellan % gcloud storage buckets list
---
creation_time: 2023-10-31T14:20:38+0000
default_storage_class: STANDARD
location: NORTHAMERICA-NORTHEAST1
location_type: region
metageneration: 1
name: empty-bigquery-ol
public_access_prevention: enforced
storage_url: gs://empty-bigquery-ol/
uniform_bucket_level_access: true
update_time: 2023-10-31T14:20:38+0000

impersonate at the cli level

michaelobrien@mbp7 magellan % gcloud config set auth/impersonate_service_account $SERVICE_ACCT_EMAIL
Updated property [auth/impersonate_service_account].
michaelobrien@mbp7 magellan % gcloud storage buckets list                                           
WARNING: This command is using service account impersonation. All API calls will be executed as [bigquery-ol-sa@bigquery-ol.iam.gserviceaccount.com].
WARNING: This command is using service account impersonation. All API calls will be executed as [bigquery-ol-sa@bigquery-ol.iam.gserviceaccount.com].
---
creation_time: 2023-10-31T14:20:38+0000
default_storage_class: STANDARD
location: NORTHAMERICA-NORTHEAST1
location_type: region
metageneration: 1
name: empty-bigquery-ol
public_access_prevention: enforced
storage_url: gs://empty-bigquery-ol/
uniform_bucket_level_access: true
update_time: 2023-10-31T14:20:38+0000
michaelobrien@mbp7 magellan %

Switch to curl

get the token from the previous auth request

michaelobrien@mbp7 magellan % GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json
michaelobrien@mbp7 magellan % gcloud auth application-default print-access-token                                  
ya29.c.c0AY_....2cyh62haq1k8-cmjb


michaelobrien@mbp7 magellan % ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
michaelobrien@mbp7 magellan % echo $ACCESS_TOKEN
ya29.c.......8VYaYhr_c6ooqq6
michaelobrien@mbp7 magellan % curl -H "Authorization: Bearer $ACCESS_TOKEN" "https://www.googleapis.com/bigquery/v2/projects/$GOOGLE_CLOUD_PROJECT/datasets"
{
  "kind": "bigquery#datasetList",
  "etag": "irRvAdMvQcDWywbnuMPyBg==",
  "datasets": [
    {
      "kind": "bigquery#dataset",
      "id": "bigquery-ol:rollerblade",
      "datasetReference": {
        "datasetId": "rollerblade",
        "projectId": "bigquery-ol"
      },
      "location": "northamerica-northeast1"
    }
  ]
}

obriensystems · 2023-10-31T14:37:10Z

Query bucket list using service account impersonation for a CURL REST call

https://cloud.google.com/storage/docs/listing-buckets#rest-list-buckets
FAQ on https://github.com/GoogleCloudPlatform/pubsec-declarative-toolkit/wiki/DevOps#query-bucket-list-using-service-account-impersonation-for-a-curl-rest-call

michaelobrien@mbp7 magellan % ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"
michaelobrien@mbp7 magellan % curl -X GET -H "Authorization: Bearer $ACCESS_TOKEN" "https://storage.googleapis.com/storage/v1/b?project=$GOOGLE_CLOUD_PROJECT"
{
  "kind": "storage#buckets",
  "items": [
    {
      "kind": "storage#bucket",
      "selfLink": "https://www.googleapis.com/storage/v1/b/empty-bigquery-ol",
      "id": "empty-bigquery-ol",
      "name": "empty-bigquery-ol",
      "projectNumber": "951469276805",
      "metageneration": "1",
      "location": "NORTHAMERICA-NORTHEAST1",
      "storageClass": "STANDARD",
      "etag": "CAE=",
      "timeCreated": "2023-10-31T14:20:38.638Z",
      "updated": "2023-10-31T14:20:38.638Z",
      "iamConfiguration": {
        "bucketPolicyOnly": {
          "enabled": true,
          "lockedTime": "2024-01-29T14:20:38.638Z"
        },
        "uniformBucketLevelAccess": {
          "enabled": true,
          "lockedTime": "2024-01-29T14:20:38.638Z"
        },
        "publicAccessPrevention": "enforced"
      },
      "locationType": "region"
    }
  ]
}


note credentials last a hour or more only
michaelobrien@mbp7 magellan % curl -X GET -H "Authorization: Bearer $ACCESS_TOKEN" "https://storage.googleapis.com/storage/v1/b?project=$GOOGLE_CLOUD_PROJECT"
{
  "error": {
    "code": 401,
    "message": "Invalid Credentials",
    "errors": [
      {
        "message": "Invalid Credentials",
        "domain": "global",
        "reason": "authError",
        "locationType": "header",
        "location": "Authorization"
      }
    ]
  }
}
michaelobrien@mbp7 magellan % ACCESS_TOKEN="$(gcloud auth application-default print-access-token)"                                                            
michaelobrien@mbp7 magellan % curl -X GET -H "Authorization: Bearer $ACCESS_TOKEN" "https://storage.googleapis.com/storage/v1/b?project=$GOOGLE_CLOUD_PROJECT"
{
  "kind": "storage#buckets",
  "items": [
    {
      "kind": "storage#bucket",
      "selfLink": "https://www.googleapis.com/storage/v1/b/empty-bigquery-ol",
      "id": "empty-bigquery-ol",
      "name": "empty-bigquery-ol",
      "projectNumber": "951469276805",
      "metageneration": "1",
      "location": "NORTHAMERICA-NORTHEAST1",
      "storageClass": "STANDARD",
      "etag": "CAE=",
      "timeCreated": "2023-10-31T14:20:38.638Z",
      "updated": "2023-10-31T14:20:38.638Z",
      "iamConfiguration": {
        "bucketPolicyOnly": {
          "enabled": true,
          "lockedTime": "2024-01-29T14:20:38.638Z"
        },
        "uniformBucketLevelAccess": {
          "enabled": true,
          "lockedTime": "2024-01-29T14:20:38.638Z"
        },
        "publicAccessPrevention": "enforced"
      },
      "locationType": "region"
    }
  ]
}

obriensystems · 2023-10-31T18:34:43Z

updating for in-place ADC in eclipse via the resources section of the jar/war

http://127.0.0.1:8080/nbi/swagger-ui.html#/application-service-controller/getGCPViaFileUsingGET
obrienlabs/magellan#26
obrienlabs/magellan@5264199


	// https://github.com/obrienlabs/magellan/commit/52641990082c32b22f32137f8316382d88b55264
	// https://cloud.google.com/bigquery/docs/authentication/service-account-file
	// https://github.com/googleapis/java-bigquery/blob/main/samples/snippets/src/main/java/com/example/bigquery/AuthSnippets.java
	@Override
	public String gcpViaFile() {
		String response = null;
		try {
			response = authenticateFileWithAdc(PROJECT_ID);
			//response = explicit(PROJECT_ID);
		} catch (IOException io) {
			logger.info(io.getMessage());
		}
		return response;
	}
	
	  public String explicit(String project) throws IOException {
		  String response = "none";
		    // TODO(developer): Replace these variables before running the sample.
		    // move inside the resource path - for jav/war deployment
		    //File credentialsPath = new File("~/keys/gcp-gen-ai-old-sa.json");
		    File credentialsPath = ResourceUtils.getFile("classpath:gcp-gen-ai-old-sa.json");

		    // Load credentials from JSON key file. If you can't set the GOOGLE_APPLICATION_CREDENTIALS
		    // environment variable, you can explicitly load the credentials file to construct the
		    // credentials.
		    GoogleCredentials credentials;
		    try (FileInputStream serviceAccountStream = new FileInputStream(credentialsPath)) {
		      credentials = ServiceAccountCredentials.fromStream(serviceAccountStream);
		    }
		    /*
		     * credentials	ServiceAccountCredentials  (id=155)	
		     * clientEmail	"gen....n-ai-old.iam.gserviceaccount.com" (id=165)
		     * clientId	"1085...6390" (id=166)	
		     */

		    // Instantiate a client.
		    BigQuery bigquery =
		        BigQueryOptions.newBuilder()
		            .setCredentials(credentials)
		            .setProjectId(project)
		            .build()
		            .getService();

		    // Use the client.
		    System.out.println("Datasets:");
		    for (Dataset dataset : bigquery.listDatasets().iterateAll()) {
		      System.out.printf("%s%n", dataset.getDatasetId().getDataset());
		    }
		    return response;
		  }

json file checked into

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	KEY_FILE
	magellan-nbi/src/main/resources/gcp-gen-ai-old-sa.json

obriensystems · 2023-11-07T16:11:59Z

https://github.com/ObrienlabsDev/rest-client-java/tree/main

public class RestClient {
	
	// from 
	
	static Logger logger = Logger.getLogger(RestClient.class.getName());
		
	private static final String URL_CREATE_RECORD =
	            "http://biometric.elasticbeanstalk.com/FrontController?action=activeid";
	    
	public static void restCall() {		
		HttpRequest request = HttpRequest.newBuilder()
				.uri(URI.create(URL_CREATE_RECORD))
				.GET()
				.build();

		
	//HttpClient client = HttpClient.newBuilder().build();
	HttpClient client = HttpClient.newBuilder()
	        .version(Version.HTTP_1_1)
	        .followRedirects(Redirect.NORMAL)
	        .connectTimeout(Duration.ofSeconds(20))
	        //.proxy(ProxySelector.of(new InetSocketAddress("proxy.example.com", 80)))
	        //.authenticator(Authenticator.getDefault())
	        .build();
	try {
		HttpResponse<String> response = client.send(request, BodyHandlers.ofString());
		String body = response.body();
		logger.info("Response: " + body);
		   
		System.out.println(response.statusCode());
		System.out.println(response.body()); 
	} catch (IOException ioe) {
		ioe.printStackTrace();
	} catch (InterruptedException ie) {
		ie.printStackTrace();
	}	
	}
	
    public static void main( String[] args )
    {
        RestClient app = new RestClient();
        app.restCall();
    }
   
}

200
{ "id" : 202311035}

Nov 07, 2023 11:08:38 AM dev.obrienlabs.RestClient restCall
INFO: Response: { "id" : 202311035}

fmichaelobrien · 2023-11-08T13:33:49Z

Google Cloud Functions - HTTPS endpoint

Use the following GCP Google Cloud Functions V2 https endpoint (org: oldev - proj: dev/eventstream-dev
list parameter is input into a random selector

https://northamerica-northeast1-eventstream-dev.cloudfunctions.net/random2?list=first,second,third,forth

{key: 1, value: second}

https://github.com/ObrienlabsDev/rest-client-java/tree/main

package gcfv2;

import java.io.BufferedWriter;
import java.io.IOException;

import com.google.cloud.functions.HttpFunction;
import com.google.cloud.functions.HttpRequest;
import com.google.cloud.functions.HttpResponse;

import java.util.Collections;
import java.util.List;
import java.util.Optional;
import java.util.StringTokenizer;
import java.util.stream.Collectors;


public class HelloHttpFunction implements HttpFunction {

  public String random(BufferedWriter writer, String input) throws java.lang.Exception {
		List<String> strings = Collections.list(new StringTokenizer(input, ",")).stream()
	      .map(token -> (String) token)
	      .collect(Collectors.toList());
    int index = (int)(Math.random() * strings.size());
    writer.write("{key: " + index + ", ");
		return strings.get(index);
  }

  public void service(final HttpRequest request, final HttpResponse response) throws java.lang.Exception, IOException {
    final BufferedWriter writer = response.getWriter();
    Optional<String> aCSVString = aCSVString = request.getFirstQueryParameter("list");
    if(aCSVString.isPresent()) {
      writer.write("value: " + random(writer, aCSVString.get()) + "}");
    } else {
      writer.write("append ?list=first,second,third....to get a random indexed string back in json");
    }
  }
}

fmichaelobrien · 2023-11-14T18:28:50Z

See https proxy ENV variables set for java apps in ObrienlabsDev/rest-client-java#1 (comment)

fmichaelobrien · 2023-11-14T18:41:38Z

https proxy settings must be at the war/jar classloader level to allow for AD authentication via System.setProperty()

fmichaelobrien added documentation Improvements or additions to documentation developer-experience canary labels Oct 16, 2023

fmichaelobrien self-assigned this Oct 16, 2023

fmichaelobrien mentioned this issue Oct 16, 2023

Add GCP PSC endpoint googleapis.com REST call to BigQuery or Vertex/GenAI obrienlabs/magellan#26

Open

obriensystems mentioned this issue Oct 16, 2023

Developer Docs: Add local dev environment/authentication docs with googleapis.com example #559

Open

fmichaelobrien added the api label Oct 31, 2023

This was referenced Nov 2, 2023

Porting: adapt LZ V2 logging into the LZ V1 - specifically organization/folder/project level routing for buckets and 3rd party pubsub integration #634

Open

HttpClient SE REST client for SG2C PSC googleapis.com calls without using spring boot #645

Open

obriensystems mentioned this issue Nov 7, 2023

Java SE REST Client ObrienlabsDev/rest-client-java#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Developer: add generative-ai-app-builder client REST api developer config - ideally in spring boot #562

Developer: add generative-ai-app-builder client REST api developer config - ideally in spring boot #562

obriensystems commented Oct 16, 2023 •

edited by fmichaelobrien

obriensystems commented Oct 16, 2023 •

edited

obriensystems commented Oct 16, 2023

obriensystems commented Oct 16, 2023 •

edited

obriensystems commented Oct 18, 2023 •

edited

obriensystems commented Oct 31, 2023 •

edited by fmichaelobrien

obriensystems commented Oct 31, 2023 •

edited

obriensystems commented Oct 31, 2023

obriensystems commented Nov 7, 2023

fmichaelobrien commented Nov 8, 2023 •

edited

fmichaelobrien commented Nov 14, 2023

fmichaelobrien commented Nov 14, 2023

Developer: add generative-ai-app-builder client REST api developer config - ideally in spring boot #562

Developer: add generative-ai-app-builder client REST api developer config - ideally in spring boot #562

Comments

obriensystems commented Oct 16, 2023 • edited by fmichaelobrien

obriensystems commented Oct 16, 2023 • edited

Create Service Account

Add roles to SA

Create Service Account Key

Export key

Add service account impersonation

obriensystems commented Oct 16, 2023

Use ADC credentials via SA

obriensystems commented Oct 16, 2023 • edited

obriensystems commented Oct 18, 2023 • edited

obriensystems commented Oct 31, 2023 • edited by fmichaelobrien

ADD service account impersonation

Switch to curl

obriensystems commented Oct 31, 2023 • edited

Query bucket list using service account impersonation for a CURL REST call

obriensystems commented Oct 31, 2023

obriensystems commented Nov 7, 2023

fmichaelobrien commented Nov 8, 2023 • edited

Google Cloud Functions - HTTPS endpoint

fmichaelobrien commented Nov 14, 2023

fmichaelobrien commented Nov 14, 2023

obriensystems commented Oct 16, 2023 •

edited by fmichaelobrien

obriensystems commented Oct 16, 2023 •

edited

obriensystems commented Oct 16, 2023 •

edited

obriensystems commented Oct 18, 2023 •

edited

obriensystems commented Oct 31, 2023 •

edited by fmichaelobrien

obriensystems commented Oct 31, 2023 •

edited

fmichaelobrien commented Nov 8, 2023 •

edited