App Engine can't call Cloud Functions with Example code #1508

mheripsos · 2022-12-28T20:55:34Z

Is this a client library issue or a product issue?
Definitely seems like a client library issue.
Did someone already solve this?
Not this specific issue it doesn't seem like it.
Do you have a support contract?
Yes, but I already have a work around anyway, and don't need a timely response., and it definitely seems like an issue with this library from what I can tell.

If the support paths suggested above still do not result in a resolution, please provide the following details.

Environment details

OS: Windows (or App Engine)
Node.js version: v18.2.0 (or App Engine + v16)
npm version: 8.9.0
google-auth-library version: 8.7.0

I am trying to make a call from App Engine to an http Cloud Function using the example code in this library.

There actually two separate issues which are that the example code for using Application Default Credentials does not appear to work for me neither in 1. a local environment nor in 2. App Engine in the cloud itself.

I could not get any of the client library request methods to work correctly locally and have to use a fetch like method directly. Because a fetch works, but the request methods don't, it seems highly likely that this is an issue in this library itself.

Steps to reproduce

Set up Cloud Function:

plain index.js:

export function respond(_, response) {
    response.send({status: 'ok'});
}

Deploy Cloud Function:

gcloud functions deploy repro --region=us-central1 --runtime=nodejs16 --source=./function --entry-point=respond --trigger-http --vpc-connector=projects/<MY_PROJECT_ID>/locations/us-central1/connectors/<MY_VPC_CONNECTOR> --ingress-settings all --egress-settings all

Run Express Server Locally:

const express = require('express');
const GoogleAuth = require('google-auth-library').GoogleAuth;

const app = express();
const auth = new GoogleAuth({
    scopes: 'https://www.googleapis.com/auth/cloud-platform'
});
const fnaud = 'https://<MY_PROJECT_ID>.cloudfunctions.net';
const fnurl = fnaud + '/repro';

async function withPlainClient(url) {
    try {
        const client = await auth.getClient();
        const value = await client.request({
            url: url,
            method: 'POST',
            data: JSON.stringify({hello: 'world'}),
            headers: {'Content-Type': 'application/json'}
        });
        return({
            result: 'success',
            value: value
        });
    } catch (error) {
        return({
            result: 'failure',
            value: {...error, message: error.message}
        })
    }
}

app.get('/', async(_, response) => {
    const data = await withPlainClient(fnurl);
    response.send(data);
})

app.listen(
    process.env.PORT || 8080,
    () => console.log(`Server listening on port ${process.env.PORT || 8080}...`)
);

Visiting the page results in the following response from the call:

                "response": {
                    "config": {
                        "url": "https://<MY_PROJECT_ID>.cloudfunctions.net/repro",
                        "method": "POST",
                        "data": "{\"hello\":\"world\"}",
                        "headers": {
                            "Content-Type": "application/json",
                            "x-goog-user-project": "<MY_PROJECT_ID>",
                            "Authorization": "Bearer <TOKEN_HERE>",
                            "User-Agent": "google-api-nodejs-client/8.7.0",
                            "x-goog-api-client": "gl-node/18.2.0 auth/8.7.0",
                            "Accept": "application/json"
                        },
                        "body": "{\"hello\":\"world\"}",
                        "responseType": "json"
                    },
                    "data": "\n<html><head>\n<meta http-equiv=\"content-type\" content=\"text/html;charset=utf-8\">\n<title>401 Unauthorized</title>\n</head>\n<body text=#000000 bgcolor=#ffffff>\n<h1>Error: Unauthorized</h1>\n<h2>Your client does not have permission to the requested URL <code>/repro</code>.</h2>\n<h2></h2>\n</body></html>\n",
                    "headers": {
                        "alt-svc": "<I REPLACED THE TEXT HERE, not sure if it's important, it's not human readable>",
                        "connection": "close",
                        "content-length": "299",
                        "content-type": "text/html; charset=UTF-8",
                        "date": "Wed, 28 Dec 2022 00:42:07 GMT",
                        "server": "Google Frontend",
                        "www-authenticate": "Bearer error=\"invalid_token\" error_description=\"The access token could not be verified\""
                    },
                    "status": 401,
                    "statusText": "Unauthorized",
                    "request": {
                        "responseURL": "https://<MY_PROJECT_ID>.cloudfunctions.net/repro"
                    }
                },

The workaround:
I tried almost every possible example case given in the README but could not get any of them to work (usually getting a 401 like above with the same "The access token could not be verified" message. The only way I could get it to work was to make an HTTP call directly outside of the client library and stuff the token into the headers. I mean that I had to use a fetch equivalent to (I'm guessing) bypass the over-riding logic in the client library request calls.

//...
require('isomorphic-fetch');
const dots = /\.*$/;
//...

async function withFetchIDToken(url) {
    try {
        const client = await auth.getClient();
        // this call appears to be necessary, although we don't use an access token here from what I can tell
        await client.getAccessToken();
        const value = await fetch(url, {
            method: 'POST',
            body: JSON.stringify({hello: 'world'}),
            // the `dots` replace is just an extra precaution since tokens are sometimes generated with a bunch of trailing dots (there is another ticket for this #1499). But I don't think it's actually related here. 
            headers: {
                'Authorization': 'Bearer ' +  client.credentials.id_token.replace(dots, ''),
                'Content-Type': 'application/json'
            }
        });
        return({
            result: 'success',
            value: value
        });
    } catch (error) {
        return({
            result: 'failure',
            value: {...error, message: error.message}
        })
    }
}
//...
app.get('/', async(_, response) => {
    const data = await withFetchIDToken(fnurl);
    response.send(data);
})
//...

However, neither the example code nor the above workaround works in deployed-to-App-Engine code. There, in the deployed code, I had to do two things:

use the IDTokenClient example
remove VPC-egress settings on App Engine

The example in (1) does not work locally, but does work on deployed-to-App-Engine

I am kind of confused with (2) above, but deploying with the following in the app.yaml will prevent this from working:

vpc_access_connector:
  name: "projects/<MY_PROJECT_ID>/connectors/<MY_VPC_CONNECTOR>"
  egress_setting: all-traffic

If anything, I imagined that that setting would be necessary, but it actually prevents the fix.

In any case, because I have fixes for both cases, things are ok. But either the documentation is unclear on how to use these classes/methods, or there is some sort of bug in the request methods that does not work for Cloud Function urls.

This may also be related to issue #1432.

The text was updated successfully, but these errors were encountered:

danielbankhead · 2023-04-13T00:29:14Z

Hey @mheripsos,

Thanks for your patience and details - let's get this resolved for you.

First thing: Cloud Run and Cloud Functions require OpenID Connect (OIDC) tokens (ID tokens) (ref: https://cloud.google.com/docs/authentication/use-cases#run-functions). This is likely why your client.credentials.id_token sample works locally, but not the request method. If using a UserRefreshClient, here's a bit of background on a fetchIdToken proposal: #876.

Second: egress_setting: all-traffic means "Requests are [then] subject to the firewall, DNS, and routing rules of your VPC network" (ref: https://cloud.google.com/appengine/docs/standard/connecting-vpc#egress). Might you have some restrictions within the VPC network preventing this from working?

mheripsos · 2023-04-19T18:34:03Z

Hi @danielbankhead, thanks for the response. You'll have to bear with me a little as my understanding of auth stuff isn't totally solid here.

I think I had misunderstood what the egress_settings meant and that makes sense now (and yes there are org restrictions here), so just putting that aside entirely.

The rest of my understanding of your post and links (again, a little shaky here), is that there is actually no class/method in the library that will let me do a .request() in the local case? Is there some other client class that I could use? It's not really all that important because it's just for local dev environment and we have something that works, so it's fine.

It's also been a while, so I don't really remember how I got to the point I did, but it was after trying a bunch of stuff and ending up with a sort of hacked together solution above.

I guess what I wanted to get out of this was "Is there some sort of documented (intended) way I could do this to cover both cases?" The solution I ended up with gives me strong "I'm doing something wrong" vibes.

In any case if you want to respond with more information, you can and I would appreciate it. But it's not a big deal and you can close the ticket.

danielbankhead · 2023-07-13T20:53:55Z

getIdTokenClient

The rest of my understanding of your post and links (again, a little shaky here), is that there is actually no class/method in the library that will let me do a .request() in the local case? Is there some other client class that I could use? It's not really all that important because it's just for local dev environment and we have something that works, so it's fine.

That's correct - nothing local at the moment until #876 is resolved (unless you're interested in the service account impersonation, which then GoogleAuth#getIdTokenClient would work).

For now, it might be easiest to use the IDTokenClient sample on the server-side, while falling-back to your 'Authorization': 'Bearer ' + client.credentials.id_token logic when developing locally.

danielbankhead · 2023-10-12T17:55:59Z

I'll close this issue as #876 is the root feature request. Please follow along there!

mheripsos added priority: p2 Moderately-important priority. Fix may not be included in next release. type: bug Error or flaw in code with unintended results or allowing sub-optimal usage patterns. labels Dec 28, 2022

sofisl assigned danielbankhead Jan 5, 2023

danielbankhead added the priority: p2 Moderately-important priority. Fix may not be included in next release. label Apr 12, 2023

sofisl added priority: p3 Desirable enhancement or fix. May not be included in next release. and removed priority: p2 Moderately-important priority. Fix may not be included in next release. labels May 11, 2023

danielbankhead closed this as not planned Won't fix, can't repro, duplicate, stale Oct 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

App Engine can't call Cloud Functions with Example code #1508

App Engine can't call Cloud Functions with Example code #1508

mheripsos commented Dec 28, 2022

danielbankhead commented Apr 13, 2023

mheripsos commented Apr 19, 2023

danielbankhead commented Jul 13, 2023

danielbankhead commented Oct 12, 2023

App Engine can't call Cloud Functions with Example code #1508

App Engine can't call Cloud Functions with Example code #1508

Comments

mheripsos commented Dec 28, 2022

Environment details

Steps to reproduce

danielbankhead commented Apr 13, 2023

mheripsos commented Apr 19, 2023

danielbankhead commented Jul 13, 2023

danielbankhead commented Oct 12, 2023