Unable to connect to emulator running on docker compose with client 7.5.1

[pokutuna]

I have development environments and CI to run Datastore emulator and an application that connect to it on Docker Compose.
Those connections are resolved by the service name on the overlay network within it, such as datastore:8081.

Since client version 7.5.1, these cannot connect to the emulator.

This was triggered by this PR: #1101
When the baseUrl_ does not include a part that means the local network, grpc.credentials.createInsecure is no longer used.

This change is for a custom endpoint, and the endpoint is given by the DATASTORE_EMULATOR_HOST environment variable.
As a result, authentication is not skipped when the emulator host is something like datastore:8081.

To support custom endpoints requiring authentication, how about using a another environment variable instead of reusing the existing one? (like DATASTORE_CUSTOM_HOST) I think users who set the DATASTORE_EMULATOR_HOST are expecting development use and not expecting authentication to be required.

Workaround

Setting network_mode: "host" and expose the emulator port for joining the host network, we can include localhost in endpoint url. However, this occupies a port on the host and may need to be adjusted.

---

Environment details

• OS: macOS(host) & Linux(container)
• Node.js version: v20.2.0
• npm version: 9.6.6
• @google-cloud/datastore version: 7.5.1

Steps to reproduce

1. Using Docker Compose, set up the Datastore emulator and an application container that uses the client library.
2. You will encounter the error: "Could not load the default credentials." during connection.

This is the reproduction code using two different client versions, including docker-compose.yaml:
https://gist.github.com/pokutuna/314248d183f6fbfe60154f63751d3655

[BenedictQk]

I believe I've also hit this problem. The error in my CI pipeline started happening since upgrading to 7.5.1 and is

Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.
at GoogleAuth.getApplicationDefaultAsync (/__w/1/s/node_modules/google-auth-library/build/src/auth/googleauth.js:207:19)
at processTicksAndRejections (node:internal/process/task_queues:95:5)
at GoogleAuth.getClient (/__w/1/s/node_modules/google-auth-library/build/src/auth/googleauth.js:620:17)
at GrpcClient._getCredentials (/__w/1/s/node_modules/google-gax/src/grpc.ts:207:20)
at GrpcClient.createStub (/__w/1/s/node_modules/google-gax/src/grpc.ts:409:19)

We are using testcontainers to spin up a local datastore emulator for Jest tests. The code looks like this:

const { GenericContainer, AlwaysPullPolicy } = require('testcontainers');

 const container = await GenericContainer.fromDockerfile(buildContext)
      .withPullPolicy(new AlwaysPullPolicy())
      .build();

    const startedContainer = await container.withExposedPorts(8081).start();

    const ip = startedContainer.getHost();
    const port = startedContainer.getMappedPort(8081);
    const host = `${ip}:${port}`;
...

    apiEndpoint: `http://${host}`,
    projectId: 'local-test-stub-project',

In our local setup this starts the emulator on localhost so this works fine. In our CI pipeline the host is 172.17.0.1:32769 so this change has broken our tests.

To be honest I think the code in https://github.com/googleapis/nodejs-datastore/pull/1101/files seems pretty flawed - just because you're not using localhost or 127.0.0.1 doesn't mean you're not using the local emulator.

[danieljbruce]

For now please use the following workaround. Provide sslCreds in the constructor as shown by the following script.

const {Datastore} = require('@google-cloud/datastore');
const grpc = require('@grpc/grpc-js');

const datastore = new Datastore({sslCreds: grpc.credentials.createInsecure()});