Stackdriver Output When not on GCP/GCE

[GLStephen]

I keep getting "[2022/06/11 19:10:48] [debug] [output:stackdriver:stackdriver.0] [logging.googleapis.com/monitored_resource] not found in the payload".

Config is:

[OUTPUT]
    Name                        stackdriver
    Match                       bidder.event_log*
    google_service_credentials  /fluent-bit/etc/gcp-project-d017ead76452.json
    resource                    global
    export_to_project_id        gcp-project
    labels                      logging.googleapis.com/monitored_resource=bidder

I've tried with and without the labels entry. I see the metadata server always being set, but I'm not on GCE. I see that the code should parse the credentials file which I am using, but it seems it's still not building the monitored_resource properly or at all.

[BartVB]

Normally the project ID is already in the credentials file?

We also get:

[2022/06/04 03:04:14] [ info] [output:stackdriver:stackdriver.1] metadata_server set to http://metadata.google.internal

while we are not on Google Cloud but that doesn't seem to affect operations.

I also ran into this 'monitored resource not found' issue in a k8s environment but this was a few months ago. Can't remember what caused it, should keep better notes :) Enabling debugging on FluentBit would be my first step to resolve this.

(BTW great work on PubWise!)

[GLStephen]

@BartVB It seems this error doesn't really cause a failure. It looks like the fluent-bit side code expects something in the API return that isn't required, but the request completes. I think the monitored resource issue was fixed based on looking through a bunch of commits. I'll try to get someone here to give this some deeper testing and see if we can polish off some rough spots on this and understand it better.

Also, thanks for the compliment.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.

[paulleroyza]

I'm getting a similar issue, using fleuntbit 3.0.3 using workload identity federation which is working for other workloads (gcloud containers, google managed prometheus and node red) but for fluentbit it keeps looking for the metadata service. The credential configuration file has the correct details for the KSA/GSA federation and I have changed the container from fluentbit to gcloud-sdk to test that the pod has connection which it does. This is not a network or credential issue.

config snippet

[OUTPUT]
    Name        stackdriver
    Match       *
    google_service_credentials /etc/workload-identity/credential-configuration.json

error message

[2024/05/08 16:47:04] [ info]      stackdriver.0
[2024/05/08 16:47:04] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2024/05/08 16:47:05] [debug] [stackdriver:stackdriver.0] created event channels: read=850 write=896
[2024/05/08 16:47:05] [ info] [output:stackdriver:stackdriver.0] metadata_server set to http://metadata.google.internal
[2024/05/08 16:47:05] [ warn] [output:stackdriver:stackdriver.0] client_email is not defined, using a default one
[2024/05/08 16:47:05] [ warn] [output:stackdriver:stackdriver.0] private_key is not defined, fetching it from metadata server
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] failed to create metadata connection
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] can't fetch token from the metadata server
[2024/05/08 16:47:05] [ warn] [output:stackdriver:stackdriver.0] token retrieval failed
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] failed to create metadata connection
[2024/05/08 16:47:05] [error] [output:stackdriver:stackdriver.0] can't fetch project id from the metadata server
[2024/05/08 16:47:05] [error] [output] failed to initialize 'stackdriver' plugin

It seems like it ignores the environment variables which is why I hard coded the credentials file. I have tried with and without the project_id_key and export_to_project_id set.