Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Logs dissapear after a while when using dd-trace +4 in Cloud Run #3704

Closed
abarone-btf opened this issue Oct 11, 2023 · 1 comment
Closed
Labels
bug Something isn't working

Comments

@abarone-btf
Copy link

abarone-btf commented Oct 11, 2023

Hi all, this is a really weird behavior I start noticing. The logs from my cloud run instances suddently stop appearing in the Cloud Logging view (using dd-trace 4.16.0 and serverless-init 1.1.1)

Its weird because when I restart the instance the logs start appearing again without any issue, but after some hours new logs are never shown until I restart the instance again.

I even try console.log (am using Node 18) for every request inside the instance to test, but even that log dissapear.

I've been debugging this behavior for almost a week, and recently try to remove everything form datadog and the logs never dissapear again.

Maybe something happend in the newer version of dd-trace or the newer version of the serverless-init image that all of the sudden starts "eating" the logs made to stdout.

Some images to show the behavior, the first is when everything works, the logs are showing up on the stackdriver panel:
everything-ok

After sometime, the logs made from my instance never appear again, only showing the default Cloud Run request log:
after-some-time

Here are the env variables that my cloud run is using:

DD_SERVICE: some-service-name 
DD_ENV: some-env   
DD_VERSION: 0.0.1
DD_LOGS_ENABLED: true
DD_TRACE_TELEMETRY_ENABLED: false 
DD_APPSEC_ENABLED: true
---
DD_API_KEY: using secret manager integration, injected as env variable

Thanks for any help!

@IvanTopolcic
Copy link

I'm going to close this for now as this is likely an issue with the agent, which is being tracked here. If not, feel free to reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants