-
Notifications
You must be signed in to change notification settings - Fork 9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak when using the /metrics
endpoint (PrometheusExporter)
#95
Comments
@krynju I think this may be an issue of the incremental GC not being able to keep up. I was trying to recreate this to see if I could fix it, but what I ended up trying was in the background having it do a full GC every few seconds. With that I don't see the memory to continue to grow, even after an hour. Before the Here's my diff: 4c4
< using OpenTelemetryExporterPrometheus
---
> using OpenTelemetryExporterPrometheus
7a8,9
> GC.enable_logging(true)
>
172a175,182
> function gc_full_occasionally()
> while true
> sleep(2)
> GC.gc()
> end
>
> end
>
198c208,217
<
---
> @async begin
> try
> gc_full_occasionally()
> catch ex
> @error(
> "exception initializing",
> exception = (ex, catch_backtrace())
> )
> end
> end |
Notes from today:
|
I ran the reproducer on 1.10 on a custom branch of OpenTelemetry.jl that has proper 1.10 support This is potentially good news, but I'm not 100% sure of it yet. |
EDIT: This is observable on 1.9, but not on 1.10
I've identified a memory leak in a long running service coming from OpenTelemetry.
I'm attaching an MWE split off from the main service, which can reproduce the issue identically
The example attached is pretty extreme and can get you to ~2GB memory usage in about 10 minutes.
The leak is dependent on measure activity and capturing measures through the
/metrics
endpoint.metricsspam
emulates a running service, which generates some metric activityab
to call the/metrics
endpoint extensivelyVersion info:
julia 1.9.3, linux
OpenTelemetry versions as the attached Manifest.toml
or simpler, this repo at commit 4975ecd
Reproducer: memleak.zip
Tar contents:
Steps:
julia --project=. -e "include(\"run.jl\");init();"
ab -c 500 -n 1000000 http://localhost:9967/
Example runtime (~4gb in 25min)
The text was updated successfully, but these errors were encountered: