Aborted/restarted Container due to some memory-errors

[Jman1993]

Hi,
I am getting error messages and my container are restarting occasionally with an error.
These are some of the errors i am getting.

malloc(): unsorted double linked list corrupted
SIGABRT: abort
PC=0x7fb51ab1a5ef m=0 sigcode=18446744073709551610
signal arrived during cgo execution

double free or corruption (!prev)
SIGABRT: abort
PC=0x7f50d6ace5ef m=3 sigcode=18446744073709551610
signal arrived during cgo execution

corrupted size vs. prev_size
SIGABRT: abort
PC=0x7f69802815ef m=14 sigcode=18446744073709551610
signal arrived during cgo execution

These Exporters are running on a Docker-Host on Ubuntu 22.04.05 LTS

Client: Docker Engine - Community
 Version:    27.5.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.20.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.4
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

I have 16GB of RAM available and 10GB of these are free.(Because these errors are looking like memory-problems)

I don't know if this is relevant, but i am Querying a Exadata-Machine in a RAC-Configuration. Because of this i am using a custom default metrics file using the gv$-Views.
I added docker logs-excerpt for one of my exporters and one example docker compose file.

docker_logs.txt

docker-compose-example.txt

Thank you very much!

[markxnelson]

HI @Jman1993 - thanks for reporting, we will take a look and get back to you. At first glance, looks more like a concurrent memory access issue, not a lack of memory.

[oey]

OP and I get the same stack trace and same issues.

I'm no Go developer but it seems that this might be an issue with Godror.
And it points to this section in Godror.

[jlembeck06]

Hi guys, I'm getting the same errors on my Cloud Foundry deployment (1.5.3)

As Oey said, it seems like there is some problem in the Godror https://github.com/godror/godror/tree/v0.46.0

ERR malloc(): unsorted double linked list corrupted
ERR double free or corruption (out)
ERR SIGABRT: abort
ERR PC=0x7f7f175f95ef m=11 sigcode=18446744073709551610
ERR signal arrived during cgo execution
ERR goroutine 135 gp=0xc000502e00 m=11 mp=0xc00032c008 [syscall, locked to thread]:
ERR runtime.cgocall(0xa4bd90, 0xc0002c5b80)
ERR /usr/local/go/src/runtime/cgocall.go:157 +0x4b fp=0xc0002c5b58 sp=0xc0002c5b20 pc=0x4170ab
ERR github.com/godror/godror._Cfunc_dpiConn_create(0x2ca6950, 0x7f7e9c0f1df0, 0x7, 0x7f7e9c0f1d10, 0x16, 0x7f7e9c00c9a0, 0x66, 0xc000288320, 0xc0002ae090, 0xc0002a4010)

[markxnelson]

Thanks very much @jlembeck06 , we are trying to debug it and hope to have a fix in the next few days.
If it is possible to try 1.5.2 and let us know if you still see the same issue, that would be a big help.

[jlembeck06]

Thank you @markxnelson

Yes, I have tested versions 1.0 to 1.5.2, and they are working well.

I only encountered the reported issue with version 1.5.3.

[markxnelson]

Thanks very much, that helps!

[markxnelson]

Hi, just letting you know we are working on a potential fix, doing some testing now. Hope to have an update in the next few days.

[markxnelson]

We believe we have this fixed in 1.5.4 - if you still see the error, please reopen, or open a new issue to let you know. Thanks very much for your input.

[Jman1993]

Hi, unfortunately, we are still encountering the same error after about 2–4 minutes of the container running. The error looks the same, but I cannot say for certain. Here are the Docker logs for the container.

docker_logs.txt

[markxnelson]

@Jman1993 thanks for lettung us know. Could you please share the parameters used for the exporter, as well as the following details about the environment you're running it on:

• OS
• Architecture
• Containerized or bare metal/VM?
• version of instant client
• version of glibc

In our testing, we found that setting the following connection pool parameters will resolve this problem:

      --database.maxIdleConns=10
      --database.maxOpenConns=10

[VictorErmakov]

@Jman1993 Try to add following lines to your docker compose file

[brunobergamo]

Hi

I had the same problem with 1.5.4. Worked well with 1.5.2 when I curl localhost:9161/metrics.

I am using in K8S

[markxnelson]

Hi @brunobergamo - can you confirm what configuration variables you used? Did you set the connection parameters as shown above? Thanks for reporting the issue!

[markxnelson]

Hi @brunobergamo and others. We just did a 1.5.5 release, which changes the default values, so that you won't need to set any extra params. It should work now without those pool settings.

[brunobergamo]

Hi @markxnelson , Sorry the late answer.

The new version is working properly!

Thanks a lot and have a nice weekend !

[markxnelson]

Thanks very much for the update. I will close this issue, but please feel free to reopen, or create a new issue if you have additional problems.