-
Notifications
You must be signed in to change notification settings - Fork 569
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
upgrade (new install) from 1.11.6 -> 3.0.3 (x86_64/linux) CRASH after long run. #2294
Comments
Another thing worth noting is memory usage and CPU appear stable. What seems to expediate the crash is many transaction time outs. forcing timeouts on the network side help to re-create the issue. Hope the crash dump can help. |
Updated crash file and gdb dump; gdb /usr/local/opensips3.x/sbin/opensips core.opensips.sig11.32688.2 |
Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days. |
@cc3283 , thank you for your report and sorry for the delay here. Your back trace shows a memory corruption, a double free. Do you see in the logs something like "freeing already freed 0xnnnnnnn pointer" ?? Also try to grab the latest version from GIT, I see your checkout is a bit old. |
Hello Bogdan,
Yes, that is what I see in the core.
*"Using host libthread_db library
"/lib/x86_64-linux-gnu/libthread_db.so.1".*
*Core was generated by `/usr/local/opensips3.x/sbin/opensips -E -w
/var/crash -n 16 -P /var/run/opensip'.*
*Program terminated with signal SIGSEGV, Segmentation fault.*
*#0 0x00007fcb643bfcd0 in _IO_vfprintf_internal (s=s@entry=0x21b5030,
format=<optimized out>,*
* format@entry=0x68a200 "CRITICAL:core:%s: freeing already freed %s
pointer (%p), first free: %s: %s(%ld) - aborting!\n", ap=0x7ffca6b1e688)*
* at vfprintf.c:1632*
*1632 vfprintf.c: No such file or directory.*
*"*
This seems to happen when the traffic backs up and I see the udp buffers
start to climb. but really that is just a guess. We did not see this on
the 2.x version. We are seeing this on both 3.0.3 and 3.1.0.
I will pull the latest, but will take a while to test and report back to
you.
…On Tue, Jan 5, 2021 at 8:28 AM Bogdan Andrei IANCU ***@***.***> wrote:
@cc3283 <https://github.com/cc3283> , thank you for your report and sorry
for the delay here.
Your back trace shows a memory corruption, a double free. Do you see in
the logs something like "freeing already freed 0xnnnnnnn pointer" ??
Also try to grab the latest version from GIT, I see your checkout is a bit
old.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2294 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB4OTADMEFHVPTDQNAS7MDTSYMHY5ANCNFSM4TFFZE2A>
.
|
Looks similar to #2362 . |
@cc3283 , let's see if upgrading to latest 3.0 will also do the job here. |
Thank you both, I'll download 3.1.1LTS asap and start testing.
…On Fri, Jan 8, 2021 at 1:50 AM Alexey Vasilyev ***@***.***> wrote:
Looks similar to #2362 <#2362>
.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2294 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB4OTAH3K4CC4UDQZOTQWR3SY2TMNANCNFSM4TFFZE2A>
.
|
Hello, I would like to update you on the testing of 3.1,1 vs 3.0.x.
I have testing is LAB and have one instance in production at 300+cps/qps.
It has been stable with no crashes for the past two weeks. In that time
the other instance running 3.0.2 has crashed three times.
I would be interested in any thought you guys might have on a reason, but
in any case this looks like a success!
Thanks again for your excellent support,
Cory
…On Fri, Jan 8, 2021 at 2:37 PM Cory Cartwright ***@***.***> wrote:
Thank you both, I'll download 3.1.1LTS asap and start testing.
On Fri, Jan 8, 2021 at 1:50 AM Alexey Vasilyev ***@***.***>
wrote:
> Looks similar to #2362 <#2362>
> .
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#2294 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB4OTAH3K4CC4UDQZOTQWR3SY2TMNANCNFSM4TFFZE2A>
> .
>
|
Any updates here? No progress has been made in the last 15 days, marking as stale. Will close this issue if no further updates are made in the next 30 days. |
Hello, After moving traffic to new install of 3.0.1, opensips 3.0.1 started crashing seemingly at random times after about a week of traffic load. found bug related to auto_scaling. Disabled auto_scaling, and upgraded to 3.0.3. Still crashing, but it appears I am able to create the environment with non-production traffic at fairly low rate. I have uploaded the crash file. we run both 1.11.6-notls and 1.11.1-tls without issue for years and same traffic sources and type.
version: opensips 3.0.3 (x86_64/linux)
flags: STATS: On, DISABLE_NAGLE, USE_MCAST, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, HP_MALLOC, DBG_MALLOC, FAST_LOCK-ADAPTIVE_WAIT
ADAPTIVE_WAIT_LOOPS=1024, MAX_RECV_BUFFER_SIZE 262144, MAX_LISTEN 16, MAX_URI_SIZE 1024, BUF_SIZE 65535
poll method support: poll, epoll, sigio_rt, select.
git revision: cf2c490
main.c compiled on 05:17:45 Oct 7 2020 with gcc 5.4.0
core.opensips.sig11.22486.gz
The text was updated successfully, but these errors were encountered: