Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory tracking miscount #1437

Closed
gbour opened this issue Oct 1, 2019 · 4 comments

Comments

@gbour
Copy link

@gbour gbour commented Oct 1, 2019

Hello,

We hit a memory tracking issue in mosquitto (both 1.6.4 and master tested). This leads to client connections being rejected when used memory exceed the limit set in mosquitto.conf (memory tracking enabled).
We identified that at some point, there is a memcount counter underflow and because it is a unsigned long, its value becomes greater than max_memcount and all new client connections are rejected.
The role of our mosquitto-memleak.patch patch is to print a stacktrace and force-crashing mosquitto when the underflow happens.

100 void mosquitto__free(void *mem)                                                                      
101 {                                                                                                    
102 #ifdef REAL_WITH_MEMORY_TRACKING                                                                     
103     if(!mem){                                                                                        
104         return;                                                                                      
105     }                                                                                                
106     if (memcount < malloc_usable_size(mem)) {                                                        
107         print_stacktrace();                                                                          
108         assert (0);                                                                                  
109     } 

This is 100% reproducible with following scenario:

shell1> patch -p1 < mosquitto-memleak.patch
shell1> ./src/mosquitto -c mosquitto-memleak.conf
shell2> while true; do mosquitto_sub -h 127.0.0.1 -t '$PROBE' -E -d; sleep .1; done

in less than 10 minutes, mosquitto is crashing with following stacktrace:

=== BACKTRACE ===
./src/mosquitto(mosquitto__free+0x48)[0x4162d8]
./src/mosquitto(context__free_disused+0x37)[0x40e9a7]
./src/mosquitto(mosquitto_main_loop+0x1d6)[0x415756]
./src/mosquitto(main+0x338)[0x409148]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xeb)[0x7f91695ecbbb]
./src/mosquitto(_start+0x2a)[0x4092ca]
=== /BACKTRACE ===

mosquitto-memleak.zip contains:

  • mosquitto-memleak.patch
  • mosquitto-memleak.conf
  • mosquitto-memleak.core
@gbour gbour changed the title Memory leak Memory tracking miscount Oct 1, 2019
ralight added a commit that referenced this issue Oct 1, 2019
Closes #1437. Thanks to Guillaume Bour.
@ralight

This comment has been minimized.

Copy link
Contributor

@ralight ralight commented Oct 1, 2019

Thanks for the report - as you've just noted it's a memory tracking problem not a leak. I've just pushed a fix.

@gbour

This comment has been minimized.

Copy link
Author

@gbour gbour commented Oct 1, 2019

You're awesome 👍
I'll test it

@gbour

This comment has been minimized.

Copy link
Author

@gbour gbour commented Oct 3, 2019

works perfectly fine, I cannot hit the issue anymore

@gbour gbour closed this Oct 3, 2019
@ralight

This comment has been minimized.

Copy link
Contributor

@ralight ralight commented Oct 8, 2019

Thank you for confirming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.