Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unexpected growth in the .db file #389

Closed
ashishbajaj99 opened this Issue Feb 25, 2017 · 11 comments

Comments

Projects
None yet
5 participants
@ashishbajaj99
Copy link

ashishbajaj99 commented Feb 25, 2017

I have a broker configured with the following settings:

<snip>
persistence true
<snip>

Further I have a phone client that connects over websocket interface with a fixed unchanged clientid each time, and sets up its cleanSessionFlag to true, so its not a persistent session.

But after every such connection and disconnection, I see the mosquitto.db file growing. On opening the db file in an editor, I see multiple sections of data for each of the connections.

But if the cleanSessionFlag is set to false, the file doesn't grow - there is only one instance of the clientid maintained. But as expected I do receive all the QoS 1 and QoS2 retained messages that I may have missed during the time I didn't have connection. I dont want any of those messages delivered to the client, which is why I changed my client to connect using cleanSessionFlag to false.

This is not something expected right? I am on mosquitto 1.4.10...is there any known issue?

Let me know if you need any logs?

@ralight

This comment has been minimized.

Copy link
Contributor

ralight commented Feb 28, 2017

The first thing to look at would be what is actually in the persistence file - you could use the mosquitto_db_dump to take a look: https://github.com/eclipse/mosquitto/tree/master/src/db_dump

It might require some fiddling with the db_dump source to find out exactly what is happening.

@PierreF

This comment has been minimized.

Copy link
Contributor

PierreF commented Jan 5, 2018

Do you still experience this issue ?

@ashishbajaj99

This comment has been minimized.

Copy link
Author

ashishbajaj99 commented Jan 5, 2018

@pjchx

This comment has been minimized.

Copy link

pjchx commented Feb 12, 2019

Hi, I am having similar issue. The db file is growing quickly, I have to manualy delete it every 7-10 days.
Can't install the db analyser log (my linux version is too old for it I think), but looking into raw db I see nothing suspicious - just a large log of my usual mqtt traffic in it.
I had this setup operating for about a year, but this problem started to appear cca 2 months back.
Any hint what I should be looking for in the db file to find the root cause ?

@karlp

This comment has been minimized.

Copy link
Contributor

karlp commented Feb 12, 2019

try harder to compile the db dump tools. You probably have a persistent client with a subscripion thhat you didn't intend. An option that can be very useful to aovid this is persistent_client_expiration set to say, 7d or similar, or whatever's appropriate for your environment. THe MQTT spec itself unfortunately says that by default messages are stored infinitely.

@ralight

This comment has been minimized.

Copy link
Contributor

ralight commented Feb 13, 2019

@pjchx Try the following:

git clone https://github.com/eclipse/mosquitto
cd mosquitto
git checkout v1.4.15
cd src/db_dump
make

If you have errors, report them here and we'll help - but try and install missing tools first if you can.
This should give you mosquitto_db_dump, which you could then run as ./mosquitto_db_dump --stats <path/to/mosquitto.db> and post the output here.

@pjchx

This comment has been minimized.

Copy link

pjchx commented Feb 13, 2019

hi, I managed somehow to compile on rPi running Jessie. The another rPi running Wheezy (and mosquitto client) compilation did not work.

Now the dump tool is working. However I deleted the big file yesterday so I have to wait until it builds up again in few days. Then I will analyse.Thanks for the help so far.

p.s.
btw, when compiling on Jessie I had to fix this in db_dump source code:

line 192 for(int i=0; ipayloadlen; i++)

I moved the the i variable definition before the loop.

@ralight

This comment has been minimized.

Copy link
Contributor

ralight commented Feb 13, 2019

Actually I may have isolated the problem. If you're comfortable with compiling the whole mosquitto source, then try:

git clone https://github.com/eclipse/mosquitto
cd mosquitto
git checkout fixes
make

If you need websockets, then run make WITH_WEBSOCKETS=yes instead. Then run mosquitto from the src directory with your config file.

@ralight ralight closed this in 321e566 Feb 14, 2019

@pjchx

This comment has been minimized.

Copy link

pjchx commented Feb 22, 2019

Hi, my file has just grown again to 230MB
the db_dump says this :

root@xxx:/opt/mosquitto-master/src/db_dump# ./mosquitto_db_dump --stats mosquitto.db
DB_CHUNK_CFG: 1
DB_CHUNK_MSG_STORE: 2884449
DB_CHUNK_CLIENT_MSG: 0
DB_CHUNK_RETAIN: 270
DB_CHUNK_SUB: 0
DB_CHUNK_CLIENT: 0

on --client.stats parameter
root@xxx:/opt/mosquitto-master/src/db_dump# ./mosquitto_db_dump --client-stats mosquitto.db
root@xxx:/opt/mosquitto-master/src/db_dump#

...it returns nothing.

in my mosquitto.conf I have the following parameters
persistence true
persistence_location /var/lib/mosquitto/

Is it possible that due to the the persistence true parameter the in-memory database content is "appended" to mosquitto.db on disk and that is why it is growing ?

thanks for helping on this.

@pjchx

This comment has been minimized.

Copy link

pjchx commented Feb 22, 2019

...one more thing.

Actually I may have isolated the problem. If you're comfortable with compiling the whole mosquitto source, then try:

git clone https://github.com/eclipse/mosquitto
cd mosquitto
git checkout fixes
make

If you need websockets, then run make WITH_WEBSOCKETS=yes instead. Then run mosquitto from the src directory with your config file.

I tried to compile on my Wheezy but no luck:

> cc sub_client.o sub_client_output.o client_shared.o -o mosquitto_sub  -L../lib ../lib/libmosquitto.so.1
> /usr/bin/ld: sub_client_output.o: undefined reference to symbol 'clock_gettime@@GLIBC_2.4'
> //lib/arm-linux-gnueabihf/librt.so.1: error adding symbols: DSO missing from command line
> collect2: ld returned 1 exit status
> Makefile:33: recipe for target 'mosquitto_sub' failed
> make[1]: *** [mosquitto_sub] Error 1
> make[1]: Leaving directory '/tmp/mosquitto/client'
> Makefile:26: recipe for target 'mosquitto' failed
> make: *** [mosquitto] Error 2
@ralight

This comment has been minimized.

Copy link
Contributor

ralight commented Feb 27, 2019

The problem isn't an append, it is that in some circumstances some messages were being persisted even when they weren't in use, and then never being freed. This is fixed in the latest release.

I'm sorry, but wheezy is about 9 months past its end of life, it's not supported by Debian and not supported by the Mosquitto project either. Having said that, if you just want the broker then try:

make -C src
sudo make -C src install

This will build just the broker, which should be ok I think.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.