fast growing database since update 1.8.0 -> 1.8.7

[rasos]

We run a large etherpad instance since many years. After updating from ep 1.8.0 to 1.8.7 the size of the database has nearly doubled the last 12 days. Usage was even less during holiday season (as seen in traffic monitoring).

3 days ago:

# ls -lh /var/lib/mysql/etherpadlite
total 44G
-rw-rw---- 1 mysql mysql   65 Feb 20  2013 db.opt
-rw-rw---- 1 mysql mysql 1.4K Jul 30  2018 store.frm
-rw-rw---- 1 mysql mysql  42G Jan  7 14:28 store.MYD
-rw-rw---- 1 mysql mysql 1.6G Jan  7 14:28 store.MYI

today:

# ls -lh /var/lib/mysql/etherpadlite
total 53G
-rw-rw---- 1 mysql mysql   65 Feb 20  2013 db.opt
-rw-rw---- 1 mysql mysql 1,4K Jul 30  2018 store.frm
-rw-rw---- 1 mysql mysql  51G Jän 10 18:34 store.MYD
-rw-rw---- 1 mysql mysql 1,7G Jän 10 18:34 store.MYI

File upload plugin is enabled, but that loads files into the files system (checked).

List of installed plugins

tables4
ep_image_upload
ep_toc
headings
slideshow
scrolltoanchor
code_formatting 
message_all
print
loading_message

Anybody observing similar grow of database size? Is there any ep tool to observe what is really going on in the database?

[JohnMcLear]

I'm not seeing it on other instances but that doesn't mean it isn't happening. Is there any way I can share a call with you to debug?

[JohnMcLear]

The trick to debugging is getting previous dB entries and looking to see what's new/changed.

I did make one change months ago that stored more atext on every 100 revs instead of 1000 afaik. It shouldn't have the level of impact you have experienced though....

[JohnMcLear]

I put a proposed patch in for this which is currently being tested.

[rasos]

The proposed patch was successful. DB storage usage and traffic is again normal.

-rw-rw---- 1 mysql mysql 57998901380 Jan 11 20:45 store.MYD
-rw-rw---- 1 mysql mysql 58046926616 Jan 13 02:27 store.MYD

DB growth was 48025236 MB in 30 hours -> 38 MB per day (and not 1 GB).

[JohnMcLear]

Can you please uncomment the patched code and see if the growth comes and see how long the growth remains.

I think the growth will only be experienced on first time a pad is stored then you shop stop seeing the growth.

Let's monitor it for 7 days with the code back in please. If we still keep seeing exponetential growth we can patch else this is expected behavior as your pads were moving to a more stable storage format.

[rasos]

Uncommented the patch only for one day due to the rapid growth of the DB: nearly 2 GB on a Saturday, which is usually a low traffic day.

-rw-rw---- 1 mysql mysql 58167704672 Jan 16 10:35 store.MYD
-rw-rw---- 1 mysql mysql 59844015700 Jan 17 08:48 store.MYD

Re-applied the patch by commenting in Pad.js line 93

    // newRevData.meta.pool = this.pool;

[JohnMcLear]

Please leave the code in. It is important. I expect to see signific growth for at least a few days but then it should settle down.

[rasos]

Okay. Re-applied patch. Let's hope that the situation stabilizes in some days.

[JohnMcLear]

How goes it @rasos ?

[rasos]

No end of the growth in sight.

-rw-rw---- 1 mysql mysql 70855520056 Jan 22 21:56 store.MYD

The DB was growing again 11 GB the last 5 days (20% of its total size). My infrastructure colleagues will kill me tomorrow when hitting again 90% of available LVM space ;-/

Disc usages shown in percent of available disc space. Applied upgrade on Dec 28th, applied patch on Jan 11, removed patch on Jan 16th. The steps down are increases of the LVM volume.

[JohnMcLear]

I think give it another 4-5 days and it will have leveled off to it's new point.

[rasos]

9 days later our instanced consumed another 6,7 GB or 9.5% more space

     -rw-rw---- 1 mysql mysql 77573075324 Jan 31 16:48 store.MYD

However, growth seemed to have stopped 3 days ago after we had to increase disc space once again (which you see as the step down as graphs are percentage)

So let's hope that the etherpad storage situation now has been consolidated exactly one month after the update.

[JohnMcLear]

Yep this is about the behavior we expected from that change :) This matches our public test instance so all good here.

Thanks for reporting and sending graphs and updates etc :)