goaccess crashing when using --keep-db-files --load-from-disk

[brian77777]

here is the error:

==10202== GoAccess 1.0.2 crashed by Sig 11
==10202==
==10202== VALUES AT CRASH POINT
==10202==
==10202== Line number: 1864853
==10202== Offset: 0
==10202== Invalid data: 1653
==10202== Piping: 0
==10202== Response size: 120276307679 bytes
==10202==
==10202== STACK TRACE:
==10202==
==10202== 0 goaccess(sigsegv_handler+0x188) [0x40b8b8]
==10202== 1 /lib64/libc.so.6(+0x35250) [0x7f662682d250]
==10202== 2 /lib64/libc.so.6(+0x14ac86) [0x7f6626942c86]
==10202== 3 /lib64/libtokyocabinet.so.9(+0x3fd94) [0x7f6626e00d94]
==10202== 4 /lib64/libtokyocabinet.so.9(tcbdbput+0x8f) [0x7f6626e0198f]
==10202== 5 /lib64/libtokyocabinet.so.9(tcadbput+0x1a9) [0x7f6626e22409]
==10202== 6 goaccess() [0x424d89]
==10202== 7 goaccess() [0x418325]
==10202== 8 goaccess(parse_log+0x129) [0x418aa9]
==10202== 9 goaccess(main+0x1fe) [0x40813e]
==10202== 10 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f6626819b35]
==10202== 11 goaccess() [0x4090cd]
==10202==
==10202== Please report it by opening an issue on GitHub:
==10202== https://github.com/allinurl/goaccess/issues

Here is how I am running goaccess:

goaccess -f /var/log/jigsaw/jigsaw.log -o /var/www/goaccess/report.html --real-time-html --ws-url=hostname.com --keep-db-files --load-from-disk

It works perfectly without: --keep-db-files --load-from-disk

Proof that Tokyo Cabinet is installed:

goaccess -s
Built using Tokyo Cabinet On-Disk B+ Tree.

Please let me know how I should tune the memory in the configuration file to make this work properly.

[allinurl]

Thank you for reporting this. I was trying to replicate this on my side without success so far, so I'm thinking it may be a log specific issue. Could you please provide a few more details on how to reproduce this? For instance:

1. Are you still able to reproduce this?
2. Was that the first time running goaccess or did you already run it before and were you only appending new data to an existing data set?
3. Does this occur with a smaller log? e.g., tail -500000 /var/log/jigsaw/jigsaw.log > test.log
4. What distro/version are you using? and how did you install tokyocabinet?

Thanks for any insights you might have!

[brian77777]

1. Are you still able to reproduce this? Yes, but only when using the options: --keep-db-files --load-from-disk 1. Was that the first time running goaccess or did you already run it before and were you only appending new data to an existing data set? This was not the first time running the application. I got it running manually and gradually added options. It ran perfectly for a week handling one to two gigs of logs daily without issues. 1. Does this occur with a smaller log? e.g., tail -500000 /var/log/jigsaw/jigsaw.log > test.log I have not tried this. I will try to do this and see if it works. Is there any known issues with larger data sets? 1. What distro/version are you using? and how did you install tokyocabinet? This is a centos linux box version 7.0. Tokyocabinet was installed when I installed: *goaccess*.x86_64 1.0.2-2.el7 I maintain my own repo, and I simply added the goaccess rpm to it. Nothing else appeared to be necessary. Is it possible that I need to adjust the memory settings in my config file?

…

On Sun, Jun 18, 2017 at 10:13 AM, Gerardo O. ***@***.***> wrote: Thank you for reporting this. I was trying to replicate this on my side without success so far, so I'm thinking it may be a log specific issue. Could you please provide a few more details on how to reproduce this? For instance: 1. Are you still able to reproduce this? 2. Was that the first time running goaccess or did you already run it before and were you only appending new data to an existing data set? 3. Does this occur with a smaller log? e.g., tail -500000 /var/log/jigsaw/jigsaw.log > test.log 4. What distro/version are you using? and how did you install tokyocabinet? Thanks for any insights you might have! — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#813 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AcIA5DrKkzuoUI2zQ9DRy9cAbyfAIuipks5sFT6sgaJpZM4N81A5> .

-- Brian Keithly Director of Operations 314-258-1927

[allinurl]

Yes, but only when using the options: --keep-db-files --load-from-disk

Great, would you be able to build from source and run the latest version (v1.2) and see if it still occurs in there?

I have not tried this. I will try to do this and see if it works. Is there any known issues with larger data sets?

There are no known issues with larger data sets. However, this appears to be the same issue as #696.

BTW, does it crash if you remove --real-time-html?, e.g.,

goaccess -f /var/log/jigsaw/jigsaw.log -o /var/www/goaccess/report.html --ws-url=hostname.com --keep-db-files --load-from-disk

[brian77777]

I can try to run it form source, but that will take some effort as system is already in production. The reports are being used by several departments. If I remove the real-time part will the web report cease to work? Also I was not able to capture the message, but before things started crashing with the db configuration turned on I was getting some out of memory errors. They preceded the crashing, and went away when I stopped running from disk. So I hope I don't sound like a broken record but that is why I am curious if tweaking memory in the config file might resolve the issue. The system I have this running on is dedicated solely to collecting logs and then running goaccess. It has 8G of memory with 7G free. The log files are not exceeding 2G, and I can easily increase the memory on the system with no limit.

…

On Mon, Jun 19, 2017 at 7:45 PM, Gerardo O. ***@***.***> wrote: Yes, but only when using the options: --keep-db-files --load-from-disk Great, would you be able to build from *source* <https://goaccess.io/download#installation> and run the latest version (v1.2) and see if it still occurs in there? I have not tried this. I will try to do this and see if it works. Is there any known issues with larger data sets? There are no known issues with larger data sets. However, this appears to be the same issue as #696 <#696>. BTW, does it crash if you remove --real-time-html?, e.g., goaccess -f /var/log/jigsaw/jigsaw.log -o /var/www/goaccess/report.html --ws-url=hostname.com --keep-db-files --load-from-disk — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#813 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AcIA5Fgsho1DdSiuYepgWb6qhJMIsb6Rks5sFxYpgaJpZM4N81A5> .

-- Brian Keithly Director of Operations 314-258-1927

[allinurl]

Let me run a some tests using v1.0.2 and then I may ask you to try it out with v1.2.

Removing the --real-time-html flag will only generate a static report. If you need to update the report, you will need to rerun goaccess which will overwrite the html file. So there shouldn't be any issues as long as you are okay with a static report, mainly just to be sure that this flag it's not causing the issue.

The on-disk storage it's pretty lightweight when it comes to memory usage. However, a few bugs have been fixed regarding memory consumption on the WebSocket server, so it could be that if it was actually running out of memory, the DB files got somehow corrupted leading to the crash. Does it crash for you with existing DB files?, or does it crash even after deleting the existing DB files and generating the report again from scratch?

As far as tweaking memory in the config file, it shouldn't be necessary, other than improving performance, but I'm curious, how big are your goaccess DB files?

[brian77777]

Great thanks for the updates. I will perform the test without the real time report and update this thread. I am fine with the static report but my company loves it so it will be hard to not offer that to them. If we can't figure it out from there then I will certainly run it with v1.2. If I may ask about the using the DB configuration. If I do get it working will this allow the report to cover a time frame that spans the age of the DB? For example the title on the HTML report appears to reflect the earliest and oldest time stamp in the log file when go access is initially launched. So will that time frame just continue to grow or will I have to relaunch goaccess to get it to update? Thanks for an awesome tool! Even without the DB working properly it is still very useful.

…

On Tue, Jun 20, 2017 at 9:18 AM, Gerardo O. ***@***.***> wrote: Let me run a some tests using v1.0.2 and then I may ask you to try it out with v1.2. Removing the --real-time-html flag will only generate a static report. If you need to update the report, you will need to rerun goaccess which will overwrite the html file. So there shouldn't be any issues as long as you are okay with a static report, mainly just to be sure that this flag it's not causing the issue. The on-disk storage it's pretty lightweight when it comes to memory usage. However, a few bugs have been fixed regarding memory consumption on the WebSocket server, so it could be that if it was actually running out of memory, the DB files got somehow corrupted leading to the crash. Does it crash for you with existing DB files?, or does it crash even after deleting the existing DB files and generating the report again from scratch? As far as tweaking memory in the config file, it shouldn't be necessary, other than improving performance, but I'm curious, how big are your goaccess DB files? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#813 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AcIA5JUTO0DXsgigTjpn9V9didG9xMvYks5sF9StgaJpZM4N81A5> .

-- Brian Keithly Director of Operations 314-258-1927

[allinurl]

If I follow your question, yes, the time frame will continue to grow. — depending on your setup, if it's real-time it will automatically update as dates are added, otherwise the time frame will be reflected every time you rerun goaccess to generate the new report.

[brian77777]

Perfect thanks! I will update as I perform the testing steps you suggested.

…

On Tue, Jun 20, 2017 at 9:36 AM, Gerardo O. ***@***.***> wrote: If I follow your question, yes, the time frame will continue to grow. — depending on your setup, if it's real-time it will automatically update as dates are added, otherwise the time frame will be reflected every time you rerun goaccess to generate the new report. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#813 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AcIA5CARYpjV4mtGINAdpOtEjiqymWkLks5sF9jigaJpZM4N81A5> .

-- Brian Keithly Director of Operations 314-258-1927

[brian77777]

I am installing from source on a second machine because my company really would like to keep the real-time functionality. Which compilation option should I use when compiling: memhash or btree? I would prefer the fastest of the two. Server resources are not a concern in regards to disk or memory. I can easily increase either should their be a need. I would assume memory is the fastest? On Tue, Jun 20, 2017 at 9:44 AM, Brian Keithly < brian.keithly@gatewayblend.com> wrote:

…

Perfect thanks! I will update as I perform the testing steps you suggested. On Tue, Jun 20, 2017 at 9:36 AM, Gerardo O. ***@***.***> wrote: > If I follow your question, yes, the time frame will continue to grow. — > depending on your setup, if it's real-time it will automatically update as > dates are added, otherwise the time frame will be reflected every time you > rerun goaccess to generate the new report. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#813 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AcIA5CARYpjV4mtGINAdpOtEjiqymWkLks5sF9jigaJpZM4N81A5> > . > -- Brian Keithly Director of Operations 314-258-1927 <(314)%20258-1927>

-- Brian Keithly Director of Operations 314-258-1927

[brian77777]

I meant memhash. On Wed, Jun 21, 2017 at 10:03 AM, Brian Keithly < brian.keithly@gatewayblend.com> wrote:

…

I am installing from source on a second machine because my company really would like to keep the real-time functionality. Which compilation option should I use when compiling: memhash or btree? I would prefer the fastest of the two. Server resources are not a concern in regards to disk or memory. I can easily increase either should their be a need. I would assume memory is the fastest? On Tue, Jun 20, 2017 at 9:44 AM, Brian Keithly < ***@***.***> wrote: > Perfect thanks! > > I will update as I perform the testing steps you suggested. > > > > On Tue, Jun 20, 2017 at 9:36 AM, Gerardo O. ***@***.***> > wrote: > >> If I follow your question, yes, the time frame will continue to grow. — >> depending on your setup, if it's real-time it will automatically update as >> dates are added, otherwise the time frame will be reflected every time you >> rerun goaccess to generate the new report. >> >> — >> You are receiving this because you authored the thread. >> Reply to this email directly, view it on GitHub >> <#813 (comment)>, >> or mute the thread >> <https://github.com/notifications/unsubscribe-auth/AcIA5CARYpjV4mtGINAdpOtEjiqymWkLks5sF9jigaJpZM4N81A5> >> . >> > > > > -- > Brian Keithly > Director of Operations > 314-258-1927 <(314)%20258-1927> > -- Brian Keithly Director of Operations 314-258-1927 <(314)%20258-1927>

-- Brian Keithly Director of Operations 314-258-1927

[brian77777]

Sorry for all the spam, but I went ahead stood up the second server, copied log files over to it, installed goaccess from source (v1.2) and I am still seeing the program crash: [SERVER_NAME]# /home/users/USER/goaccess-1.2/goaccess -p goaccess.conf -f /var/log/jigsaw/jigsaw.log -o /var/www/goaccess/report.html --ws-url=ws_url --keep-db-files --load-from-disk & ==32423== GoAccess 1.2 crashed by Sig 11 ==32423== ==32423== VALUES AT CRASH POINT ==32423== ==32423== Line number: 242434 ==32423== Offset: 0 ==32423== Invalid data: 252 ==32423== Piping: 0 ==32423== Response size: 14604585248 bytes ==32423== ==32423== STACK TRACE: ==32423== ==32423== 0 /home/users/USER/goaccess-1.2/goaccess(sigsegv_handler+0x9a) [0x40b8ac] ==32423== 1 /lib64/libc.so.6(+0x35250) [0x7f2b87fab250] ==32423== 2 /lib64/libtokyocabinet.so.9(+0x3fd5d) [0x7f2b8857ed5d] ==32423== 3 /lib64/libtokyocabinet.so.9(tcbdbput+0x8f) [0x7f2b8857f98f] ==32423== 4 /lib64/libtokyocabinet.so.9(tcadbput+0x1a9) [0x7f2b885a0409] ==32423== 5 /home/users/USER/goaccess-1.2/goaccess() [0x434ecd] ==32423== 6 /home/users/USER/goaccess-1.2/goaccess() [0x435327] ==32423== 7 /home/users/USER/goaccess-1.2/goaccess(ht_insert_uniqmap+0x6d) [0x435c7a] ==32423== 8 /home/users/USER/goaccess-1.2/goaccess() [0x421802] ==32423== 9 /home/users/USER/goaccess-1.2/goaccess() [0x422beb] ==32423== 10 /home/users/USER/goaccess-1.2/goaccess() [0x422d09] ==32423== 11 /home/users/USER/goaccess-1.2/goaccess() [0x422e7c] ==32423== 12 /home/users/USER/goaccess-1.2/goaccess() [0x422ed2] ==32423== 13 /home/users/USER/goaccess-1.2/goaccess() [0x42312f] ==32423== 14 /home/users/USER/goaccess-1.2/goaccess() [0x42338d] ==32423== 15 /home/users/USER/goaccess-1.2/goaccess(parse_log+0x1bd) [0x423580] ==32423== 16 /home/users/USER/goaccess-1.2/goaccess(main+0xad) [0x414c89] ==32423== 17 /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f2b87f97b35] ==32423== 18 /home/users/USER/goaccess-1.2/goaccess() [0x4085f9] ==32423== ==32423== Please report it by opening an issue on GitHub: ==32423== https://github.com/allinurl/goaccess/issues [1]+ Exit 1 /home/users/USER/goaccess-1.2/goaccess -p goaccess.conf -f /var/log/jigsaw/jigsaw.log -o /var/www/goaccess/report.html --ws-url=ws_url --keep-db-files --load-from-disk [SERVER_NAME]# /home/users/USER/goaccess-1.2/goaccess -p goaccess.conf -f /var/log/jigsaw/jigsaw.log -o /var/www/goaccess/report.html --ws-url=ws_url --keep-db-files --load-from-disk & [SERVER_NAME]# /home/users/USER/goaccess-1.2/goaccess -V GoAccess - 1.2. For more details visit: http://goaccess.io Copyright (C) 2009-2016 by Gerardo Orellana [SERVER_NAME]# /home/users/USER/goaccess-1.2/goaccess -s Built using Tokyo Cabinet On-Disk B+ Tree. [SERVER_NAME]# Here are the options enabled in my config file: time-format %H:%M:%S date-format %Y-%m-%d log-format %^ %^ %^ %^ %^: %dT%t+%^,%h,%^,%T,%b,%s,%^,%r,%R,%u config-dialog false hl-header true json-pretty-print false no-color false no-column-names false no-csv-summary false no-progress false no-tab-scroll false with-mouse false addr 0.0.0.0 port 7890 real-time-html true debug-file /var/log/debug.log agent-list false with-output-resolver false http-method yes http-protocol yes no-query-string false no-term-resolver false 444-as-404 false 4xx-to-unique-count false double-decode false ignore-crawlers false ignore-panel REFERRERS ignore-panel KEYPHRASES real-os true

[allinurl]

Thanks for running that test on v1.2. I'm looking into it.

As far as which version to install, I'd certainly recommend the default hash tables. I've been running it for months on a high traffic server without issues. Just build as:

$ wget http://tar.goaccess.io/goaccess-1.2.tar.gz
$ tar -xzvf goaccess-1.2.tar.gz
$ cd goaccess-1.2/
$ ./configure --enable-utf8 --enable-geoip=legacy
$ make
# make install

[brian77777]

That is what I did except I added:

…

--*enable*-tcb=btree To the configure line, was that not necessary? Also this is very interesting, but I wiped out the database, and tried again and it worked fine. This is on a second box with 1.2 vs 1.0.2 installed. I have since then pipped 2G more logs into it and still no issues. I believe you mentioned that the database could get corrupted?

On Wed, Jun 21, 2017 at 1:49 PM, Gerardo O. ***@***.***> wrote: Thanks for running that test on v1.2. I'm looking into it. As far as which version to install, I'd certainly recommend the default hash tables. I've been running it for months on a high traffic server without issues. Just build as: $ wget http://tar.goaccess.io/goaccess-1.2.tar.gz $ tar -xzvf goaccess-1.2.tar.gz $ cd goaccess-1.2/ $ ./configure --enable-utf8 --enable-geoip=legacy $ make # make install — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#813 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AcIA5PnbPU_VIXZUCE-COZqUZVVkIismks5sGWWfgaJpZM4N81A5> .

-- Brian Keithly Director of Operations 314-258-1927

[allinurl]

That works too plus you get the benefit of committing data to disk (less memory). If you choose not to use --enable-tcb=btree, then speed performance will be much better at the cost of using more ram.

I see. So that must be why I haven't been able to reproduce this since it looks like the database got corrupted and wiping all the files solved the issue. However, I'm still curious how it got corrupted... I'm suspicion about the --real-time-html flag combined with btree.

Please run it for some time and let me know if v1.2 somehow also corrupts the database or if it fixed the issue.

[brian77777]

Will do. And I am running it with --real-time-html AND

…

--keep-db-files --load-from-disk On a second box now with no issues.

On Wed, Jun 21, 2017 at 2:46 PM, Gerardo O. ***@***.***> wrote: That works too plus you get the benefit of committing data to disk (less memory). If you choose not to use --enable-tcb=btree, then speed performance will be much better at the cost of using more ram. I see. So that must be why I haven't been able to reproduce this since it looks like the database got corrupted and wiping all the files solved the issue. However, I'm still curious how it got corrupted... I'm suspicion about the --real-time-html flag combined with btree. Please run it for some time and let me know if v1.2 somehow also corrupts the database or if it fixed the issue. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#813 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AcIA5B3LiTrxw0aWgf8TqCo9KJvEJu8oks5sGXMUgaJpZM4N81A5> .

-- Brian Keithly Director of Operations 314-258-1927

[brian77777]

Is there a clean way to shut down the app? On Wed, Jun 21, 2017 at 3:00 PM, Brian Keithly < brian.keithly@gatewayblend.com> wrote:

…

Will do. And I am running it with --real-time-html AND --keep-db-files --load-from-disk On a second box now with no issues. On Wed, Jun 21, 2017 at 2:46 PM, Gerardo O. ***@***.***> wrote: > That works too plus you get the benefit of committing data to disk (less > memory). If you choose not to use --enable-tcb=btree, then speed > performance will be much better at the cost of using more ram. > > I see. So that must be why I haven't been able to reproduce this since it > looks like the database got corrupted and wiping all the files solved the > issue. However, I'm still curious how it got corrupted... I'm suspicion > about the --real-time-html flag combined with btree. > > Please run it for some time and let me know if v1.2 somehow also corrupts > the database or if it fixed the issue. > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#813 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/AcIA5B3LiTrxw0aWgf8TqCo9KJvEJu8oks5sGXMUgaJpZM4N81A5> > . > -- Brian Keithly Director of Operations 314-258-1927 <(314)%20258-1927>

-- Brian Keithly Director of Operations 314-258-1927

[allinurl]

Either Ctrl+c or SIGPIPE|SIGTERM will cleanly terminate it.

[brian77777]

Thanks. I am running it in the background so I will have to figure out how I can do that. I was stopping it by killing the PID, running it again but then generating a CSV, emailing that internally to certain team members, cleaning up the CSV, and kicking off again with --real-time-html. As I mentioned earlier I wonder if I kill the proc before the DB had been fully built thus corrupting it, and then it work from then on out. Anyway I appreciate your time and application very much.

…

On Wed, Jun 21, 2017 at 3:08 PM, Gerardo O. ***@***.***> wrote: Either Ctrl+c or SIGPIPE|SIGTERM will cleanly terminate it. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#813 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AcIA5FkG1vHD6PzvdwmkzxPQ9P0qLSFaks5sGXg4gaJpZM4N81A5> .

-- Brian Keithly Director of Operations 314-258-1927

[allinurl]

Assuming you don't specify a signal, killing the process works fine since it will send a SIGTERM. No worries, let me know how it goes. Thanks.

[allinurl]

The on-disk storage issue has been addressed. It now supports native on-disk persistence storage (no dependencies). New command line options are --persist, --restore, --keep-last=<num_days>. It also adds the ability to re-parse the same log(s) without counting data twice (see #334 or for exact details this post).

This will be pushed out in the upcoming release. However, it would be great if you could test this out before deployment by building from development.

Closing this as fixed. Feel free to post back as needed.