Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error: [RedisException] Redis server went away #2104

Closed
mkoelewijn opened this issue Apr 3, 2017 · 11 comments
Closed

Error: [RedisException] Redis server went away #2104

mkoelewijn opened this issue Apr 3, 2017 · 11 comments

Comments

@mkoelewijn
Copy link

mkoelewijn commented Apr 3, 2017 •
edited

MISP worked fine, but after i connected to another MISP instance to sync (pull) i got the following error:

An Internal Error Has Occurred.
Error: An Internal Error Has Occurred.

Also can't get into the "server settings" or "jobs" pages anymore. Can anyone help me to fix this?

(and yes redis-cli is working fine: 127.0.0.1:6379> )

Work environment

Questions Answers
Type of issue support...
OS version (server) Debian 8.7
OS version (client) OS X
PHP version PHP 5.6.30-0+deb8u1
MISP version / git hash 2.4.69
Browser Firefox

Expected behavior

Sync should pull all events from other MISP instance

Actual behavior

An Internal Error Has Occurred.
Error: An Internal Error Has Occurred.

Steps to reproduce the behavior

I am not 100% sure but it looks like this happened after i tried to "pull all" (Sync) from a other MISP.

Logs, screenshots, configuration dump, ...

2017-04-03 15:27:52 Error: [RedisException] Redis server went away
Request URL: /servers/pull/3/full
Stack Trace:
#0 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(28): Redis->setOption(2, 'resque:')
#1 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(18): RedisApi->establishConnection()
#2 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(168): RedisApi->__construct('localhost', '6379', 5, NULL)
#3 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque.php(102): Resque_Redis->__construct('localhost', '6379', NULL)
#4 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque.php(123): Resque::redis()
#5 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(77): Resque::push('default', Array)
#6 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque.php(169): Resque_Job::create('default', 'ServerShell', Array, false)
#7 [internal function]: Resque::enqueue('default', 'ServerShell', Array, false)
#8 /var/www/MISP/app/Plugin/CakeResque/Lib/CakeResque.php(140): call_user_func_array('Resque::enqueue', Array)
#9 /var/www/MISP/app/Controller/ServersController.php(479): CakeResque::enqueue('default', 'ServerShell', Array)
#10 [internal function]: ServersController->pull('3', 'full')
#11 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(491): ReflectionMethod->invokeArgs(Object(ServersController), Array)
#12 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction(Object(CakeRequest))
#13 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke(Object(ServersController), Object(CakeRequest))
#14 /var/www/MISP/app/webroot/index.php(92): Dispatcher->dispatch(Object(CakeRequest), Object(CakeResponse))
#15 {main}

2017-04-03 19:10:00 Error: [RedisException] Redis server went away
Request URL: /servers/serverSettings
Stack Trace:
#0 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(28): Redis->setOption(2, 'resque:')
#1 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(18): RedisApi->establishConnection()
#2 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(168): RedisApi->__construct('localhost', '6379', 5, NULL)
#3 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque.php(102): Resque_Redis->__construct('localhost', '6379', NULL)
#4 /var/www/MISP/app/Model/Server.php(2906): Resque::redis()
#5 /var/www/MISP/app/Controller/ServersController.php(753): Server->workerDiagnostics(0)
#6 [internal function]: ServersController->serverSettings()
#7 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(491): ReflectionMethod->invokeArgs(Object(ServersController), Array)
#8 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction(Object(CakeRequest))
#9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke(Object(ServersController), Object(CakeRequest))
#10 /var/www/MISP/app/webroot/index.php(92): Dispatcher->dispatch(Object(CakeRequest), Object(CakeResponse))
#11 {main}

2017-04-03 17:04:33 Error: [RedisException] Redis server went away
Request URL: /jobs/index
Stack Trace:
#0 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(28): Redis->setOption(2, 'resque:')
#1 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(18): RedisApi->establishConnection()
#2 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(168): RedisApi->__construct('localhost', '6379', 5, NULL)
#3 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque.php(102): Resque_Redis->__construct('localhost', '6379', NULL)
#4 /var/www/MISP/app/Model/Server.php(2906): Resque::redis()
#5 /var/www/MISP/app/Controller/JobsController.php(20): Server->workerDiagnostics(0)
#6 [internal function]: JobsController->index()
#7 /var/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(491): ReflectionMethod->invokeArgs(Object(JobsController), Array)
#8 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction(Object(CakeRequest))
#9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke(Object(JobsController), Object(CakeRequest))
#10 /var/www/MISP/app/webroot/index.php(92): Dispatcher->dispatch(Object(CakeRequest), Object(CakeResponse))
#11 {main}

Also tried to restart the workers;

root@misp:/var/www/MISP/app/Model# su - www-data -s /bin/bash -c 'bash /var/www/MISP/app/Console/worker/start.sh'
Error: Redis server went away
#0 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(28): Redis->setOption(2, 'resque:')
#1 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(18): RedisApi->establishConnection()
#2 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(168): RedisApi->__construct('localhost', '6379', 5, NULL)
#3 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque.php(102): Resque_Redis->__construct('localhost', '6379', NULL)
#4 /var/www/MISP/app/Plugin/CakeResque/Console/Command/CakeResqueShell.php(77): Resque::redis()
#5 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/Shell.php(450): CakeResqueShell->startup()
#6 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(219): Shell->runCommand('stop', Array)
#7 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(66): ShellDispatcher->dispatch()
#8 /var/www/MISP/app/Console/cake.php(32): ShellDispatcher::run(Array)
#9 {main}
@mkoelewijn
Copy link
Author

Anyone?

@adulau
Copy link
Member

adulau commented Apr 10, 2017

Are the workers running? If not, can you restart those?

@mkoelewijn
Copy link
Author

mkoelewijn commented Apr 10, 2017 •
edited

The workers are not running. Already stated in my Issue that if i try to (re)start the workers i get;

root@misp:/var/www/MISP/app/Model# su - www-data -s /bin/bash -c 'bash /var/www/MISP/app/Console/worker/start.sh'
Error: Redis server went away
#0 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(28): Redis->setOption(2, 'resque:')
#1 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(18): RedisApi->establishConnection()
#2 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Redis.php(168): RedisApi->__construct('localhost', '6379', 5, NULL)
#3 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque.php(102): Resque_Redis->__construct('localhost', '6379', NULL)
#4 /var/www/MISP/app/Plugin/CakeResque/Console/Command/CakeResqueShell.php(77): Resque::redis()
#5 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/Shell.php(450): CakeResqueShell->startup()
#6 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(219): Shell->runCommand('stop', Array)
#7 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(66): ShellDispatcher->dispatch()
#8 /var/www/MISP/app/Console/cake.php(32): ShellDispatcher::run(Array)
#9 {main}

@iglocska
Copy link
Member

If you type redis-cli does the redis cli load fine for you?

@mkoelewijn
Copy link
Author

Also already stated in my issue; "(and yes redis-cli is working fine: 127.0.0.1:6379> )"

@iglocska
Copy link
Member

If you only use redis for MISP, could you run a flushall and see if it persists?

@mkoelewijn
Copy link
Author

I also did that :), didn't make any difference...

@iglocska
Copy link
Member

Hmph. Weird, especially since it worked before, haven't seen this happen yet.

@RichieB2B
Copy link
Contributor

A silly question perhaps, but does localhost resolve to 127.0.0.1 on this system? Since you are testing redis-cli with 127.0.0.1 but resque is using localhost.

Also, is it possible to tcpdump the lo interface to see what is going on at the network level?

tcpdump -n -i lo port 6379

@mkoelewijn
Copy link
Author

melvin@misp:~$ host localhost
localhost has address 127.0.0.1
localhost has IPv6 address ::1

So i first started redis-cli and afterwords i tried restarting the workers:

root@misp:/home/melvin# tcpdump -n -i lo port 6379
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 262144 bytes
11:32:28.402259 IP 127.0.0.1.35562 > 127.0.0.1.6379: Flags [S], seq 2352039417, win 43690, options [mss 65495,sackOK,TS val 129849280 ecr 0,nop,wscale 7], length 0
11:32:28.402272 IP 127.0.0.1.6379 > 127.0.0.1.35562: Flags [S.], seq 3804327221, ack 2352039418, win 43690, options [mss 65495,sackOK,TS val 129849280 ecr 129849280,nop,wscale 7], length 0
11:32:28.402286 IP 127.0.0.1.35562 > 127.0.0.1.6379: Flags [.], ack 1, win 342, options [nop,nop,TS val 129849280 ecr 129849280], length 0
11:32:37.394137 IP 127.0.0.1.35562 > 127.0.0.1.6379: Flags [F.], seq 1, ack 1, win 342, options [nop,nop,TS val 129851528 ecr 129849280], length 0
11:32:37.394493 IP 127.0.0.1.6379 > 127.0.0.1.35562: Flags [F.], seq 1, ack 2, win 342, options [nop,nop,TS val 129851528 ecr 129851528], length 0
11:32:37.394504 IP 127.0.0.1.35562 > 127.0.0.1.6379: Flags [.], ack 2, win 342, options [nop,nop,TS val 129851528 ecr 129851528], length 0
11:33:13.457560 IP6 ::1.48921 > ::1.6379: Flags [S], seq 1342282318, win 43690, options [mss 65476,sackOK,TS val 129860543 ecr 0,nop,wscale 7], length 0
11:33:14.453746 IP6 ::1.48921 > ::1.6379: Flags [S], seq 1342282318, win 43690, options [mss 65476,sackOK,TS val 129860793 ecr 0,nop,wscale 7], length 0
11:33:16.457735 IP6 ::1.48921 > ::1.6379: Flags [S], seq 1342282318, win 43690, options [mss 65476,sackOK,TS val 129861294 ecr 0,nop,wscale 7], length 0
11:33:18.524450 IP6 ::1.48922 > ::1.6379: Flags [S], seq 2216701658, win 43690, options [mss 65476,sackOK,TS val 129861810 ecr 0,nop,wscale 7], length 0
11:33:19.521747 IP6 ::1.48922 > ::1.6379: Flags [S], seq 2216701658, win 43690, options [mss 65476,sackOK,TS val 129862060 ecr 0,nop,wscale 7], length 0
11:33:21.525744 IP6 ::1.48922 > ::1.6379: Flags [S], seq 2216701658, win 43690, options [mss 65476,sackOK,TS val 129862561 ecr 0,nop,wscale 7], length 0
11:33:23.585907 IP6 ::1.48923 > ::1.6379: Flags [S], seq 466317389, win 43690, options [mss 65476,sackOK,TS val 129863076 ecr 0,nop,wscale 7], length 0
11:33:24.585770 IP6 ::1.48923 > ::1.6379: Flags [S], seq 466317389, win 43690, options [mss 65476,sackOK,TS val 129863326 ecr 0,nop,wscale 7], length 0
11:33:26.589750 IP6 ::1.48923 > ::1.6379: Flags [S], seq 466317389, win 43690, options [mss 65476,sackOK,TS val 129863827 ecr 0,nop,wscale 7], length 0
11:33:28.655088 IP6 ::1.48924 > ::1.6379: Flags [S], seq 1995344273, win 43690, options [mss 65476,sackOK,TS val 129864343 ecr 0,nop,wscale 7], length 0
11:33:29.653747 IP6 ::1.48924 > ::1.6379: Flags [S], seq 1995344273, win 43690, options [mss 65476,sackOK,TS val 129864593 ecr 0,nop,wscale 7], length 0
11:33:31.657735 IP6 ::1.48924 > ::1.6379: Flags [S], seq 1995344273, win 43690, options [mss 65476,sackOK,TS val 129865094 ecr 0,nop,wscale 7], length 0
11:33:33.731434 IP6 ::1.48925 > ::1.6379: Flags [S], seq 3135173221, win 43690, options [mss 65476,sackOK,TS val 129865612 ecr 0,nop,wscale 7], length 0
11:33:34.729756 IP6 ::1.48925 > ::1.6379: Flags [S], seq 3135173221, win 43690, options [mss 65476,sackOK,TS val 129865862 ecr 0,nop,wscale 7], length 0
11:33:36.733742 IP6 ::1.48925 > ::1.6379: Flags [S], seq 3135173221, win 43690, options [mss 65476,sackOK,TS val 129866363 ecr 0,nop,wscale 7], length 0
11:33:38.797819 IP6 ::1.48926 > ::1.6379: Flags [S], seq 446281418, win 43690, options [mss 65476,sackOK,TS val 129866879 ecr 0,nop,wscale 7], length 0
11:33:39.797761 IP6 ::1.48926 > ::1.6379: Flags [S], seq 446281418, win 43690, options [mss 65476,sackOK,TS val 129867129 ecr 0,nop,wscale 7], length 0
11:33:41.801749 IP6 ::1.48926 > ::1.6379: Flags [S], seq 446281418, win 43690, options [mss 65476,sackOK,TS val 129867630 ecr 0,nop,wscale 7], length 0

@mkoelewijn
Copy link
Author

Ok this is embarrassing, looks like its a IPv6 FW mistake... Thanks for pointing this out Richie

@adulau adulau mentioned this issue Jan 28, 2018
Closed
@adulau adulau mentioned this issue Nov 22, 2018
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@adulau @RichieB2B @iglocska @mkoelewijn