Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cortex and MISP unclear and error-loop #29

Closed
crackytsi opened this Issue May 19, 2017 · 7 comments

Comments

Projects
None yet
4 participants
@crackytsi
Copy link

crackytsi commented May 19, 2017

Request Type

Bug

Work Environment

Question Answer
OS version (server) Debian 8
Cortex version / git hash 1.1.1-2
Package Type Debian Package

Problem Description

Hello,
Thanks a lot for your really, really good work!!!
Sorry maybe its my fault, but I don't have any further idear, so I use this way to adress it:

  1. My Cortex Config loops persmantenly this messages:
May 19 21:02:46 debian-8-user cortex[19470]: import misp_modules
May 19 21:02:46 debian-8-user cortex[19470]: ImportError: No module named 'misp_modules'
May 19 21:02:46 debian-8-user cortex[19470]: [#033[37minfo#033[0m] application - GET /api/analyzer returned 500
May 19 21:02:46 debian-8-user cortex[19470]: java.lang.RuntimeException: Nonzero exit value: 1
May 19 21:02:46 debian-8-user cortex[19470]: at scala.sys.package$.error(package.scala:27)
May 19 21:02:46 debian-8-user cortex[19470]: at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.slurp(ProcessBuilderImpl.scala:132)
May 19 21:02:46 debian-8-user cortex[19470]: at scala.sys.process.ProcessBuilderImpl$AbstractBuilder.$bang$bang(ProcessBuilderImpl.scala:102)
May 19 21:02:46 debian-8-user cortex[19470]: at services.MispSrv.list$lzycompute(MispSrv.scala:46)
May 19 21:02:46 debian-8-user cortex[19470]: at services.MispSrv.list(MispSrv.scala:45)
May 19 21:02:46 debian-8-user cortex[19470]: at services.AnalyzerSrv.list(AnalyzerSrv.scala:18)
May 19 21:02:46 debian-8-user cortex[19470]: at controllers.AnalyzerCtrl$$anonfun$list$1.apply(AnalyzerCtrl.scala:19)
May 19 21:02:46 debian-8-user cortex[19470]: at controllers.AnalyzerCtrl$$anonfun$list$1.apply(AnalyzerCtrl.scala:18)
May 19 21:02:46 debian-8-user cortex[19470]: at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
May 19 21:02:46 debian-8-user cortex[19470]: at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:370)
May 19 21:02:47 debian-8-user cortex[19470]: Traceback (most recent call last):
May 19 21:02:47 debian-8-user cortex[19470]: File "/opt/cortex/contrib/misp-modules-loader.py", line 10, in <module>

  1. Why are there now 2 different stancas? What do they affect?
analyzer {
  path = "/opt/Cortex-Analyzers/analyzers"
  config {
  ...
} 

AND

misp.modules {
  enabled = true

  config {
  ...
}
  1. Is it possible provide one "full" config file containing all possible parameters (as remark?)
    I'm not sure if the old configuration of MISP (as source) is still correct:
    MISP {
    url="https://server"
    key="mykey"
    certpath=["/etc/ssl/private/misp.local.crt", ""]
    name="instance-1"
    }
@saadkadhi

This comment has been minimized.

Copy link
Contributor

saadkadhi commented May 20, 2017

Hi @crackytsi,

The latest version of Cortex as of this writing (v 1.1.1) lets you invoke MISP expansion modules if you'd like, as described in the MISP Integration Guide. That is the reason why there is now a separate stanza for them. We prefer to keep the configuration of Cortex native analyzers separate from the configuration of the MISP expansion modules.

I successfully reproduced your problem. It is due to the fact that the MISP modules are enabled by default and Cortex is looking for them. Please disable the MISP modules in application.conf:

misp.modules {
  enabled = false
...

And restart Cortex. This should fix your problem temporarily. Now that we are aware of this issue, we are going to look for a permanent solution very soon.

Thank you for your nice comments about our work. We should held ourselves to better QA standards though to prevent such errors from happening in the first place. This will be our priority in the next few weeks.

@saadkadhi saadkadhi added the bug label May 20, 2017

@saadkadhi saadkadhi added this to the 1.1.2 milestone May 20, 2017

@crackytsi

This comment has been minimized.

Copy link
Author

crackytsi commented May 20, 2017

Thank you so much. I understand it now :)

@crackytsi

This comment has been minimized.

Copy link
Author

crackytsi commented May 22, 2017

Hi,
I installed the plugins from MISP as described.
Never the less I'm still faced with issues (also in a loop):
I have no idear where uwhois is defined (a grep did not help me).
Any hint?


2017-05-22 10:36:33,228 [INFO] from application in application-akka.actor.default-dispatcher-5 - GET /api/analyzer returned 500
com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'uwhois': was expecting ('true', 'false' or 'null')
 at [Source: uwhois module not installed.
["wiki", "asn_history", "dns", "sourcecache", "eupi", "whois", "circl_passivedns", "virustotal", "cve", "shodan", "circl_passivessl", "geoip_country", "ipasn", "passivetotal", "domaintools", "iprep", "reversedns", "countrycode", "vmray_submit", "threatminer"]
; line: 1, column: 7]
        at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586)
        at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._reportInvalidToken(ReaderBasedJsonParser.java:2749)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleOddValue(ReaderBasedJsonParser.java:1820)
        at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:708)
        at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3847)
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:3765)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2050)
        at play.api.libs.json.jackson.JacksonJson$.parseJsValue(JacksonJson.scala:238)
        at play.api.libs.json.Json$.parse(Json.scala:21)
        at services.MispSrv.list$lzycompute(MispSrv.scala:46)
        at services.MispSrv.list(MispSrv.scala:45)
        at services.AnalyzerSrv.list(AnalyzerSrv.scala:18)
        at controllers.AnalyzerCtrl$$anonfun$list$1.apply(AnalyzerCtrl.scala:19)
        at controllers.AnalyzerCtrl$$anonfun$list$1.apply(AnalyzerCtrl.scala:18)
        at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:371)
        at play.api.mvc.ActionBuilder$$anonfun$apply$13.apply(Action.scala:370)
        at play.api.mvc.Action$.invokeBlock(Action.scala:498)
        at play.api.mvc.Action$.invokeBlock(Action.scala:495)
        at play.api.mvc.ActionBuilder$$anon$2.apply(Action.scala:458)
        at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonfun$apply$6.apply(Action.scala:112)
        at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5$$anonfun$apply$6.apply(Action.scala:112)
        at play.utils.Threads$.withContextClassLoader(Threads.scala:21)
        at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(Action.scala:111)
        at play.api.mvc.Action$$anonfun$apply$2$$anonfun$apply$5.apply(Action.scala:110)
        at scala.Option.map(Option.scala:146)
        at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:110)
        at play.api.mvc.Action$$anonfun$apply$2.apply(Action.scala:103)
        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:253)
        at scala.concurrent.Future$$anonfun$flatMap$1.apply(Future.scala:251)
        at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
        at akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
        at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
        at akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:409)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

@nadouani

This comment has been minimized.

Copy link
Contributor

nadouani commented May 22, 2017

This uwhois is a dependency of one of the misp-modules

@crackytsi

This comment has been minimized.

Copy link
Author

crackytsi commented May 22, 2017

Hmmm. But I disabled all except dns lookup...

@nadouani

This comment has been minimized.

Copy link
Contributor

nadouani commented May 22, 2017

Well, When misp-modules is enabled in Cortex, the latter will try to load the list of all misp-modules available (undependently from what modules you have configured)

When listing the misp modules, the operation fail because of the missing uwhois package.

The issue is that this package is not available on PIP (it's a fork a another uwhoisd project...)

To get the uwhois installed, make sure your pip is up to date:

sudo pip install pip --upgrade

and then run

sudo pip install 'git+https://github.com/Rafiot/uwhoisd.git@testing#egg=uwhois&subdirectory=client'
@crackytsi

This comment has been minimized.

Copy link
Author

crackytsi commented May 22, 2017

Thanks, I got it. At the time time you wrote this, I finally found this commit: Rafiot/uwhoisd@3f16d42 :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.