Cannot subscribe to MISP and Failed to start MISP plugin

[nillay]

Hello sir,
I am very new to the threatbus and misp. So sincere apology for asking the naive question here.
I am trying to figure out how threatbus works with zeek and misp. Also I referred the youtube video put my tenzir team regarding that. I am using misp and zeek all on same host inside a ubuntu 20.04 vm. I am able to access the misp through https://localhost from inside And using https://localhost:8443 from outside vm. I am struggling through the following problem, I will be very grateful if someone can take a look about why I am getting this error:

Observed Behavior

(venv) root@ubuntu-pc:/home/ubuntu# threatbus -c config-new.yaml
2020-12-09 20:52:48 INFO [threatbus] Starting plugins...
2020-12-09 20:52:48 INFO [threatbus_inmem.plugin] In-memory backbone started.
2020-12-09 20:52:48 INFO [threatbus_inmem.plugin] Adding subscription to: threatbus/snapshotrequest
2020-12-09 20:52:48 INFO [threatbus_inmem.plugin] Adding subscription to: threatbus/snapshotenvelope
2020-12-09 20:52:48 INFO [threatbus_zeek.plugin] Zeek plugin started
2020-12-09 20:52:48 ERROR [threatbus_misp.plugin] Cannot subscribe to MISP at https://localhost, using SSL: True
2020-12-09 20:52:48 ERROR [threatbus_misp.plugin] Failed to start MISP plugin

• Version:
• Compiler:
• Operating System: Ubuntu 20.04
• ..

[0snap]

Hi @nillay
I need some more information from you to debug this.

• Can you please paste the contents of your config.yaml file? You can redact the MISP API key.
• How did you setup and install MISP?
• What Python version are you using?

[nillay]

Sir,
thanks a lot for your reply. Sir this is version I am using in python:

============
(venv) root@ubuntu-pc:/home/ubuntu# python
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

============

config-new.yaml content:
logging:
console: true
console_verbosity: DEBUG
file: false
file_verbosity: DEBUG
filename: threatbus.log

plugins:
backbones:
inmem:
apps:
zeek:
host: "127.0.0.1"
port: 47760
module_namespace: Tenzir
misp:
api:
host: https://localhost
ssl: true
key: GBV9jL***********************IJyWXxoUri7X2tfgkcd3N
zmq:
host: localhost
port: 50000

Sir as suggested in here. I have installed misp using:

===================
Sir I installed misp using following command ): 
# Please check the installer options first to make the best choice for your install
wget -O /tmp/INSTALL.sh https://raw.githubusercontent.com/MISP/MISP/2.4/INSTALL/INSTALL.sh
bash /tmp/INSTALL.sh

# This will install MISP Core
wget -O /tmp/INSTALL.sh https://raw.githubusercontent.com/MISP/MISP/2.4/INSTALL/INSTALL.sh
bash /tmp/INSTALL.sh -c
=============

[0snap]

Can you try setting ssl: false in the misp section of the config.yaml file, please?

[nillay]

Wow, sir magically it started. But my question is that even if my misp is running on https:// will it affect the working of the threatbus+misp integration ?

[nillay]

Thanks a lot @0ortmann sir. I can't thank you enough for helping me out, in which I was stuck for so long.

[0snap]

In the default installation, MISP creates a self-signed certificate. The certificate check errors, because it is not a valid certificate. Using ssl: false disables that the Python MISP library (PyMISP) checks the certificate. That does not influence the MISP<->Threat Bus communication.

[0snap]

You're very welcome :)

[nillay]

Thank you @0ortmann sir for such an crucial information from bottom of my heart. A great learning for me today. Sir, You really made my day.

[nillay]

@0ortmann sir, sorry to bother you again but I am little stuck in one more issue.

Sir I will be very thankful if you could guide me for the same, I know I am doing very silly mistake like before, my apologies for same:

So with threatbus plugin, I am successfully able to detect the intel into the zeek and threatbus... Although I am not sure that will these intel will be deleted when we remove it from MISP, or some other config will be required to do that !

But the primary issue I am facing is that "After sighting got recorded in zeek" as well as "Threatbus" console. Misp is "Not" reflecting the same. Its stuck to 0 count. I published the event with IDS checkbox "on". but sighting not been updating to MISP.
Sir, sorry again for such a silly question. Please pardon me for the same.

Attaching screenshot here:

Kind regards

[0snap]

Hi @nillay
I see you closed this issue again, did you resolve the problem?
It could be that the MISP web view simply did not update. Does it still show the wrong count when you reload the page?

[nillay]

Hello @0ortmann sir,
Thanks for your reply. No sir issue didn't solved. I thought my question is silly. So closed it because of that. Although problem is not solved yet.

[0snap]

From the logs in the screenshot you provided, we can see that the MISP plugin reports back the sighting to your MISP instance. When you reload the MISP event page in your browser (CRTL+Shift+R), does the it show the sighting?

[nillay]

No sir no sighting shown in MISP. Reloaded multiple times. Zeek and threatbus talking perfectly. Generating Intel logs and so on. But MISP seems to into 1-Way communication mode.

[0snap]

I will try to reproduce the behavior with the MISP version I see in your screenshot 2.4.135. I'll get back to you by end of the day (CET).

[nillay]

@0ortmann Thank you sir.

[0snap]

Hi @nillay
I reconstructed the setup using with the following application versions:

• MISP 2.4.135
• Zeek v3.2.0
• Threat Bus 2020.11.26 and Threat Bus GitHub master

Unfortunately, I cannot reproduce your error. Sightings get reported back to MISP as they should. We just merged a PR #87 to add error logging in case the sighting reporting to MISP errors.

Can I kindly ask you to install the latest Threat Bus version from GitHub master and retry? In case you see error logs, please paste them in this GitHub issue.
Cheers!

[nillay]

sure sir doing the same, will let you know asap. thanks a lot for checking the issue from your side. @0ortmann.

[nillay]

@0ortmann Sir I pulled the threatbus, threatbus misp, threatbus inmem from pip3..... can you please refer me to the document where i can manully build all three tools from source code..... or can you suggest me the right way to install the most updated threatbus..... ? Also is there a way to check the current running version, if yes please let me know. I will be very grateful for you. thanks.

[nillay]

@0ortmann Sir seems like I am already into:
Threat Bus: 2020.11.26
Zeek Version: 3.2.2
MISP Version: 2.4.135

[mavam]

I don't want to derail the conversation, and have limited ability to aid in debugging, but it looks like there's an error on the Zeek side: can show the contents of threatbus2.zeek near line 255, e.g., +/- 10 lines?

[nillay]

@mavam Sir, my apology, actually that second tab is not related to the that, I am doing some log related experiment in zeek script, so I duplicated the zeek script for that. Otherwise with original script its working fine. But I thank you for adding your observation to it. Please guide me through in handling the issue. These screenshots are only meant to show the versions i am running.....
Attaching the working version below:

[nillay]

@0ortmann Sir, After re-updating everything I am facing following error, can you help me out in what I am missing: Seems like Zeek in not seeing threatbus for some reason: But MISP talking to threatbus.... And threatbus opened a listening port too....
Thanks a lot.

I have also enabled the verbose on broker.conf
and getting the debug messages as follows w.r.t. above mentioned error:

[0snap]

Hi @nillay

You seem to run into multiple issues at once. When you first opened this issue, the Threat Bus <-> Zeek connection worked fine as can be seen in the first screenshot you provided here. Please restore your setup back to that point.

As per the question how you can install Threat Bus from source, without PyPI:

• clone the tenzir/threatbus repository via git clone git@github.com:tenzir/threatbus.git
• navigate to the repository root folder cd threatbus
• install the plugin directly from source: pip3 install . && pip3 install plugins/apps/threatbus_misp && pip3 install plugins/apps/threatbus_zeek && pip3 install plugins/backbones/threatbus_inmem

[nillay]

@0ortmann Sir thank you for your revert. I am trying hard to restore the setup of last known good config but I seems like no luck in that. Just tried as per recommendation #and still getting the same error, sir, should I create a separate thread for this issue ? Also If you think I should try it in fresh installed system then I will try to do that too please let me know, Thanks :

Hi @nillay

You seem to run into multiple issues at once. When you first opened this issue, the Threat Bus <-> Zeek connection worked fine as can be seen in the first screenshot you provided here. Please restore your setup back to that point.

As per the question how you can install Threat Bus from source, without pip3:

* clone the [tenzir/threatbus](https://github.com/tenzir/threatbus/) repository via `git clone git@github.com:tenzir/threatbus.git`

* navigate to the repository root folder `cd threatbus`

* install the plugin directly from source: `pip3 install . && pip3 install plugins/apps/threatbus_misp && pip3 install plugins/apps/threatbus_zeek && pip3 install plugins/backbones/threatbus_inmem`

[nillay]

@0ortmann sir, Just checked, seems like no changes in sighting, only modification graph which was flat has been change for once... Thanks a lot.

[0snap]

@nillay The Modification map shows an upwards trend. That correlates with the Threat Bus logs, which indicate a successful reporting. This all seems to be a visibility issue in your MISP instance.

Dumb question: when you manually add a Sighting in your MISP web-interface, does it show up? Also, when you login as Administrator in MISP, can you then see the Sightings?

[nillay]

@nillay The Modification map shows an upwards trend. That correlates with the Threat Bus logs, which indicate a successful reporting. This all seems to be a visibility issue in your MISP instance.

Dumb question: when you manually add a Sighting in your MISP web-interface, does it show up? Also, when you login as Administrator in MISP, can you then see the Sightings?

@0ortmann sir, sounds interesting. Yeap sir after being frustrated a lot I started added the sighting manually when sighting was occurred 😁 at least I was feeling little better while doing so 😆
Also sir since I am new to MISP too I am not taking any risk of permissions, so I am logging
in using admin account in MISP. Thanks a lot sir. 🙏

[0snap]

Does the web-interface show the sighting that you added manually?

[nillay]

Does the web-interface show the sighting that you added manually?

Sorry @0ortmann sir, just saw the message. somehow I missed this message. Yes sir, manual sighting is shown in web-interface. Please accept my apology for the late reply.

[0snap]

Hi @nillay

I now extended the misp testutil so it also reports an artificial sighting. Could you please download the updated script and run it again? Please note that you need to update the config.yaml to include report_sighting: true (see the example config here).

Please paste the logs in this issue. It should create an attribute (if it does not exist already) and report a sighting.

[nillay]

@0ortmann sir, thanks a lot for providing the updated script. I ran this one and found that with this script MISP is reflecting the sighting properly. Ran this script 2 times and got 2 sightings back.
Please take a look at the screenshot attached below: Thank you very much.

[0snap]

Hi @nillay
Thanks for running the script. I'm at a loss understanding why the sightings from Threat Bus don't show up in the UI, while this test-sighting did.

Could you please run another test?

• Start Zeek to monitor your standard network interface
• Start Threat Bus as you did in your original question, with MISP and Zeek connected.
• Run the test script again -> that should result in Threat Bus forwarding the IoC to Zeek
• Now do a simple curl test-2020-12-23.vast (that is the test IoC from the script) -> that should trigger a sighting in Zeek, that forwards it to Threat Bus and ideally it would show up in MISP.

[nillay]

@0ortmann sir, I tried as you instructed, so here is the outcome as follows:

• I started the threatbus
• Then the zeek for monitoring the test interface
• Then I ran the script you have provided
• Two things happened simultaneously
  • a. threatbus and zeek both registered the ioc
  • b. Misp created new ioc and _updated the count to 1
• Then I went to my test interface and generated a event using the curl command as "curl test-2020-12-23.vast"
• This time threatbus and zeek both responded when sighting occcured.
• But Misp is "Not" updated the sighting count. It is frozen to 1, which was increased from 0 to 1 while triggering the test misp-ioc script.

Attaching the screenshots for the same.
Thank you.

[0snap]

Hi @nillay
thanks for your patience and all the tests you have done so far! I really appreciate that :)
I created a new debug branch for the MISP plugin. Please install that branch and run the above test scenario again. The updated version will log the MISP response from the API call to report the sighting.
You can install the branch like this:

pip uninstall threatbus-misp
pip install git+https://github.com/tenzir/threatbus.git@topic/misp-debug#subdirectory=plugins/apps/threatbus_misp

Once installed, please run the test again and post the new Threat Bus logs.

[nillay]

@0ortmann sir, thanks a lot for your kind words. Sir, I am the newbie in these domain. So I am grateful that you are giving your precious time and effort in finding the root cause of the issue. I have no words to thank you. In this noble work, if I can become useful in anyway, I will consider myself lucky enough.

Sir, I have ran the procedures as you instructed, Please find the screenshots of the same.Although its was just for debugging as you said but just for the record Count is still not increased.

Kind Regards
Nillay

[0snap]

Hi @nillay,

Happy new year and thanks for your patience! Were you able to make any progress yet?
I have pushed another debug version of the MISP plugin to the debug branch. It produces more logs. Can you please run that again, using the same instructions as above?

pip uninstall threatbus-misp
pip install git+https://github.com/tenzir/threatbus.git@topic/misp-debug#subdirectory=plugins/apps/threatbus_misp

Then generate a sighting with Zeek as we did before.

[nillay]

Hello @0snap sir,
Wishing you a warm Happy new year 2021. Thank you for you kind reply. Sir, actually I am kind of clueless about the misp sighting issue. So I'm still stuck at the same spot. Also I had one question in mind, pardon me if its not related to this issue.

• What if I want to use MISP which is not on the same machine as threatbus ? Do I need to do some specific changes in order to do so or in the config.yaml just need to mention the ip of the MISP host in place of 'localhost' ?

• Now coming to the updates, sir I re-ran all the requirements as your instructed. Please find the screenshots below for the same. just for the records, MISP is still not updating the count.

[0snap]

Thanks for running the test, @nillay
I'm trying to figure out what the problem is with MISP not updating the sighting. From the new DEBUG logs we can see that

• The MISP API is queried, the API TOKEN is correct
• MISP responds with a generic message that prints a description for the API endpoint. That message usually indicates that the API call is executed without parameters. So now we log the JSON object that is sent to MISP.

Going forward with the debug output from the new logs, can you please run the following cURL command against your MISP instance?

curl --insecure --header "Authorization: <API_TOKEN> " --header "Accept: application/json" --header "Content-Type: application/json" https://<MISP_IP>/sightings/add -d '{"id": 504, "type": 0, "timestamp": "1609755019"}' 

Please replace <API_TOKEN> and <MISP_IP> with the actual values of your installation.
The command instructs MISP to do the same as Threat Bus does: it should add a sighting to the Attribute with id 504.
When you run that command, what is the API response from MISP? Does it add a sighting?

To answer your other question:

• MISP and Threat Bus are designed to run on different host systems. Simply place the correct IP address in the config.yaml.

[nillay]

@0snap sir, thank you for your kind reply. After running the API call seems like its not responding to MISP for some reasons.
no ioc has been added.

[0snap]

Hi @nillay,
The cURL command was successful. We can see that, because the API returns the Sighting JSON structure. When you check the event with ID 89, it should have new sightings attached, one for each of the cURL commands you have run.
Do these sightings show up in the Web UI?

[nillay]

Hi @nillay,

The cURL command was successful. We can see that, because the API returns the Sighting JSON structure. When you check the event with ID 89, it should have new sightings attached, one for each of the cURL commands you have run.

Do these sightings show up in the Web UI?

Thanks sir for reply. No sir, No sightings were attached in webui.

[0snap]

When the web-ui does not show the sightings, even when added directly with cURL, this is an issue for the MISP Project.

How did you install MISP? I still cannot reproduce this issue (I use the pre-built virtual machines provided on the MISP website, i.e., this one).

[nillay]

When the web-ui does not show the sightings, even when added directly with cURL, this is an issue for the MISP Project.

How did you install MISP? I still cannot reproduce this issue (I use the pre-built virtual machines provided on the MISP website, i.e., this one).

Sir I installed from misp script provided in their documentation. I think they created 2 shell scripts for the same. Seems like some internal config issue is occurring when we are not using their prebuilt MISP.

Sir let me use their Vm and hopefully then I don't need to manually look under the hood.

Will update you about at first either I am able to connect the misp MV with threatbus and secondly if it's responding well or not. Thanks a lot sir taking your precious time and efforts to look into my issue. It really means a lot to me. 🙏

[nillay]

@0snap sir,
I just setup the cicle misp vm. And for some reasons on misp vm when I am adding an ioc. Its not reflecting on threatbus, seems like I am doing something wrong. I will be thankful if you could guide me to solve the issue. Thank you.
Attaching screenshots below:

[0snap]

Hi @nillay
Your Threat Bus config.yaml shows a wrong value for the zmq MISP endpoint - it should use the IP address from the MISP instance, not localhost.

misp:
 api:
  ...
 zmq:
  host: 10.0.3.9       # <- this
  port: 50000

You also need to enable the zmq plugin:

• Go to Administration -> Server Settings & Maintenance -> Diagnostics Tab
• Find the ZeroMQ plugin section and enable it
• Go to Administration -> Server Settings & Maintenance -> Plugin settings Tab
• Set the entry Plugin.ZeroMQ_attribute_notifications_enable to true

[nillay]

@0snap Sir,
Thank you very much. I was doing silly mistake. Your solution worked like charm. But the issue in Circle Misp is looking as same as my local MISP copy. Now sightings are reflected in threatbus and zeek when I add them in Misp. But agan Misp sight count is not increasing. I have checked the sighting settings and set it for everyone for now.

Also I gave curl command as in previous instruction you gave, throwing some kind of error:
Attaching the screenshot for the same:
Thank a lots sir again.

Kind Regards

[0snap]

Hi @nillay

Thanks for your tests. In the last curl command, please replace the "id": 504 with "id": 4. Since you now use a new MISP installation, you need to update the attribute ID. We can see the ID in the Threat Bus logs.

[nillay]

@0snap sir,
Thank you for your kind reply.
Please find the output below when I ran curl command, Thanks:

[0snap]

The curl commands indicate success, they return a Sighting JSON structure. Did the sightings web-view update, i.e., are these sightings visible now?

[nillay]

no sir, sightings are not visible, even new event or any attribute is not added.....

[0snap]

Hi @nillay,

Sorry, I'm at a loss understanding why these sightings won't show up in your MISP web view.
This does not seem to relate to Threat Bus, because even though the cURL response indicates success, your sightings view is not updated. Please report the issue in the MISP repo. Please use the curl command as example in the new issue.

[nillay]

Hi @nillay,

Sorry, I'm at a loss understanding why these sightings won't show up in your MISP web view.
This does not seem to relate to Threat Bus, because even though the cURL response indicates success, your sightings view is not updated. Please report the issue in the MISP repo. Please use the curl command as example in the new issue.

Hi @0snap sir,
Sure. But I would like to thank you from bottom of my heart for the humongous time and efforts you spend in helping with my issue. It means a lot and I learnt a lot in your company. It was a truly special communication for me. And will always be. So Thank you again for everything.... And my apology for any inconvenience caused to you because of me.

Kind Regards,
Nillay

[0snap]

You're welcome! Please come over to our chat for future questions about running Threat Bus.