Error: STIX or CyBox or mixbox library not installed correctly New Ubuntu 16 Install

[krypto29s]

This template is meant for bug reports, if you have a feature request, please be as descriptive as possible and delete the template

If you would like to report a bug, please fill the template bellow

Work environment

Questions	Answers
Type of issue support...
OS version (server) ubuntu
OS version (client) Mac OS -X
PHP version 7.0.18
MISP version / git hash	2.4.77
Browser Chrome

Expected behavior

Stix, Cybox, Mixbox report ok in sever diagnostics

Actual behavior

Mitre's STIX and Cybox python libraries have to be installed in order for MISP's STIX export to work. Make sure that you install them (as described in the MISP installation instructions) if you receive an error below.
If you run into any issues here, make sure that both STIX and CyBox are installed as described in the INSTALL.txt file. The required versions are:
STIX: 1.1.1.4
CyBox: 2.1.0.12
mixbox: 1.0.2
Other versions might work but are not tested / recommended.

STIX and Cybox libraries....STIX or CyBox or mixbox library not installed correctly

Steps to reproduce the behavior

I have logged into the web portal, and click Administration, Server Settings Diagnostics

Logs, screenshots, configuration dump, ...

/var/www/MISP/app/tmp/logs# more exec-errors.log
Traceback (most recent call last):
File "/var/www/MISP/app/files/scripts/misp2stix_framing.py", line 2, in
from misp2cybox import *
File "/var/www/MISP/app/files/scripts/misp2cybox.py", line 1, in
from cybox.core import Object, Observable, ObservableComposition
ImportError: No module named cybox.core

[FloatingGhost]

Using that exact cybox version I cannot replicate the behaviour.

Can you try this?

$ python2

from cybox.core import Object

does that go through without errors?

(I'm assuming MISP uses py2, as that version of cybox doesn't install under py3 ;))

[krypto29s]

python2.7
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
from cybox.core import Object

No errors on the import. What does that mean, and how do I fix it permanently?

[krypto29s]

Current running version of python is
python -V
Python 2.7.12

[krypto29s]

Is there a different version we should run. Is there a different log?

[FloatingGhost]

It means you have an issue with STIX libs and should probably cry a bit

You clearly have cybox installed, but MISP can't access it or something?

That.... shouldn't happen. I can't replicate it, so that's VERY weird.

[krypto29s]

Looks like another issue, I having is the workers stop working randomly.

[kallix]

@krypto29s Are you sure stix and friends were installed system wide ?
Can you confirm your python session with from cybox.core import Object was run from your www-data user ?

[krypto29s]

kallix, it seems that is the actual issue. As the www-data user, we cant run the above that you posted, but we can as any other user. This also seems to affect our misp modules in the same way, as the www-data user is not able to import any python module.

Below is the permissions for the following:

/usr/local/lib/python3.5$ ls -al
total 20
drwxrwsr-x   3 root staff  4096 Apr 25  2016 .
drwxr-xr-x   4 root root   4096 Apr 25  2016 ..
drwxrwsr-x 153 root staff 12288 Jul 14 14:11 dist-packages

/usr/local/lib/python3.5/dist-packages$ ls -al | grep cybox
drwx--S---   9 root staff   4096 Jul 14 10:57 cybox
drwx--S---   2 root staff   4096 Jul 14 14:12 cybox-2.1.0.14.dist-info

/var/www/MISP/app/files/scripts# ls -al
total 112
drwxrws---  7 www-data www-data  4096 Jul 21 09:57 .
drwxrws--- 93 www-data www-data  4096 Jul 24 12:13 ..
-rwxrws---  1 www-data www-data     0 Jul 14 09:03 empty
-rwxrws---  1 www-data www-data  1594 Jul 14 09:03 misp2ciq.py
-rwxrws---  1 www-data www-data 16736 Jul 14 09:03 misp2cybox.py
-rw-r-S---  1 www-data www-data 13917 Jul 21 09:57 misp2cybox.pyc
-rwxrws---  1 www-data www-data  8137 Jul 14 09:03 misp2stix_framing.py
-rwxrws---  1 www-data www-data 14094 Jul 14 09:03 misp2stix.py
drwxrws---  2 www-data www-data  4096 Jul 14 09:03 mispzmq
drwxr-sr-x  8 www-data www-data  4096 Jul 17 13:08 mixbox
drwxr-sr-x  9 www-data www-data  4096 Jul 17 13:06 python-cybox
drwxr-sr-x  9 www-data www-data  4096 Jul 17 13:07 python-stix
-rwxrws---  1 www-data www-data   242 Jul 14 09:03 selftest.php
-rwxrws---  1 www-data www-data  2218 Jul 14 09:03 stixsighting2misp.py
-rwxrws---  1 www-data www-data   421 Jul 14 09:03 stixtest.py
-rwxrws---  1 www-data www-data  6254 Jul 14 09:03 test_payload.txt
drwxrws---  2 www-data www-data  4096 Jul 24 12:13 tmp

[FloatingGhost]

♫Then you didn't follow the install instructions ♫

Go install them as root.

[krypto29s]

When I attempt to import cybox.core as www-data it loos as below. All other users function correctly

>>> from cybox.core import Object
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'cybox.core'

[FloatingGhost]

sudo -u www-data python3 setup.py
y'know
♫follow the install instructions ♫

[FloatingGhost]

or rather python2 I think
Because apparently python3 isn't the default
heh

[krypto29s]

These are what the instructions say

# install Mitre's STIX and its dependencies by running the following commands:
sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev python-setuptools
cd /var/www/MISP/app/files/scripts
sudo -u www-data git clone https://github.com/CybOXProject/python-cybox.git
sudo -u www-data git clone https://github.com/STIXProject/python-stix.git
cd /var/www/MISP/app/files/scripts/python-cybox
sudo -u www-data git checkout v2.1.0.12
sudo python setup.py install
cd /var/www/MISP/app/files/scripts/python-stix
sudo -u www-data git checkout v1.1.1.4
sudo python setup.py install

# install mixbox to accomodate the new STIX dependencies:
cd /var/www/MISP/app/files/scripts/
sudo -u www-data git clone https://github.com/CybOXProject/mixbox.git
cd /var/www/MISP/app/files/scripts/mixbox
sudo -u www-data git checkout v1.0.2
python setup.py install

We followed those. After install the owner and group of everything inside of /scripts was root/root. That was obviously never going to work, so I changed the perms to www-data:www-data.

I am open to doing what makes it work as it should.

[elreydetoda]

@FloatingGhost in the Ubuntu install instructions it says that they need to be owned by www-data

/var/www/MISP/app/files/scripts# ls -la
total 96
drwxrws--- 7 www-data www-data  4096 Jul 12 14:58 .
drwxrws--- 8 www-data www-data  4096 Jul 24 11:54 ..
-rwxrws--- 1 www-data www-data     0 Jul 12 14:56 empty
-rwxrws--- 1 www-data www-data  1594 Jul 12 14:56 misp2ciq.py
-rwxrws--- 1 www-data www-data 16736 Jul 12 14:56 misp2cybox.py
-rwxrws--- 1 www-data www-data  8137 Jul 12 14:56 misp2stix_framing.py
-rwxrws--- 1 www-data www-data 14094 Jul 12 14:56 misp2stix.py
drwxrws--- 2 www-data www-data  4096 Jul 12 14:56 mispzmq
drwxrws--- 5 www-data www-data  4096 Jul 12 14:58 mixbox
drwxrws--- 9 www-data www-data  4096 Jul 12 14:57 python-cybox
drwxrws--- 9 www-data www-data  4096 Jul 12 14:58 python-stix
-rwxrws--- 1 www-data www-data   242 Jul 12 14:56 selftest.php
-rwxrws--- 1 www-data www-data  2218 Jul 12 14:56 stixsighting2misp.py
-rwxrws--- 1 www-data www-data   421 Jul 12 14:56 stixtest.py
-rwxrws--- 1 www-data www-data  6254 Jul 12 14:56 test_payload.txt
drwxrws--- 2 www-data www-data  4096 Jul 24 11:54 tmp

this is on our test server that is working

[krypto29s]

The instructions do not say anything about running as root, or www-data as for the install. You have to sudo to run the python setup. Should i sudo -u www-data python setup.py install?

[kallix]

Two different things :

1. The owner of the files in /var/www/MISP/app/files/scripts -> Might be a problem, but we're not quite there just yet :-)
2. Whether or not cybox is installed in a place available to www-data user
   The documented way to install cybox is to install system-wide (=as root):
   sudo python setup.py install

[krypto29s]

@kallix should we do it as python3, or python 2. default is 2.

[kallix]

python2 (and i believe it must be the default python)

[krypto29s]

@kallix At this point should we delete the contents of scripts and start over, or just install over top of?

[kallix]

Wait until you can actually import cybox from a python interpreter under the www-data user before tackling the next (potential) problem.
Whatever the perms in scripts/, if cybox is not installed where it should, it won't work

[krypto29s]

@kallix I think that is the point we are stuck at. Trying to determine why cybox is not importable as www-data.

[kallix]

Precisely.
And the most likely candidate as to why it cannot be imported is that it is not installed properly.
Which is why you should make sure that cybox is installed properly.
The simplest way is to run sudo python setup.py installfrom your python-cybox directory.
After that, we will know for sure that it is installed correctly

[krypto29s]

@kallix I have executed the task as you suggested.

[kallix]

Now, my hope is that if you run the python interpreter as www-data, the from cybox.core import Object line should execute fine

[krypto29s]

It doesn't work. Same symptom.

[krypto29s]

/usr/local/lib/python2.7/dist-packages/
usr/local/lib/python2.7/dist-packages# ls -al
total 960
drwxrwsr-x 5 root staff 4096 Jul 24 13:21 .
drwxrwsr-x 4 root staff 4096 Jan 5 2017 ..
drwx--S--- 4 root staff 4096 Jul 24 13:16 cybox-2.1.0.12-py2.7.egg
-rw------- 1 root staff 437 Jul 14 09:04 easy-install.pth
drwx--S--- 4 root staff 4096 Jul 14 09:04 lxml-3.8.0-py2.7-linux-x86_64.egg
drwx--S--- 4 root staff 4096 Jul 24 13:21 mixbox-1.0.2-py2.7.egg
-rw------- 1 root staff 6510 Jul 14 09:04 ordered_set-2.0.2-py2.7.egg
-rw------- 1 root staff 262012 Jul 14 09:00 python_dateutil-2.6.1-py2.7.egg
-rw-r--r-- 1 root staff 679592 Jul 24 13:20 stix-1.1.1.4-py2.7.egg
-rw------- 1 root staff 4245 Jul 14 09:04 weakrefmethod-1.0.3-py2.7.egg

[kallix]

from a python interpreter running as www-data, what is the output of

import sys
sys.path

(you may have confidential info there, be cautious)

[krypto29s]

@kallix

Thanks, I purged the above post. thank you for looking out.

>>> import sys
>>> sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-x86_64-linux-gnu', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

[kallix]

Hmm...what if you do
sudo chmod -R ugo+rX /usr/local/lib/python2.7/dist-packages/cybox-2.1.0.12-py2.7.egg

[krypto29s]

sudo chmod -R ugo+rX /usr/local/lib/python2.7/dist-packages/cybox-2.1.0.12-py2.7.egg
sudo -u www-data python
Python 2.7.12 (default, Nov 19 2016, 06:48:10) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from cybox.core import Object
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named cybox.core

[kallix]

Arf.. can you try
sudo chmod go+r /usr/local/lib/python2.7/dist-packages/easy-install.pth

This file definitely has to be readable. That's how python knows what was installed by easy_install (easy_install is the backend program called when doing python setup.py install)

And also check that cybox is present in this file.
The thing I find weird is that the files/dirs in your /usr/local/lib/python2.7/dist-packages/ don't have the typical perms usually set by setup.py

[krypto29s]

This is about as far as i have been able to take it

>>> from cybox.core import Object
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "cybox/__init__.py", line 9, in <module>
    import cybox.bindings as bindings
  File "cybox/bindings/__init__.py", line 10, in <module>
    from lxml import etree as etree_
ImportError: No module named lxml

[krypto29s]

ls -al /usr/local/lib/python2.7/dist-packages/
total 976
drwx--S--- 8 root staff 4096 Jul 24 15:34 .
drwxrwsr-x 4 root staff 4096 Jul 24 14:49 ..
drwxr-sr-x 4 root staff 4096 Jul 24 15:31 cybox-2.1.0.12-py2.7.egg
-rw-r--r-- 1 root staff 296 Jul 24 15:34 easy-install.pth
-rw------- 1 root staff 351 Jul 24 15:16 easy-install.pth.bkp
drwxr-sr-x 5 root staff 4096 Jul 24 15:19 lxml
drwxr-sr-x 2 root staff 4096 Jul 24 15:19 lxml-3.8.0.dist-info
drwx--S--- 4 root staff 4096 Jul 24 14:12 mixbox-1.0.2-py2.7.egg
-rw------- 1 root staff 6510 Jul 24 14:12 ordered_set-2.0.2-py2.7.egg
drwxr-sr-x 10 root staff 4096 Jul 24 15:11 pip
drwxr-sr-x 2 root staff 4096 Jul 24 15:12 pip-9.0.1.dist-info
-rw------- 1 root staff 262012 Jul 24 14:12 python_dateutil-2.6.1-py2.7.egg
-rw------- 1 root staff 679592 Jul 24 14:12 stix-1.1.1.4-py2.7.egg
-rw------- 1 root staff 4245 Jul 24 14:12 weakrefmethod-1.0.3-py2.7.egg

[FloatingGhost]

Wait a gosh darn diddly minute right there

staff

What version of ubuntu are you running?

That... shouldn't be a group on newer versions, I haven't seen that on ubuntu in... ever

[krypto29s]

This was definately a permissions issue. I had to reset the perms for everyone on the dist-packages directory, and cybox/stix/mixbox all show they are running.

[krypto29s]

@FloatingGhost This is on Ubuntu 16.0.4.4

[georgyana]

krypto29s , can you be more specific regarding the permission you used. I have almost the same problem. The files are looking like in your description, except that I'm in python2.7.
I installed the packets like in INSTALL instruction for Debian 9 but I have that error message in the MISP interface.
webadmin@www1:/usr/local/lib/python2.7/dist-packages$ sudo -u www-data python
Python 2.7.13 (default, Jan 19 2017, 14:48:08)
[GCC 6.3.0 20170118] on linux2
Type "help", "copyright", "credits" or "license" for more information.

from cybox.core import Object

---No error in importing cybox module

[krypto29s]

you will notice the dist-packages folder does not have permissions to other. You need read an execute recursive starting on that directory for other.

[georgyana]

Unfortunately didn't solve my problem. Any new ideas?
/usr/local/lib/python2.7/dist-packages$ ls -al
total 960
drwxrwsr-x 5 root staff 4096 Aug 7 19:27 .
drwxrwsr-x 4 root staff 4096 Jul 27 14:05 ..
drwxr-sr-x 4 root staff 4096 Aug 7 19:24 cybox-2.1.0.12-py2.7.egg
-rwxr-xr-x 1 root staff 209 Aug 7 19:27 easy-install.pth
drwxr-sr-x 4 root staff 4096 Aug 5 13:50 lxml-3.8.0-py2.7-linux-x86_64.egg
drwxr-sr-x 4 root staff 4096 Aug 7 19:27 mixbox-1.0.2-py2.7.egg
-rwxr-xr-x 1 root staff 6510 Aug 5 13:51 ordered_set-2.0.2-py2.7.egg
-rwxr-xr-x 1 root staff 261981 Aug 5 13:48 python_dateutil-2.6.1-py2.7.egg
-rwxr-xr-x 1 root staff 679531 Aug 7 19:25 stix-1.1.1.4-py2.7.egg
-rwxr-xr-x 1 root staff 4237 Aug 5 13:51 weakrefmethod-1.0.3-py2.7.egg

[krypto29s]

This was a bit of a chase of permissions.
What happens when you do
python2
from cybox.core import Object

[krypto29s]

Also backup one folder and do a ls -al on /usr/local/lib/python2.7

I am looking for what are the perms on dist-packages itself.

[georgyana]

SOLVED: In Debian 9 PHP 7 is configred for security to disable functions: escapesshellarg, proc_open, proc_close, proc_get_status, shell_exec.
The errors regarding this functions were in var/www/MISP/app/tmp/logs$ tail -f error.log

Therefor I deleted from disable_functions modules presented.
It sims that package where installed correctly, but the web page couldn't test it.

[iglocska]

Interesting, didn't know about the default disabled functions on debian. Nice find!

[BenjaminJoly]

I close the issue as you have resolved it. Feel free to tell me if I should reopen it for any reason.