emba installation with proxy

[torabi12]

Describe the bug
I am using kali 2021.4 and I have to set up proxy to get internet access. I set the following parameters on my linux:
HTTP_PROXY=A.B.C.D:1234
HTTPS_PROXY=A.B.C.D:1234
NO_PROXY=localhost,127.0.,127.36.

echo "http_proxy=http://${HTTP_PROXY}/" >>/etc/environment &&
echo "https_proxy=http://${HTTPS_PROXY}/" >>/etc/environment &&
echo "no_proxy=${NO_PROXY}" >>/etc/environment

I set the proxy for APT:
touch /etc/apt/apt.conf.d/proxy.conf &&
echo "Acquire::http::Proxy "http://${HTTP_PROXY}";" >>/etc/apt/apt.conf.d/proxy.conf &&
echo "Acquire::https::Proxy "http://${HTTPS_PROXY}";" >>/etc/apt/apt.conf.d/proxy.conf

export:
export http_proxy=${HTTP_PROXY}
export https_proxy=${HTTPS_PROXY}
export no_proxy=${NO_PROXY}

proxy for wget as root:
cp /etc/wgetrc /root/.wgetrc
echo -e "use_proxy = on\nhttp_proxy = http://${HTTP_PROXY}\nhttps_proxy = http://${HTTPS_PROXY}\nftp_proxy = http://${HTTP_PROXY}" >>/root/.wgetrc

proxy for wget as kali:
cp /etc/wgetrc /home/kali/.wgetrc
echo -e "use_proxy = on\nhttp_proxy = http://${HTTP_PROXY}\nhttps_proxy = http://${HTTPS_PROXY}\nftp_proxy = http://${HTTP_PROXY}" >>/home/kali/.wgetrc

and I set proxy for Git:
git config --global http.proxy http://${HTTP_PROXY}
git config --global https.proxy http://${HTTPS_PROXY}

apt upgrade and reboot, and I git lone the emba, then cd ~emba, then:
sudo ./isntaller.sh -d

All the things are fine but after mongoDB installation the setup failes. This is on the screen:

Adding user mongodb' to group mongodb' ...
Adding user mongodb to group mongodb
Done.
Setting up mongodb-org-shell (4.4.12) ...
Setting up mongodb-database-tools (100.5.2) ...
Setting up mongodb-org-mongos (4.4.12) ...
Setting up mongodb-org-database-tools-extra (4.4.12) ...
Setting up mongodb-org-tools (4.4.12) ...
Setting up mongodb-org (4.4.12) ...
Processing triggers for man-db (2.9.4-2) ...
Processing triggers for kali-menu (2021.4.2) ...
Scanning processes...
Scanning linux images...

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.
Created symlink /etc/systemd/system/multi-user.target.wants/mongod.service → /lib/systemd/system/mongod.service.

The cve-search database will be downloaded and updated!

Check if the cve-search database is already installed.

cve-search database not ready.

The installer is going to populate the database.
Starting redis-server (via systemctl): redis-server.service.
Traceback (most recent call last):
File "/home/kali/emba/external/cve-search/./sbin/db_mgmt_cpe_dictionary.py", line 27, in
from lib.Sources_process import CPEDownloads
File "/home/kali/emba/external/cve-search/sbin/../lib/Sources_process.py", line 13, in
from pymongo import TEXT, ASCENDING
ModuleNotFoundError: No module named 'pymongo'
Traceback (most recent call last):
File "/home/kali/emba/external/cve-search/./sbin/db_mgmt_json.py", line 22, in
from lib.Sources_process import CVEDownloads
File "/home/kali/emba/external/cve-search/sbin/../lib/Sources_process.py", line 13, in
from pymongo import TEXT, ASCENDING
ModuleNotFoundError: No module named 'pymongo'
Traceback (most recent call last):
File "/home/kali/emba/external/cve-search/./sbin/db_updater.py", line 21, in
from lib.DatabaseSchemaChecker import SchemaChecker
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseSchemaChecker.py", line 7, in
from lib.DatabaseHandler import DatabaseHandler
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseHandler.py", line 1, in
from lib.ApiRequests import JSONApiRequest
File "/home/kali/emba/external/cve-search/sbin/../lib/ApiRequests.py", line 5, in
from nested_lookup import nested_lookup, nested_update
ModuleNotFoundError: No module named 'nested_lookup'

The cron.daily update script for EMBA is located in config/emba_updater
For automatic updates it should be copied to /etc/cron.daily/

Installation notes:

INFO: The cron.daily update script for EMBA is located in config/emba_updater
INFO: For automatic updates it should be copied to /etc/cron.daily/
INFO: For manual updates just start it via sudo ./config/emba_updater

WARNING: If you plan using the emulator (-E switch) your host and your internal network needs to be protected.

INFO: Do not forget to checkout current development of EMBA at https://github.com/e-m-b-a.
EMBA installation finished

and I get back the prompt.
Could you help what other proxy settings required to get the installation go through the entire process?

To Reproduce
Steps to reproduce the behavior:

1. Proxy setup
2. EMBA installation (default mode)
3. See error

Expected behavior
Please verify my proxy settings and if possible please give any advice what to do for population the CVE database by the instalation script as kali user:
$ sudo ./installer.sh -d

Screenshots
I copied the error message above.

Desktop (please complete the following information):

• OS: [Kali Linux 2021.04]

Additional context
Proxy must be set but I am not sure I defined all the needed parameters for the emba installer.

[m-1-k-3]

I have currently not tried to install EMBA with a proxy. So let's try it ;)

First, could you try to update/populate the cve-database manually:
$ sudo ./external/cve-search/sbin/db_updater.py -v
I would also recommend using the -f parameter to ensure to get a clean database.

[torabi12]

I tried it but I get an error:

$ sudo ./external/cve-search/sbin/db_updater.py -v -f
[sudo] password for kali:
Traceback (most recent call last):
File "/home/kali/emba/./external/cve-search/sbin/db_updater.py", line 21, in
from lib.DatabaseSchemaChecker import SchemaChecker
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseSchemaChecker.py", line 7, in
from lib.DatabaseHandler import DatabaseHandler
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseHandler.py", line 1, in
from lib.ApiRequests import JSONApiRequest
File "/home/kali/emba/external/cve-search/sbin/../lib/ApiRequests.py", line 5, in
from nested_lookup import nested_lookup, nested_update
ModuleNotFoundError: No module named 'nested_lookup'

I did the same installation without proxy on a different machine and it worked well, but I would like to use the emba in a powerful company workstation and I must use proxy to get any internet access.

[m-1-k-3]

Is your installation corrupt? Could you do a quick check if the rest of cve-search is correctly installed.
The following command should show nested-lookup:
└─$ pip3 list | grep nested nested-lookup 0.2.23
If there is no output, then you probably need to re-install the complete cve-search environment as described here: https://cve-search.github.io/cve-search/getting_started/installation.html#dependencies

[torabi12]

Right now it gives this:
$ pip3 list | grep nested nested-lookup 0.2.23
grep: nested-lookup: No such file or directory
grep: 0.2.23: No such file or directory
ERROR: Pipe to stdout was broken
Exception ignored in: <_io.TextIOWrapper name='' mode='w' encoding='utf-8'>
BrokenPipeError: [Errno 32] Broken pipe

What I have done before:
Took a a new kali linux:
https://kali.download/virtual-images/kali-2021.4/kali-linux-2021.4-vmware-amd64.7z

Opened in VMware Workstation Pro.
Change shell to bash. for root and kali users:

chsh -s /bin/bash

chsh -s /bin/bash kali

Set the proxy settings as described in the original message above. Insatlled docker and set the proxy as well for docker. Then:
$ sudp apt upgrade
reboot
$ git clone https://github.com/e-m-b-a/emba.git
$ cd ~emba
$ sudo ./install.sh -d

and the installer was not able to install pip requirements in this case. If I do the same without proxy on a different machine it will works fine.
Now I start the whole process from the beginning and I will manually install cve-search before staring emba installer:

$ sudo xargs apt-get install -y < requirements.system
$ pip3 install --proxy http://A.B.C.D:1234 -r requirements.txt

I don't know how to set up proxy for pip to get the emba installer use it automatically.

[torabi12]

Now I have this:
$ pip3 list | grep nested
nested-lookup 0.2.23

[m-1-k-3]

Is the python error now gone and can you update your cve-search database now?

[torabi12]

I tried it but I couldn't update the database:

$ pip3 list | grep nested
nested-lookup 0.2.23

┌──(kali㉿kali)-[~/emba]
└─$ sudo ./external/cve-search/sbin/db_updater.py -v -f
[sudo] password for kali:
Traceback (most recent call last):
File "/home/kali/emba/./external/cve-search/sbin/db_updater.py", line 21, in
from lib.DatabaseSchemaChecker import SchemaChecker
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseSchemaChecker.py", line 7, in
from lib.DatabaseHandler import DatabaseHandler
File "/home/kali/emba/external/cve-search/sbin/../lib/DatabaseHandler.py", line 1, in
from lib.ApiRequests import JSONApiRequest
File "/home/kali/emba/external/cve-search/sbin/../lib/ApiRequests.py", line 5, in
from nested_lookup import nested_lookup, nested_update
ModuleNotFoundError: No module named 'nested_lookup'

I repeated the whole installation process and the setup just stopped again the same point with the error above.

If I do the same as root user it is working but as kali I am not able to install emba.

[torabi12]

I also tried this command:

$ ./sbin/db_mgmt_cpe_dictionary.py -p
Traceback (most recent call last):
File "/home/kali/emba/external/cve-search/./sbin/db_mgmt_cpe_dictionary.py", line 47, in
cpd = CPEDownloads()
File "/home/kali/emba/external/cve-search/sbin/../lib/Sources_process.py", line 51, in init
super().init(self.feed_type, self.prefix)
File "/home/kali/emba/external/cve-search/sbin/../lib/JSONFileHandler.py", line 10, in init
super().init(feed_type)
File "/home/kali/emba/external/cve-search/sbin/../lib/DownloadHandler.py", line 52, in init
self.file_queue.clear()
File "/home/kali/emba/external/cve-search/sbin/../lib/redis_q.py", line 45, in clear
self.__db.delete(self.key)
File "/home/kali/.local/lib/python3.9/site-packages/redis/commands/core.py", line 972, in delete
return self.execute_command('DEL', *names)
File "/home/kali/.local/lib/python3.9/site-packages/redis/client.py", line 1068, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 1173, in get_connection
connection.connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 575, in connect
self.on_connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 653, in on_connect
auth_response = self.read_response()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 765, in read_response
raise response
redis.exceptions.ResponseError: WRONGPASS invalid username-password pair

but I got an error again:(

[BenediktMKuehne]

pip packages are not global

[torabi12]

I see. I installed the packages as root & kali:

(root💀kali)-[/home/kali/emba/cve-search]
└─# pip3 freeze | grep pymongo
pymongo==3.12.1

┌──(root💀kali)-[/home/kali/emba/cve-search]
└─# exit
logout

┌──(kali㉿kali)-[~/emba/external/cve-search]
└─$ pip3 freeze | grep pymongo
pymongo==3.12.1

┌──(kali㉿kali)-[~/emba/external/cve-search]
└─$ ./sbin/db_mgmt_cpe_dictionary.py -p
Traceback (most recent call last):
File "/home/kali/emba/external/cve-search/./sbin/db_mgmt_cpe_dictionary.py", line 47, in
cpd = CPEDownloads()
File "/home/kali/emba/external/cve-search/sbin/../lib/Sources_process.py", line 51, in init
super().init(self.feed_type, self.prefix)
File "/home/kali/emba/external/cve-search/sbin/../lib/JSONFileHandler.py", line 10, in init
super().init(feed_type)
File "/home/kali/emba/external/cve-search/sbin/../lib/DownloadHandler.py", line 52, in init
self.file_queue.clear()
File "/home/kali/emba/external/cve-search/sbin/../lib/redis_q.py", line 45, in clear
self.__db.delete(self.key)
File "/home/kali/.local/lib/python3.9/site-packages/redis/commands/core.py", line 972, in delete
return self.execute_command('DEL', *names)
File "/home/kali/.local/lib/python3.9/site-packages/redis/client.py", line 1068, in execute_command
conn = self.connection or pool.get_connection(command_name, **options)
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 1173, in get_connection
connection.connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 575, in connect
self.on_connect()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 653, in on_connect
auth_response = self.read_response()
File "/home/kali/.local/lib/python3.9/site-packages/redis/connection.py", line 765, in read_response
raise response
redis.exceptions.ResponseError: WRONGPASS invalid username-password pair

The issue is that if I don't use PROXY the default installation process works perfectly:

$ git clone emba
$ cd emba
$ sudo ./installer.sh -d

If I set proxy settings the installer fails. I tried it many times. A few days ago I was install emba with kali user, the installer failed at the same point but I was able to execute these:

$ cd ~/emba/external/cve-search/
$ ./sbin/db_mgmt_cpe_dictionary.py -p
$ ./sbin/db_mgmt_json.py -p
$ ./sbin/db_updater.py -c
$ sudo /etc/init.d/redis-server restart
$ ./sbin/db_updater.py -v

and I started a firmware scan which looked fine, but the CVE-search part did not work at all.

Would it be possible to install emba with PROXY on your side and compare the results?
I am not an expert in Linux, I thought my script would cover all the proxy related things but it looks like incomplete somehow.
Or give me any hint and I will try it!

Thank you very much.

[m-1-k-3]

But the error is changing :) So, there is some progress.

Now we have a Redis auth error. Please check the credentials in external/cve-search/etc/configuration.ini and compare it with the /etc/redis/redis.conf credentials. They need to match.

Currently I have no proxy environment but I put this on my todo list.

[torabi12]

Good morning,

In the configuration.ini file I have these parameters:

[Redis]
Host: localhost
Port: 6379
Password: 9EkREV4Bk
redisQ: 9
VendorsDB: 10
NotificationsDB: 11
RefDB: 12

[Database]
Host: 172.36.0.1
Port: 27017
DB: cvedb
PluginName: mongodb

[dbmgt]
Tmpdir: ./tmp/

[FulltextIndex]
Indexdir: ./indexdir/

[Webserver]
Host: 127.0.0.1
Port: 5000
Debug: False
PageLength: 50
LoginRequired: True
OIDC: False
CLIENT_ID: xx
CLIENT_SECRET: xx
IDP_DISCOVERY_URL: xx
SSL_VERIFY: False
SSL: True
Certificate: ssl/cve-search.crt
Key: ssl/cve-search.key
WebInterface: Full
MountPath: /MOUNTY/MC/MOUNT

[Logging]
Logging: True
Logfile: log/cve-search.log
Updatelogfile: log/update_populate.log
MaxSize: 150MB
Backlog: 5

[Proxy]
http:
IgnoreCerts: False

[CVE]
StartYear: 2002

and in the redis.conf is this:

bind 127.0.0.1 ::1
protected-mode yes
port 6379
requirepass 9EkREV4Bk

In the configuration.ini file's [Proxy] section is it ok with blank parameter?

[torabi12]

Hello Michael, Hello Benedikt,

I know it is long but I did yesterday night a brand new installation with emba with proxy and I would like to share with you all the steps what I did. Some of them can be unnecessary but behind proxy the default very easy installer way is not working for me.

VMware Workstation Pro 16.2.2
Kali-Linux-2021.4-vmware-amd64.vmx
https://kali.download/virtual-images/kali-2021.4/kali-linux-2021.4-vmware-amd64.7z

As root:

chsh -s /bin/bash

chsh -s /bin/bash kali

logout / login
$ echo $0
/bin/bash

Proxy Setup
$ sudo ./proxy.sh
proxy.txt

logout / login

Docker Install & Proxy Setup
$ sudo ./docker.sh
docker.txt

$ sudp apt upgrade
reboot

As kali user:
$ git clone https://github.com/e-m-b-a/emba.git
$ git clone https://github.com/cve-search/cve-search.git

I clone the cve-search by manually because I have to install the prerequisites, let's do it:

$ cd cve-search
$ xargs sudo apt-get install -y < requirements.system
$ pip3 install -r requirements.txt
$ sudo su -

pip3 install -r requirements.txt

Verify:

pip freeze | grep pymongo

exit
$ pip freeze | grep pymongo

All of them work fine in this way.

MongoDB installation
$ cd ..
$ wget -qO - https://www.mongodb.org/static/pgp/server-4.4.asc | sudo apt-key add -
$ sudo touch /etc/apt/sources.list.d/mongodb-org-4.4.list
$ echo "deb http://repo.mongodb.org/apt/debian buster/mongodb-org/4.4 main" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.4.list
$ sudo apt update
$ sudo apt install -y mongodb-org
$ sudo systemctl daemon-reload
$ sudo systemctl start mongod
$ sudo systemctl status mongod
$ sudo systemctl enable mongod

MongoDB is running fine.

and now emba installation:

$ cd ~/emba
$ sudo ./installer.sh -d

but the installer fails at the CPE Downloads section:

and now I am doing manual download with these commands:

$ cd ~/emba/external/cve-search/
$ ./sbin/db_mgmt_cpe_dictionary.py -p
$ ./sbin/db_mgmt_json.py -p
$ ./sbin/db_updater.py -c
$ sudo /etc/init.d/redis-server restart

I am doing right now this, it takes some time to get all the DB content to be ready but if I add the command one by one it is working as kali user:

I would like to use EMBArk in this proxy environment but without successful emba installation I have no chance to install EMBArk.

[torabi12]

I finished the manul CPE DB download and I could start an emba scan with -p /scan_profiles/full_scan.emba and the cve-search part is working fine!!! This is the first time when it is working with kali user and PROXY installation.
Thank you very much for your support and advices.

In the future it would be nice to fine tune the installation process because the above way is not the best for sure but it works.

[torabi12]

During the installation I have 3 warnings:
WARNING: The FIRMWARE variable is not set. Defaulting to a blank string.
WARNING: The LOG variable is not set. Defaulting to a blank string.
WARNING: The EMBA variable is not set. Defaulting to a blank string.

I would like to mount an external filesystem in /mnt/emba_logs/ folder to have enough space for the tests, would these environment variables be fine to ignore the warning messages?

EMBA='/home/kali/emba/'
FIRMWARE='/home/kali/firmwares/'
LOG='/mnt/emba_logs/'

[m-1-k-3]

Great to hear that you have managed the installation. Thank you very much for detailed explanation and the howto. Currently I work on a redesign of the installer. As soon as we are ready with this update I will check the proxy installation topic again.

You do not need to set the variables. Just configure the emba startup command with the following parameters:
-l paramater for the log directory,
-f for the firmware
Both can be on a different media.

Hope that helps.

[torabi12]

Thanks, I just wanted to skip the warnings during the install but it does not count.

Have a nice day!
Regards,
Torabi

[m-1-k-3]

Linked this issue from the wiki https://github.com/e-m-b-a/emba/wiki/Installation#random-installation-topics

I will close this for now and come back to it later

[m-1-k-3]

FAQ is also updated: https://github.com/e-m-b-a/emba/wiki/FAQ