Installation Errors on Fresh Ubuntu 22.04.1 Install and Unable to Connect to embark.local

[martialhax]

Describe the bug
I have a fresh Minimal install of Ubuntu 22.04.1 (fully updated) on bare metal just for Emba and Embark

Installing Emba seems to have gone flawlessly but Installing Embark gave some unexpected errors (first two images below) and when starting the server via
sudo ./run-server.sh it gives a "Failed setup mysql and redis docker images" error (3rd image below). When attempting to go to embark.local it just says a browser Unable to Connect message.

To Reproduce
Steps to reproduce the behavior:

1. EMBArk installation (default mode)
2. See errors
3. Start EMBArk: sudo ./run-server.sh
4. See error

Expected behavior
I can not get http://embark.local/ in the web browser.
Message is this:
"Unable to connect
Firefox can’t establish a connection to the server at embark.local."

Screenshots

Desktop (please complete the following information):

• OS: Ubuntu 22.04.1 (Jammy) Fully Patched / Updated

Additional context
I have re-run the sudo ./installer.sh -d command after rebooting and get same issues again.

[martialhax]

@torabi12 Is this a fresh install after doing a full sudo apt update and then apt upgrade? Since a separate install of emba went fine for me I'm thinking its either a package in the version of emba with EMBArk OR a package issue specific to EMBArk and something that got updated in Ubuntu. Just guessing....

[BenediktMKuehne]

Hi @martialhax,

this appears to be a problem with pipenv.
There are 2 things I would like you to do:

1. pip3.10 install mod-wsgi-standalone in order to confirm that pip for python 3.10 is working properly
2. run pipenv --version and pipenv install inside the cloned repo and post the complete console-output

[martialhax]

Hi @BenediktMKuehne

1. pip3.10 install mod-wsgi-standalone in order to confirm that pip for python 3.10 is working properly

I got the following error messages (beginning of the output screenshot and ending...the middle was just listing of various apr-1.7.0 files -

2. run pipenv --version and pipenv install inside the cloned repo and post the complete console-output

and....

user@user-Precision-5560:~/embark$ pipenv install
/usr/lib/python3/dist-packages/pkg_resources/init.py:116: PkgResourcesDeprecationWarning: 0.1.43ubuntu1 is an invalid version and will not be supported in a future release
warnings.warn(
/usr/lib/python3/dist-packages/pkg_resources/init.py:116: PkgResourcesDeprecationWarning: 1.1build1 is an invalid version and will not be supported in a future release
warnings.warn(
Creating a virtualenv for this project...
Pipfile: /home/user/embark/Pipfile
Using /usr/bin/python3 (3.10.4) to create virtualenv...
⠹ Creating virtual environment...created virtual environment CPython3.10.4.final.0-64 in 121ms
creator CPython3Posix(dest=/home/user/.local/share/virtualenvs/embark-YFTzu2Od, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/user/.local/share/virtualenv)
added seed packages: pip==22.2.2, setuptools==63.4.1, wheel==0.37.1
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator

✔ Successfully created virtual environment!
Virtualenv location: /home/user/.local/share/virtualenvs/embark-YFTzu2Od
Installing dependencies from Pipfile.lock (2983c2)...
An error occurred while installing autobahn==22.6.1; python_version >= '3.7' --hash=sha256:fb63e946d5c2dd0df680851e84e65624a494ce87c999f2a4944e4f2d81bf4498! Will try again.
An error occurred while installing hiredis==2.0.0; python_full_version >= '3.6.0' --hash=sha256:b84f29971f0ad4adaee391c6364e6f780d5aae7e9226d41964b26b49376071d0 --hash=sha256:a00514362df15af041cc06e97aebabf2895e0a7c42c83c21894be12b84402d79 --hash=sha256:87c7c10d186f1743a8fd6a971ab6525d60abd5d5d200f31e073cd5e94d7e7a9d --hash=sha256:e3447d9e074abf0e3cd85aef8131e01ab93f9f0e86654db7ac8a3f73c63706ce --hash=sha256:a7928283143a401e72a4fad43ecc85b35c27ae699cf5d54d39e1e72d97460e1d --hash=sha256:f8196f739092a78e4f6b1b2172679ed3343c39c61a3e9d722ce6fcf1dac2824a --hash=sha256:139705ce59d94eef2ceae9fd2ad58710b02aee91e7fa0ccb485665ca0ecbec63 --hash=sha256:3dddf681284fe16d047d3ad37415b2e9ccdc6c8986c8062dbe51ab9a358b50a5 --hash=sha256:04026461eae67fdefa1949b7332e488224eac9e8f2b5c58c98b54d29af22093e --hash=sha256:81d6d8e39695f2c37954d1011c0480ef7cf444d4e3ae24bc5e89ee5de360139a --hash=sha256:0f41827028901814c709e744060843c77e78a3aca1e0d6875d2562372fcb405a --hash=sha256:0adea425b764a08270820531ec2218d0508f8ae15a448568109ffcae050fee26 --hash=sha256:3d55e36715ff06cdc0ab62f9591607c4324297b6b6ce5b58cb9928b3defe30ea --hash=sha256:a4ee8000454ad4486fb9f28b0cab7fa1cd796fc36d639882d0b34109b5b3aec9 --hash=sha256:65d653df249a2f95673976e4e9dd7ce10de61cfc6e64fa7eeaa6891a9559c581 --hash=sha256:0d5109337e1db373a892fdcf78eb145ffb6bbd66bb51989ec36117b9f7f9b579 --hash=sha256:508999bec4422e646b05c95c598b64bdbef1edf0d2b715450a078ba21b385bcc --hash=sha256:7f0055f1809b911ab347a25d786deff5e10e9cf083c3c3fd2dd04e8612e8d9db --hash=sha256:07bbf9bdcb82239f319b1f09e8ef4bdfaec50ed7d7ea51a56438f39193271163 --hash=sha256:f52010e0a44e3d8530437e7da38d11fb822acfb0d5b12e9cd5ba655509937ca0 --hash=sha256:1233e303645f468e399ec906b6b48ab7cd8391aae2d08daadbb5cad6ace4bd87 --hash=sha256:1f03d4dadd595f7a69a75709bc81902673fa31964c75f93af74feac2f134cc54 --hash=sha256:0b39ec237459922c6544d071cdcf92cbb5bc6685a30e7c6d985d8a3e3a75326e --hash=sha256:294a6697dfa41a8cba4c365dd3715abc54d29a86a40ec6405d677ca853307cfb --hash=sha256:b4c8b0bc5841e578d5fb32a16e0c305359b987b850a06964bd5a62739d688048 --hash=sha256:5d2a48c80cf5a338d58aae3c16872f4d452345e18350143b3bf7216d33ba7b99 --hash=sha256:240ce6dc19835971f38caf94b5738092cb1e641f8150a9ef9251b7825506cb05 --hash=sha256:04927a4c651a0e9ec11c68e4427d917e44ff101f761cd3b5bc76f86aaa431d27 --hash=sha256:a39efc3ade8c1fb27c097fd112baf09d7fd70b8cb10ef1de4da6efbe066d381d --hash=sha256:adf4dd19d8875ac147bf926c727215a0faf21490b22c053db464e0bf0deb0485 --hash=sha256:dcef843f8de4e2ff5e35e96ec2a4abbdf403bd0f732ead127bd27e51f38ac298 --hash=sha256:3f5f7e3a4ab824e3de1e1700f05ad76ee465f5f11f5db61c4b297ec29e692b2e --hash=sha256:ae8427a5e9062ba66fc2c62fb19a72276cf12c780e8db2b0956ea909c48acff5 --hash=sha256:8b42c0dc927b8d7c0eb59f97e6e34408e53bc489f9f90e66e568f329bff3e443 --hash=sha256:cb2126603091902767d96bcb74093bd8b14982f41809f85c9b96e519c7e1dc41 --hash=sha256:5dc7a94bb11096bc4bffd41a3c4f2b958257085c01522aa81140c68b8bf1630a --hash=sha256:09004096e953d7ebd508cded79f6b21e05dff5d7361771f59269425108e703bc --hash=sha256:c39c46d9e44447181cd502a35aad2bb178dbf1b1f86cf4db639d7b9614f837c6 --hash=sha256:11d119507bb54e81f375e638225a2c057dda748f2b1deef05c2b1a5d42686048 --hash=sha256:807b3096205c7cec861c8803a6738e33ed86c9aae76cac0e19454245a6bbbc0a --hash=sha256:7492af15f71f75ee93d2a618ca53fea8be85e7b625e323315169977fae752426! Will try again.
An error occurred while installing mod-wsgi-httpd==2.4.54.1 --hash=sha256:cbf3495e4efa5f15db60efde959e1470b36dc834da12b62d10e70042b172e31d! Will try again.
An error occurred while installing mod-wsgi-standalone==4.9.3 --hash=sha256:713a59abbf2648e1dc3e423d05628180f11f6a192c0aae08559166ab41c7dbe9! Will try again.
An error occurred while installing mysqlclient==2.1.1 --hash=sha256:c812b67e90082a840efb82a8978369e6e69fc62ce1bda4ca8f3084a9d862308b --hash=sha256:828757e419fb11dd6c5ed2576ec92c3efaa93a0f7c39e263586d1ee779c3d782 --hash=sha256:0d1cd3a5a4d28c222fa199002810e8146cffd821410b67851af4cc80aeccd97c --hash=sha256:c1ed71bd6244993b526113cca3df66428609f90e4652f37eb51c33496d478b37 --hash=sha256:996924f3483fd36a34a5812210c69e71dea5a3d5978d01199b78b7f6d485c855 --hash=sha256:dea88c8d3f5a5d9293dfe7f087c16dd350ceb175f2f6631c9cf4caf3e19b7a96 --hash=sha256:b355c8b5a7d58f2e909acdbb050858390ee1b0e13672ae759e5e784110022994! Will try again.
🐍 ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 45/45 — 00:00:13
Installing initially failed dependencies...
[pipenv.exceptions.InstallError]: Looking in indexes: https://pypi.python.org/simple
[pipenv.exceptions.InstallError]: Collecting hiredis==2.0.0
[pipenv.exceptions.InstallError]: Using cached hiredis-2.0.0.tar.gz (75 kB)
[pipenv.exceptions.InstallError]: Preparing metadata (setup.py): started
[pipenv.exceptions.InstallError]: Preparing metadata (setup.py): finished with status 'done'
[pipenv.exceptions.InstallError]: Building wheels for collected packages: hiredis
[pipenv.exceptions.InstallError]: Building wheel for hiredis (setup.py): started
[pipenv.exceptions.InstallError]: Building wheel for hiredis (setup.py): finished with status 'error'
[pipenv.exceptions.InstallError]: Running setup.py clean for hiredis
[pipenv.exceptions.InstallError]: Failed to build hiredis
[pipenv.exceptions.InstallError]: Installing collected packages: hiredis
[pipenv.exceptions.InstallError]: Running setup.py install for hiredis: started
[pipenv.exceptions.InstallError]: Running setup.py install for hiredis: finished with status 'error'
[pipenv.exceptions.InstallError]: error: subprocess-exited-with-error
[pipenv.exceptions.InstallError]:
[pipenv.exceptions.InstallError]: × python setup.py bdist_wheel did not run successfully.
[pipenv.exceptions.InstallError]: │ exit code: 1
[pipenv.exceptions.InstallError]: ╰─> [24 lines of output]
[pipenv.exceptions.InstallError]: /tmp/pip-install-larcnzbf/hiredis_31054fe51c884c7f9c26f9c281c7996e/setup.py:7: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
[pipenv.exceptions.InstallError]: import sys, imp, os, glob, io
[pipenv.exceptions.InstallError]: /home/user/.local/share/virtualenvs/embark-YFTzu2Od/lib/python3.10/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
[pipenv.exceptions.InstallError]: warnings.warn(
[pipenv.exceptions.InstallError]: running bdist_wheel
[pipenv.exceptions.InstallError]: running build
[pipenv.exceptions.InstallError]: running build_py
[pipenv.exceptions.InstallError]: creating build
[pipenv.exceptions.InstallError]: creating build/lib.linux-x86_64-cpython-310
[pipenv.exceptions.InstallError]: creating build/lib.linux-x86_64-cpython-310/hiredis
[pipenv.exceptions.InstallError]: copying hiredis/init.py -> build/lib.linux-x86_64-cpython-310/hiredis
[pipenv.exceptions.InstallError]: copying hiredis/version.py -> build/lib.linux-x86_64-cpython-310/hiredis
[pipenv.exceptions.InstallError]: copying hiredis/hiredis.pyi -> build/lib.linux-x86_64-cpython-310/hiredis
[pipenv.exceptions.InstallError]: copying hiredis/py.typed -> build/lib.linux-x86_64-cpython-310/hiredis
[pipenv.exceptions.InstallError]: warning: build_py: byte-compiling is disabled, skipping.
[pipenv.exceptions.InstallError]:
[pipenv.exceptions.InstallError]: running build_ext
[pipenv.exceptions.InstallError]: building 'hiredis.hiredis' extension
[pipenv.exceptions.InstallError]: creating build/temp.linux-x86_64-cpython-310
[pipenv.exceptions.InstallError]: creating build/temp.linux-x86_64-cpython-310/src
[pipenv.exceptions.InstallError]: creating build/temp.linux-x86_64-cpython-310/vendor
[pipenv.exceptions.InstallError]: creating build/temp.linux-x86_64-cpython-310/vendor/hiredis
[pipenv.exceptions.InstallError]: x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -Ivendor -I/home/user/.local/share/virtualenvs/embark-YFTzu2Od/include -I/usr/include/python3.10 -c src/hiredis.c -o build/temp.linux-x86_64-cpython-310/src/hiredis.o
[pipenv.exceptions.InstallError]: error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory
[pipenv.exceptions.InstallError]: [end of output]
[pipenv.exceptions.InstallError]:
[pipenv.exceptions.InstallError]: note: This error originates from a subprocess, and is likely not a problem with pip.
[pipenv.exceptions.InstallError]: ERROR: Failed building wheel for hiredis
[pipenv.exceptions.InstallError]: error: subprocess-exited-with-error
[pipenv.exceptions.InstallError]:
[pipenv.exceptions.InstallError]: × Running setup.py install for hiredis did not run successfully.
[pipenv.exceptions.InstallError]: │ exit code: 1
[pipenv.exceptions.InstallError]: ╰─> [26 lines of output]
[pipenv.exceptions.InstallError]: /tmp/pip-install-larcnzbf/hiredis_31054fe51c884c7f9c26f9c281c7996e/setup.py:7: DeprecationWarning: the imp module is deprecated in favour of importlib and slated for removal in Python 3.12; see the module's documentation for alternative uses
[pipenv.exceptions.InstallError]: import sys, imp, os, glob, io
[pipenv.exceptions.InstallError]: /home/user/.local/share/virtualenvs/embark-YFTzu2Od/lib/python3.10/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
[pipenv.exceptions.InstallError]: warnings.warn(
[pipenv.exceptions.InstallError]: running install
[pipenv.exceptions.InstallError]: /home/user/.local/share/virtualenvs/embark-YFTzu2Od/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
[pipenv.exceptions.InstallError]: warnings.warn(
[pipenv.exceptions.InstallError]: running build
[pipenv.exceptions.InstallError]: running build_py
[pipenv.exceptions.InstallError]: creating build
[pipenv.exceptions.InstallError]: creating build/lib.linux-x86_64-cpython-310
[pipenv.exceptions.InstallError]: creating build/lib.linux-x86_64-cpython-310/hiredis
[pipenv.exceptions.InstallError]: copying hiredis/init.py -> build/lib.linux-x86_64-cpython-310/hiredis
[pipenv.exceptions.InstallError]: copying hiredis/version.py -> build/lib.linux-x86_64-cpython-310/hiredis
[pipenv.exceptions.InstallError]: copying hiredis/hiredis.pyi -> build/lib.linux-x86_64-cpython-310/hiredis
[pipenv.exceptions.InstallError]: copying hiredis/py.typed -> build/lib.linux-x86_64-cpython-310/hiredis
[pipenv.exceptions.InstallError]: warning: build_py: byte-compiling is disabled, skipping.
[pipenv.exceptions.InstallError]:
[pipenv.exceptions.InstallError]: running build_ext
[pipenv.exceptions.InstallError]: building 'hiredis.hiredis' extension
[pipenv.exceptions.InstallError]: creating build/temp.linux-x86_64-cpython-310
[pipenv.exceptions.InstallError]: creating build/temp.linux-x86_64-cpython-310/src
[pipenv.exceptions.InstallError]: creating build/temp.linux-x86_64-cpython-310/vendor
[pipenv.exceptions.InstallError]: creating build/temp.linux-x86_64-cpython-310/vendor/hiredis
[pipenv.exceptions.InstallError]: x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -Ivendor -I/home/user/.local/share/virtualenvs/embark-YFTzu2Od/include -I/usr/include/python3.10 -c src/hiredis.c -o build/temp.linux-x86_64-cpython-310/src/hiredis.o
[pipenv.exceptions.InstallError]: error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory
[pipenv.exceptions.InstallError]: [end of output]
[pipenv.exceptions.InstallError]:
[pipenv.exceptions.InstallError]: note: This error originates from a subprocess, and is likely not a problem with pip.
[pipenv.exceptions.InstallError]: error: legacy-install-failure
[pipenv.exceptions.InstallError]:
[pipenv.exceptions.InstallError]: × Encountered error while trying to install package.
[pipenv.exceptions.InstallError]: ╰─> hiredis
[pipenv.exceptions.InstallError]:
[pipenv.exceptions.InstallError]: note: This is an issue with the package mentioned above, not pip.
[pipenv.exceptions.InstallError]: hint: See above for output from the failure.
ERROR: Couldn't install package: hiredis
Package installation failed...
☤ ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 4/5 — 00:00:01
/usr/lib/python3.10/subprocess.py:1067: ResourceWarning: subprocess 481223 is still running
_warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.FileIO name=6 mode='rb' closefd=True>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.FileIO name=9 mode='rb' closefd=True>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1067: ResourceWarning: subprocess 481224 is still running
_warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.FileIO name=8 mode='rb' closefd=True>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.FileIO name=11 mode='rb' closefd=True>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/usr/lib/python3.10/subprocess.py:1067: ResourceWarning: subprocess 481225 is still running
_warn("subprocess %s is still running" % self.pid,
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.FileIO name=10 mode='rb' closefd=True>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
sys:1: ResourceWarning: unclosed file <_io.FileIO name=13 mode='rb' closefd=True>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
user@user-Precision-5560:~/embark$

[BenediktMKuehne]

hi @martialhax,

according to this:
[pipenv.exceptions.InstallError]: x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -Ivendor -I/home/user/.local/share/virtualenvs/embark-YFTzu2Od/include -I/usr/include/python3.10 -c src/hiredis.c -o build/temp.linux-x86_64-cpython-310/src/hiredis.o [pipenv.exceptions.InstallError]: error: command 'x86_64-linux-gnu-gcc' failed: No such file or directory
your system is missing a C compiler.
sudo apt update && sudo apt install build-essential should fix that.

I wasn't aware of that dependency and am a bit surprised that pip doesn't check that on install.

Once gcc --version works, you can run git pull && sudo ./installer.sh -rd which should give you a clean reinstallation.

[BenediktMKuehne]

Will add this dependency in #54

[martialhax]

@BenediktMKuehne Thanks did a clean install on a reinstalled clean Ubuntu 22.0.4 with build-essential and that worked. However still saw this when starting the server:

Not sure if those - "Failed setup mysql and redis docker images" are something to address still or not.

It was strange that no CVEs found...when I know there are some and running EMBA before (not EMBArk) on the same firmware definitely found some before (Busybox etc.):

[m-1-k-3]

Could you please check the logs from the current run for the following error (right at the end):

tput: No value for $TERM and no -T specified
2
[-] EMBA failed in docker mode!

You can find the log in /var/www/emba_logs/<UUID from scan>/emba_run.log

[martialhax]

@m-1-k-3 Yes it's there... I also do see tons of CVE details in the log as well:

[m-1-k-3]

@martialhax cool! This means we fixed it in the release 1.1.1 of EMBA a few minutes ago ;)

The magic can be found here for your manual fix: https://github.com/e-m-b-a/emba/blob/188fa51e9115c7e4ba64bf6bd5c6a7abf7049917/modules/F50_base_aggregator.sh#L425

@BenediktMKuehne EMBA v1.1.1 is out now. What is your timetable to test and integrate it into EMBArk?

[martialhax]

@m-1-k-3 Hi and thanks. Can you dumb it down for me lol. Is there a set of commands I need to run to get the CVE findings showing? Thanks.

[BenediktMKuehne]

@martialhax cool! This means we fixed it in the release 1.1.1 of EMBA a few minutes ago ;)

The magic can be found here for your manual fix: https://github.com/e-m-b-a/emba/blob/188fa51e9115c7e4ba64bf6bd5c6a7abf7049917/modules/F50_base_aggregator.sh#L425

@BenediktMKuehne EMBA v1.1.1 is out now. What is your timetable to test and integrate it into EMBArk?

I'll open a quick update PR for this