Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Install failing Ubuntu 20.04 - Believe due to PyMISPGalaxies update #162

Closed
cybgit opened this issue May 2, 2023 · 7 comments
Closed

Install failing Ubuntu 20.04 - Believe due to PyMISPGalaxies update #162

cybgit opened this issue May 2, 2023 · 7 comments

Comments

@cybgit
Copy link

cybgit commented May 2, 2023 •
edited
Loading

AIL fails to install on Unbuntu 20.04.

This installed fine in March 2023 on the same Ubuntu distro.

I have raised an issue MISP/PyMISPGalaxies#8 as believe this relates to a recent update in PyMISPGalaxies

For completeness, here is the traceback with the errors

Installing collected packages: pytaxonomies
Successfully installed pytaxonomies-1.5.0
Collecting git+https://github.com/MISP/PyMISPGalaxies
  Cloning https://github.com/MISP/PyMISPGalaxies to /tmp/pip-req-build-h7spqapm
  Running command git clone --filter=blob:none --quiet https://github.com/MISP/PyMISPGalaxies /tmp/pip-req-build-h7spqapm
  Resolved https://github.com/MISP/PyMISPGalaxies to commit 14859ae37572c9d6278cbd56036dc0d0f0ad942f
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... - \ | done
  Getting requirements to build wheel ... done
ERROR: Exception:
Traceback (most recent call last):
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 169, in exc_logging_wrapper
    status = run_func(*args)
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 248, in wrapper
    return func(self, options, args)
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 491, in collect_root_requirements
    req = self._make_requirement_from_install_req(
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 453, in _make_requirement_from_install_req
    cand = self._make_candidate_from_link(
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 206, in _make_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 293, in __init__
    super().__init__(
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
    dist = self._prepare_distribution()
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 304, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 516, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 631, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 69, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 48, in prepare_distribution_metadata
    self._install_build_reqs(finder)
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 118, in _install_build_reqs
    build_reqs = self._get_build_requires_wheel()
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 95, in _get_build_requires_wheel
    return backend.get_requires_for_build_wheel()
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_internal/utils/misc.py", line 692, in get_requires_for_build_wheel
    return super().get_requires_for_build_wheel(config_settings=cs)
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 166, in get_requires_for_build_wheel
    return self._call_hook('get_requires_for_build_wheel', {
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "/home/azureuser/ail-framework/AILENV/lib/python3.8/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
  File "/usr/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'poetry.masonry'

user@hostxxxx:~/ail-framework$ ^C
@cybgit cybgit mentioned this issue May 2, 2023
Closed
@adulau
Copy link
Member

adulau commented May 8, 2023

Did you use the new KVROCKS branch (AIL 5.0 pre-release)?

@cybgit
Copy link
Author

cybgit commented May 9, 2023

Did you use the new KVROCKS branch (AIL 5.0 pre-release)?

No, but will give that a try

@cybgit
Copy link
Author

cybgit commented May 9, 2023 •
edited
Loading

Ok tried using KVROCKS fork and get this error (bottom 2 lines of below) which i think is when its compiling yara? I have further output if it helps.

  inflating: temp/yara-4.3.0/yarac.man  
~/ail-framework/var/www/temp/yara-4.3.0 ~/ail-framework/var/www
libtoolize: putting auxiliary files in AC_CONFIG_AUX_DIR, 'build-aux'.
libtoolize: copying file 'build-aux/ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
libtoolize: copying file 'm4/libtool.m4'
libtoolize: copying file 'm4/ltoptions.m4'
libtoolize: copying file 'm4/ltsugar.m4'
libtoolize: copying file 'm4/ltversion.m4'
libtoolize: copying file 'm4/lt~obsolete.m4'
configure.ac:20: installing 'build-aux/ar-lib'
configure.ac:20: installing 'build-aux/compile'
configure.ac:38: installing 'build-aux/config.guess'
configure.ac:38: installing 'build-aux/config.sub'
configure.ac:8: installing 'build-aux/install-sh'
configure.ac:8: installing 'build-aux/missing'
Makefile.am: installing 'build-aux/depcomp'
configure.ac: installing 'build-aux/ylwrap'
parallel-tests: installing 'build-aux/test-driver'
checking whether make supports nested variables... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports the include directive... yes (GNU style)
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking dependency style of gcc... gcc3
checking for ar... ar
checking the archiver (ar) interface... ar
checking for gcc... (cached) gcc
checking whether we are using the GNU C compiler... (cached) yes
checking whether gcc accepts -g... (cached) yes
checking for gcc option to accept ISO C89... (cached) none needed
checking whether gcc understands -c and -o together... (cached) yes
checking dependency style of gcc... (cached) gcc3
checking for gcc option to accept ISO C99... none needed
checking for flex... no
checking for lex... no
checking for bison... no
checking for byacc... no
checking for inline... inline
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking whether byte ordering is bigendian... no
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking how to print strings... printf
checking for a sed that does not truncate output... /usr/bin/sed
checking for fgrep... /usr/bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for a working dd... /usr/bin/dd
checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1
checking for mt... mt
checking if mt is a manifest tool... no
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for the pthreads library -lpthreads... no
checking whether pthreads work without any flags... no
checking whether pthreads work with -Kthread... no
checking whether pthreads work with -kthread... no
checking for the pthreads library -llthread... no
checking whether pthreads work with -pthread... yes
checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE
checking if more special flags are required for pthreads... no
checking whether to check for GCC pthread/shared inconsistencies... yes
checking whether -pthread is sufficient with -shared... yes
checking for isnan in -lm... yes
checking for log2 in -lm... yes
checking for strlcpy... no
checking for strlcat... no
checking for memmem... yes
checking for timegm... yes
checking for _mkgmtime... no
checking for clock_gettime... yes
checking stdbool.h usability... yes
checking stdbool.h presence... yes
checking for stdbool.h... yes
./configure: line 11516: syntax error near unexpected token `PROTOBUF_C,'
./configure: line 11516: `    PKG_CHECK_MODULES(PROTOBUF_C, libprotobuf-c >= 1.0.0)'
user@host:~/ail-framework$

@cybgit
Copy link
Author

cybgit commented May 10, 2023

Looks like there was a requirement for pkg-config which is found from this issue

After installing pkg-config with sudo apt-get install pkg-config rerunning the AIL install appears to have got to the end.

@cybgit
Copy link
Author

cybgit commented May 10, 2023 •
edited
Loading

Next issue is launching from KVROCKS branch causes an update to pull down from master branch which changes the LAUNCH.sh script (and i expect other htings).

I managed to comment out line 598 update; in launch_all function in LAUNCH.sh. I also found had to manually launch the kvrocks redis on port 6383. Did this with ./LAUNCH.sh -lk. Then doing a ./LAUNCH.sh -l managed to get things going.

@cybgit
Copy link
Author

cybgit commented May 15, 2023 •
edited
Loading

UPDATE: After commenting out the update in launch.sh ./LAUNCH.sh -l actually works without needing to ./LAUNCH.sh -lk first.

I then had to install TOR and fix playwright issues as the crawler test was failing.

This is how i did it on Ubuntu 20 if this helps anyone.

Install TOR

Instructions adapted from here

$ cd /etc/apt/sources.list.d/
$ sudo vi sudo vi tor-project.list

Add in the below 2 lines (i'd checked my architecture to be amd64)

deb     [arch=amd64 signed-by=/usr/share/keyrings/tor-archive-keyring.gpg] https://deb.torproject.org/torproject.org focal main
deb-src [arch=amd64 signed-by=/usr/share/keyrings/tor-archive-keyring.gpg] https://deb.torproject.org/torproject.org focal main

Grab the repo gpg key so apt works with it $ sudo wget -qO- https://deb.torproject.org/torproject.org/A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89.asc | gpg --dearmor | sudo tee /usr/share/keyrings/tor-archive-keyring.gpg >/dev/null

Update repository and install tor $ sudo apt update ; sudo apt install tor deb.torproject.org-keyring

Check Tor us running. I just check its listening on port 9050 with sudo netstat -luntp.

Fix playwright issues

When i ran the crawler test on AIL it failed with errors about Playwright. This is how i fixed.

Activated the venv Lacus runs under user@host:~$ source /home/azureuser/.cache/pypoetry/virtualenvs/lacus-BXtO8qME-py3.8/bin/activate

Ran playwright install as suggested by the errors with the crawler test (lacus-py3.8) user@host:~$ playwright install

That installed browsers and things but the crawler test then showed another error. Which was fixed with (lacus-py3.8) user@host:~$ playwright install-deps.

NOTE i originally ran sudo playwright install-deps as suggested by the error, but as we're in a venv, and sudo was trying to run on system it failed. Hence no sudo in the command i ran

Once it had finished installing everything, the test then worked.

I have ran into a number of other issues which i'll raise a new issue for.

@Terrtia
Copy link
Member

Terrtia commented Jun 7, 2023

Thanks for the report and the test !
Fixed in the new v5.0 release.
You can checkout the master branch

I'll add add your instruction in the documentation.

Feel free to open an issue if you encounter any problems.

@Terrtia Terrtia closed this as completed Jun 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@adulau @Terrtia @cybgit