BUG: Default installation steps failed on Ubuntu 22.04

[wiebewillems]

Describe the bug
When trying to setup Caldera via the specified installation steps, I encounter an AttributeError in the 'lib' module: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'. Did you mean: 'X509_V_FLAG_EXPLICIT_POLICY'.

To Reproduce
Steps to reproduce the behavior:

1. Spin up a fresh AWS EC2 with Ubuntu 22.04
2. Install Caldera

sudo apt update
sudo apt install python3-pip

# Install NodeJS
curl -fsSL https://deb.nodesource.com/setup_21.x | sudo -E bash - &&\
sudo apt install -y nodejs

# Clone Caldera
git clone https://github.com/mitre/caldera.git --recursive
cd caldera

# Install dependencies
pip3 install -r requirements.txt

# Build and run the server
python3 server.py --insecure --build

Expected behavior
Spinning up the Caldera server works and exposes the interface to port 8888

Screenshots
Not applicable.

Desktop (please complete the following information):

• OS: Ubuntu
• Browser N/A
• Version 5.0.0

Additional context

Traceback (most recent call last):
  File "/home/ubuntu/caldera/server.py", line 260, in <module>
    run_tasks(services=app_svc.get_services(), run_vue_server=args.uiDevHost)
  File "/home/ubuntu/caldera/server.py", line 66, in run_tasks
    loop.run_until_complete(app_svc.register_contacts())
  File "/usr/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/home/ubuntu/caldera/app/service/app_svc.py", line 150, in register_contacts
    await self.register_contact_tunnels(contact_svc)
  File "/home/ubuntu/caldera/app/service/app_svc.py", line 155, in register_contact_tunnels
    tunnel_class = import_module(tunnel_module_name).Tunnel
  File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/ubuntu/caldera/app/contacts/tunnels/tunnel_ssh.py", line 3, in <module>
    import asyncssh
  File "/home/ubuntu/.local/lib/python3.10/site-packages/asyncssh/__init__.py", line 31, in <module>
    from .agent import SSHAgentClient, SSHAgentKeyPair, connect_agent
  File "/home/ubuntu/.local/lib/python3.10/site-packages/asyncssh/agent.py", line 34, in <module>
    from .public_key import KeyPairListArg, SSHCertificate, SSHKeyPair
  File "/home/ubuntu/.local/lib/python3.10/site-packages/asyncssh/public_key.py", line 35, in <module>
    from .crypto import ed25519_available, ed448_available
  File "/home/ubuntu/.local/lib/python3.10/site-packages/asyncssh/crypto/__init__.py", line 58, in <module>
    from .x509 import X509Certificate, X509Name, X509NamePattern
  File "/home/ubuntu/.local/lib/python3.10/site-packages/asyncssh/crypto/x509.py", line 32, in <module>
    from OpenSSL import crypto
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1579, in <module>
    class X509StoreFlags(object):
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 1598, in X509StoreFlags
    NOTIFY_POLICY = _lib.X509_V_FLAG_NOTIFY_POLICY
AttributeError: module 'lib' has no attribute 'X509_V_FLAG_NOTIFY_POLICY'. Did you mean: 'X509_V_FLAG_EXPLICIT_POLICY'?

[github-actions]

Looks like your first issue -- we aim to respond to issues as quickly as possible. In the meantime, check out our documentation here: http://caldera.readthedocs.io/

[iknowjason]

@wiebewillems I am doing extensive testing of Caldera 5.0 on Ubuntu 22.04 running on AWS EC2 AMI. I am not not able to duplicate your issue, however.

There is a hidden bug that is not in Caldera docs and I noticed this a long time ago, it is only related to running on EC2. Can you try this and see if this fixes the issue? I have this in the bootstrap script of this terraform lab that builds Caldera:
https://github.com/iknowjason/AutomatedEmulation/blob/master/files/bas/bootstrap.sh.tpl#L31

# Upgrade pyOpenSSL - weird issue only impacting AWS EC2 AMI images
sudo pip3 install --upgrade pyOpenSSL

[iknowjason]

@wiebewillems Perhaps we can share notes. My 5.0 VueJS application builds, but the web app is not responsive when I try to login. Is your VueJS app responsive when submitting user credentials?

[wiebewillems]

@iknowjason I actually got the server up and running just now because of the hint of upgrading packages - it was not pyOpenSSL but cryptography for me, so before running the python3 server.py command, I ran

pip3 install cryptography --upgrade

which upgraded cryptography 42.0.2 to cryptography 42.0.3. Making a small PR for this one.

[wiebewillems]

@iknowjason for the web app being unresponsive - for me it was the same, it only works when really browsing via localhost:8888 and not via a public/other IP. That is probably being set somewhere, so might need to dig in the conf to change how it is exposed.

[iknowjason]

@iknowjason for the web app being unresponsive - for me it was the same, it only works when really browsing via localhost:8888 and not via a public/other IP. That is probably being set somewhere, so might need to dig in the conf to change how it is exposed.

That is probably the case. Might be good to link to this issue, looks like others are having the same. I ran a pcap trace of the connection. There is an HTTP response from the server, but the target VueJS application code might not be rendering due to the correct hostname, I gather.

Similar issue opened: #2881

[iknowjason]

@iknowjason I actually got the server up and running just now because of the hint of upgrading packages - it was not pyOpenSSL but cryptography for me, so before running the python3 server.py command, I ran

pip3 install cryptography --upgrade

which upgraded cryptography 42.0.2 to cryptography 42.0.3. Making a small PR for this one.

@wiebewillems I think when you upgraded cryptography, it had the same effect as the upgrade of pyOpenSSL that I used. That is good news.

But I'd also point out for your PR. This is only necessary and the issue is only seen when running on AWS EC2 running Ubuntu. From what I've seen from my testing. I just tested Ubuntu 22.04 not running on Amazon, and I didn't need to upgrade cryptography or pyOpenSSL. The default Caldera instructions just worked.

[iknowjason]

@wiebewillems I opened up a new GH issue on the non-responsive:
#2885

[wiebewillems]

But I'd also point out for your PR. This is only necessary and the issue is only seen when running on AWS EC2 running Ubuntu. From what I've seen from my testing. I just tested Ubuntu 22.04 not running on Amazon, and I didn't need to upgrade cryptography or pyOpenSSL. The default Caldera instructions just worked.

I wonder if it still makes sense to have it on 42.0.43 nonetheless? If not, I'll just remove the PR. Maybe @elegantmoose ?

[elegantmoose]

@wiebewillems @iknowjason Just FYSA, looking into.

[kslong2]

We did the following to get working in Azure:

sudo pip3 install --upgrade pyOpenSSL
pip3 install cryptography --upgrade

[0zk3y]

Upgrading Pip and Cryptography fixed the issue for me

python3 -m pip install pip --upgrade
pip install pyopenssl --upgrade

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days