Ubuntu 20.04 support

[0x6d6f7468]

It's finally time, folks!

The installer does not currently play nicely with Ubuntu 20.04. I've been given the green light to take a look.

I believe issue #562 contains relevant info. I thought I saw a few more, and will link to them if I find them.

Currently when I run the RITA installer on my Ubuntu 20.04 VM, the installer dies without error. Likely related to #543. Thankfully, I'm able to briefly see that the overwritten installer message is [-] Installing MongoDB, which indicates that the installer is silently failing to install Mongo. Good times.

Thanks!

[0x6d6f7468]

Removed a /dev/null output redirect on line 622, and that finally yielded a proper error message. Something with the usual kludge isn't working just yet, but at least we know it's a Mongo issue. Just like old times...

[0x6d6f7468]

Looks like the Ubuntu 18.04 kludge might not work on Ubuntu 20.04. Recreating the mongo repo file, I got the following error:

Chasing that down to just installing mongodb-org manually, I get a more detailed error:

Installing mongodb-org* gives the following error, and I've got a bad feeling about this one:

Yikes... I'm gonna get some actual sleep and look into this more tomorrow or this weekend. I hope there's still a reasonable way to get Mongo 3.6 installed on Ubuntu 20.04.

[obaidrahman2020]

It is almost impossible to use Rita at the moment. The issues I have faced are as follows:
I installed Ubuntu 20.04 and it will fail at mongo db install. I manually installed mongodb 4.4 as a version between 3.4-3.7 wont

install on this version of Ubuntu. however rita eventually fails with the mongodb 4.4 during log import.

I then tried to install the 18.04 version, but that is out of support and a bigger pain since there are lots of hurdle to simply make it work, and not worth the pain.
I will highly appreciate if you can suggest a workable solution on Ubuntu.

[0x6d6f7468]

@obaidrahman2020 Thanks for the comment.

Currently, our queries are built around a specific version of mongodb. I've started some internal discussion about moving away from this.

As for your comment of being out of support, I'm not sure what you mean. Ubuntu 18.04 is supported until April 2023, and looks like it will still be receiving updates until 2028. If you're referring to the old version of mongodb, it's something we're looking into a bit, but we haven't had issues in the past running Ubuntu 16.04 and Ubuntu 18.04 installations.

Thanks again for your comment!

[obaidrahman2020]

Hi, I guess I mentioned the wrong version, tried using Ubuntu 18.10 and Ubuntu 20.04. The main challenge I faced with 18.10 was that there was no way to update any of the packages or even download any using "sudo apt-get update" without manipulating "/etc/apt/sources.list"( I couldn't get it working I am not sure of the current archive urls).

Does that mean only 18.04 works?

[0x6d6f7468]

@obaidrahman2020 With the way things work now (and as far as I'm aware), we support the LTS versions of Ubuntu, so 16.04 and 18.04.

As far as I know, these are the most common server versions of Ubuntu. I would have to do some digging to confirm these two claims, however.

[obaidrahman2020]

Thanks, I got it working on 18.04, without any issues.

[0x6d6f7468]

Things are hopeful that we might be able to move past Mongo 3.6. I will hopefully spend some time this evening to check this out further. Worst case, I'll look at it again over the weekend.

[william-stearns]

Things are hopeful that we might be able to move past Mongo 3.6. I will hopefully spend some time this evening to check this out further. Worst case, I'll look at it again over the weekend.

It appears the "repo.mongodb.org" servers do not have a Mongo 3.6 version for Ubuntu 20.04 ("focal"): See http://repo.mongodb.org/apt/ubuntu/dists/focal/mongodb-org/ .

[masq]

Just ran into this issue myself. Any updates here @0x6d6f7468 ? See this issue has been cold for a bit.

[BeanBagKing]

Google search for installing Mongo 3.6 on Ubuntu 20.04 led me here. Since nobody else seemed to have solved it, I was determined to get what seemed like a very bad idea working. Please don't attempt the below in a production environment. In fact, please don't attempt it in a dev or test environment. It's full of lax permissions (777 all the things!) EOL software (python2.7), and unsupported stuff (like Mongo 3.6 working on Ubuntu 20.04...)

Here be dragons:

If you're still reading, then here's the TL;DR. If you're not going to build a package or repo for me, then I'll build it myself, with blackjack, and mongo! We're going to grab the Mongo 3.6 source and build it on 20.04. Quick steps:

Grab some prerequisites

apt install build-essential
apt install libboost-filesystem-dev libboost-program-options-dev libboost-system-dev libboost-thread-dev
apt install python2.7
wget https://bootstrap.pypa.io/pip/2.7/get-pip.py
python2.7 get-pip.py

Get an old mongo version, install a few more prereqs

git clone --branch v3.6 https://github.com/mongodb/mongo
cd mongo
python2.7 -m pip install -r buildscripts/requirements.txt

Build and install our old mongo, ignore all the warnings! We're already well into bad idea territory

python2.7 buildscripts/scons.py --disable-warnings-as-errors --prefix=/opt/mongo install

Mongo isn't started, service isn't created. Let's do that.

Thanks to Code Grepper for helping me get started
https://www.codegrepper.com/code-examples/whatever/Failed+to+enable+unit%3A+Unit+file+mongod.service+does+not+exist.

cd /lib/systemd/system
vim mongodb.service

[Unit]
Description=An object/document-oriented database
Documentation=man:mongod(1)
After=network.target

[Service]
User=mongodb
Group=mongodb
ExecStart=/opt/mongo/bin/mongod --quiet

[Install]
WantedBy=multi-user.target

Mongo complains if it's directory is missing, give it one. Oh, my permissions are wrong, I'll fix that!

mkdir /data
mkdir /data/db
chmod 777 /data/db

Start mongo

systemctl daemon-reload
systemctl enable mongodb.service
systemctl start mongodb.service

RITA works!

> db.version()
3.6.23-5-g1bddf8a6bd
> quit()
root@zeek:~# lsb_release -d
Description:    Ubuntu 20.04.2 LTS
root@zeek:~# rita import /opt/zeek/logs/current/ test

        [+] Importing [/opt/zeek/logs/current/]:
        [-] Verifying log files have not been previously parsed into the target dataset ...
        [-] Processing batch 1 of 1
        [-] Parsing logs to: test ...
        [-] Parsing /opt/zeek/logs/current/conn.log -> test
        [-] Parsing /opt/zeek/logs/current/dns.log -> test
        [-] Parsing /opt/zeek/logs/current/http.log -> test
        [-] Parsing /opt/zeek/logs/current/ssl.log -> test
        [-] Host Analysis:            757 / 757  [==================] 100 %
        [-] Uconn Analysis:           796 / 796  [==================] 100 %
        [-] Exploded DNS Analysis:    565 / 565  [==================] 100 %
        [-] Hostname Analysis:        565 / 565  [==================] 100 %
        [-] Beacon Analysis:          796 / 796  [==================] 100 %
        [-] FQDN Beacon Analysis:     565 / 565  [==================] 100 %
        [-] UserAgent Analysis:       3 / 3  [==================] 100 %
        [-] Invalid Cert Analysis:    10 / 10  [==================] 100 %
        [-] Updating blacklisted peers ...
        [-] Indexing log entries ...
        [-] Updating metadatabase ...
        [-] Done!

[PaulK1966]

So is the "final" solution either to go back to Ubuntu 18.04 or to install mongoDB 3.6 on Ubuntu 2#.4 (or 21.4 in my case)? Any chance of moving past Mongo 3.6? That seems like a more sustainable solution...

Contributor
Author
0x6d6f7468 commented on Sep 17, 2020
Things are hopeful that we might be able to move past Mongo 3.6. I will hopefully spend some time this evening to check this out further. Worst case, I'll look at it again over the weekend.

[ethack]

Rita 4.3.0 bumped the supported Mongo version to 4.2. I understand that there are still no official packages for Ubuntu 20.04 but one user reported success by installing the Mongo 4.2 package for Ubuntu 18.04 on 20.04. You may also have luck with it on 21.04.

[PaulK1966]

Rita 4.3.0 bumped the supported Mongo version to 4.2. I understand that there are still no official packages for Ubuntu 20.04 but one user reported success by installing the Mongo 4.2 package for Ubuntu 18.04 on 20.04. You may also have luck with it on 21.04.

Fantastic, I will try that route.

It doesn't help that I'm trying to do this on a Raspberry Pi....but why do things the easy way ;-)

Thanks!
-PaulK