Skip to content
Scripts to run a dockerized liquid node
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.

Readme.md

Liquid Investigations Node

Scripts and configuration to run a Liquid Node

Build Status

Requirements

Liquid Node requires a Nomad + Consul + Vault cluster where the software will be deployed. There are a few options:

Whichever option you choose, you will also need to:

  • Increase vm.max_map_count to at least 262144, to make elasticsearch happy - see the docs about elasticsearch in docker for details.
  • Make sure you have Python >= 3.7 installed.

Install a Cluster Manually

Download Consul, Vault and Nomad. For production environments follow their manuals. For development run them all in -dev mode:

./consul agent -dev &
./vault agent -dev &
./nomad agent -dev &

Install the Liquid Cluster

liquidinvestigations/cluster is a self-configuring cluster of Consul + Vault + Nomad. It's optimised for local development, testing, and demo/staging servers.

Use Vagrant

You can run a full Liquid cluster in a local virtual machine using Vagrant. The configuration has been tested with the libvirt driver but should work with the default VirtualBox driver as well.

  1. Install vagrant
  2. cd to the vagrant subfolder, everything Vagrant-related happens in here:
    cd vagrant/
  3. Start the VM:
    vagrant up

You can log into the vm using vagrant ssh. Optionally, install vagrant-env, and set environment variables in an .env local file that will be ignored by git.

Vagrant will forward the following http ports:

  • guest: 80, host: 1380 - public web server
  • guest: 8765, host: 18765 - internal web server
  • guest: 8500, host: 18500 - consul
  • guest: 8200, host: 18200 - vault
  • guest: 4646, host: 14646 - nomad

Vagrant on MacOS

On MacOS the following extra steps need to be taken:

  1. Run the following command in a terminal:
sudo socat -vvv  tcp-listen:80,fork tcp:localhost:1380
  1. Edit /etc/hosts and add the following line:
10.66.60.1       [liquid_domain] hoover.[liquid_domain]

where [liquid_domain] is the value of liquid.domain from the liquid.ini file.

Configuration

The Liquid Investigations cluster configuration is read from liquid.ini. Start with the example configuration file:

cp examples/liquid.ini .
vim liquid.ini

We need a way to access Vault. The simplest way is to use:

[liquid]
domain = liquid.example.org
debug = true

[cluster]
vault_secrets = /opt/cluster/var/vault-secrets.ini

[collection:testdata]
workers = 1

This assumes the vault-secrets.ini file exists and contains a vault token. It will be generated by the ./cluster.py autovault command, or you can create it manually in a different location with the following contents:

[vault]
root_token = s.Cmro41vNI4wIndgrPqzlqOKY

Then deploy to the cluster:

./liquid deploy

The liquid instance will listen by default on port 80 on the local machine. If you don't have a DNS domain pointing to the macine, you can add entries to /etc/hosts:

10.0.0.1 liquid.example.com
10.0.0.1 hoover.liquid.example.com
...

Create an initial admin user:

./liquid shell liquid-core ./manage.py createsuperuser

Versions

The liquid bundle comes with a versions.ini file with a known set of working versions. You can override them in liquid.ini, see examples/liquid.ini for more information.

Testdata

Set up the testdata collection. First download the data:

mkdir -p collections
git clone https://github.com/hoover/testdata collections/testdata

Next define the collection in liquid.ini:

[collection:testdata]
workers = 1

Then let the deploy command pick up the new collection:

./liquid deploy

Nextcloud

Nextcloud runs on the subdomain nextcloud.<liquid_domain>

Rocket.Chat

In order to activate liquid login, follow RocketChatAuthSetup.

Importing collections from docker-setup

Preparation

The node setup must be clean. For this, either use a new node install (recommended), or remove all collections from liquid.ini and run the following commands:

./liquid purge
./liquid halt
rm -fr ./volumes/hoover

Import

In order to import the collections from docker-setup run the following command:

./liquid importfromdockersetup [path_to_docker_setup] [method]

The method is optional. The default value is link, while the possible values are:

  • link: create links to existing directories in docker-setup
  • copy: copy the data from docker-setup to node
  • move: move the directories from docker-setup to node

When using the copy method, the import will take longer.

Debugging

Set the debug flag in liquid.ini:

[liquid]
debug = on

Then redeploy (./liquid deploy).

To log into the snoop docker container for testdata:

./liquid shell snoop-testdata-api

To dump the nginx configuration:

nomad alloc fs $(./liquid alloc liquid nginx) nginx/local/core.conf

Working on components

In order to work on Hoover Search, Hoover Snoop, or Liquid Core, first clone the repositories:

cd repos
./clone.sh https  # or ./clone.sh ssh, based on preference

After that, set this flag in your configuration:

[liquid]
...
mount_local_repos = true

Running custom jobs

You can deploy your own jobs on the cluster. First, create a nomad job file, you can use one of the existing .nomad files as a starting point. Save it in the local folder, or outside the repository, so that it doesn't interfere with updates. Then add the job to liquid.ini:

[job:foo]
template = local/foo.nomad

Afterwards, run ./liquid deploy, which will send your job foo to nomad.

You can’t perform that action at this time.