54 Initial Fixes #82

SamuelCarroll · 2019-02-06T18:56:11Z

Partially fulfills the requirements of Issue #54

The install script should now

install Docker if not present
install Docker-Compose if not present
accepts a [rita address-to-send-to] flag for non-interactive install

…d (might need to rearrange so it pops more)

…for breaking out IPFIX-RITA's installer to something nicer

… of the installer. Still in the testing/debugging stages

…and additional work

…d mongo check and test/debug non-interactive commands

…ll having awk(ward) problems

…d (might need to rearrange so it pops more)

…for breaking out IPFIX-RITA's installer to something nicer

… of the installer. Still in the testing/debugging stages

…and additional work

…d mongo check and test/debug non-interactive commands

…ll having awk(ward) problems

Zalgo2462

Went through everything but the new install script proper.
Everything looks good so far.
We need to change the README.md to reflect the new restrictions of the docker and docker-compose versions.

Would it be possible to move the install-scripts/ folder inside of dev-scripts/ ?

I'll begin looking at the install script proper.

dev-scripts/make-release.sh

Zalgo2462 · 2019-02-07T19:06:46Z

dev-scripts/make-release.sh

+cp "$INSTALLER_SCRIPTS_DIR/install_ipfix-rita.sh" "$INSTALLER_DIR"
+sed -i "s|INSTALLATION_DIR=\"REPLACE_WITH_INSTALL_DIR\"|INSTALLATION_DIR=\"$INSTALLATION_DIR\"|g" $INSTALLER_DIR/install_ipfix-rita.sh
+sed -i "s|INSTALLATION_ETC_DIR=\"REPLACE_WITH_ETC_DIR\"|INSTALLATION_ETC_DIR=\"$INSTALLATION_ETC_DIR\"|g" $INSTALLER_DIR/install_ipfix-rita.sh
+sed -i "s|DOCKER_IMAGES=\"./REPLACE_WITH_TARBALL\"|DOCKER_IMAGES=\"$DOCKER_IMAGE_OUT\"|g" $INSTALLER_DIR/install_ipfix-rita.sh


If these are the only variables we need to set in the install_ipfix-rita.sh script, I'm okay with using sed.
If this grows, we may want to turn to something like envsubst

dev-scripts/make-release.sh

Zalgo2462 · 2019-02-07T19:11:33Z

install-scripts/helpers/check_docker-compose.sh

+MIN_VERSION_MAJOR=1
+MIN_VERSION_MINOR=17
+MAX_VERSION_MAJOR=1
+MAX_VERSION_MINOR=23


We'll want to document these min/max versions of docker-compose. The min is already documented, but the max is not in the README

Zalgo2462 · 2019-02-07T19:25:03Z

install-scripts/helpers/check_docker.sh

+fi
+
+MIN_VERSION_MAJOR=17
+MIN_VERSION_MINOR=05


I listed 17.06 as the minimum in the readme, but that can be changed. I think I misread the Docker changelog when looking for when multi-stage builds were supported. For Docker EE its 17.06 for CE its 17.05. Since we use CE for everything, I think 17.05 is fine. The readme will need to be changed to reflect this.

Zalgo2462 · 2019-02-07T19:25:23Z

install-scripts/helpers/check_docker.sh

+MIN_VERSION_MAJOR=17
+MIN_VERSION_MINOR=05
+MAX_VERSION_MAJOR=18
+MAX_VERSION_MINOR=09


We need to list the max version in the readme

install-scripts/install_ipfix-rita.sh

Zalgo2462

If we get the usage_text fixed up and the read/writes all redirected to &2, I think we're ready to roll.

TBH, I'm not certain what all should be on &2 and what shouldn't be. I tried to note any echo's or read's without it.

Zalgo2462 · 2019-02-07T19:37:20Z

install-scripts/install_ipfix-rita.sh

+while [ -n "$1" ]; do
+	case "$1" in
+	rita|Rita|RITA)
+		if [ -n "$2" ] && ! echo "$2" | egrep -iq '(^rita$|^aih$)' ; then


Is && ! echo "$2" | egrep -iq '(^rita$|^aih$)' necessary here?

I believe the ! echo "$2" is but the next egrep might not be, we will definately need a second variable since rita addres-to-send-to must be bundled to denote a non-interactive install

I tried removing the egrep and it didn't work so I've added it back in

Zalgo2462 · 2019-02-07T19:39:59Z

install-scripts/install_ipfix-rita.sh

+			rita_system="$2"
+			shift
+		else
+			rita_system='127.0.0.1'


Running ./install_ipfix-rita rita will install ipfix-rita with the understanding that rita is on localhost?

That doesn't seem entirely obvious to me. This should either be documented or removed since the usage_text asks users to explicitly enter 127.0.0.1

Currently this is in line with the install_acm script from AI-Hunter.

Zalgo2462 · 2019-02-07T19:41:40Z

install-scripts/install_ipfix-rita.sh

+
+# Install docker if needed
+chmod +x "install-scripts/install_docker.sh"
+sudo install-scripts/install_docker.sh


It looks like we require administrator privileges up above with:

if [[ $EUID -ne 0 ]]; then echo "This script must be run with administrator privileges." exit 1 fi

So, we should be able to drop the sudo invocations throughout the install script

install-scripts/install_ipfix-rita.sh

Zalgo2462 · 2019-02-07T19:46:43Z

install-scripts/install_ipfix-rita.sh

+
+
+
+do_system_tests () {


This isn't called. Could we remove the function for the time being or fix it up?

Need to add this in

Zalgo2462 · 2019-02-07T19:54:58Z

install-scripts/install_ipfix-rita.sh

+	fi
+} #End of askYN
+
+check_remote_mongo() {


This will likely need to be a helper script or a rather large function to take care of all the security measures.

Zalgo2462 · 2019-02-07T19:56:03Z

install-scripts/install_ipfix-rita.sh

+			#echo "cannot connect to $@"
+		fi
+	else
+		echo "Please supply an ssh target as a command line parameter to can_ssh"


Does this need >&2

Zalgo2462 · 2019-02-07T19:59:34Z

install-scripts/install_ipfix-rita.sh

+	RITA_MONGO_AUTH="null"
+	#instead of prompting get the RITA address and split from there to fill in our
+	#  config file
+	echo ""


I think we want > or < &2 from here to the bottom of this function.

Zalgo2462 · 2019-02-07T20:01:05Z

install-scripts/install_ipfix-rita.sh

+# Stop if there are any errors
+set -e
+# Change dir to script dir
+echo "tyring to cd to $(dirname "$BASH_SOURCE[0]")"


We can probably remove this line.

Thanks, that was a debugging echo

Zalgo2462 · 2019-02-07T20:01:19Z

install-scripts/install_ipfix-rita.sh

+_OLD_DIR=$(pwd); cd "$(dirname "$BASH_SOURCE[0]")";
+
+if [[ $EUID -ne 0 ]]; then
+   echo "This script must be run with administrator privileges."


Looks like we're missing >&2 from here to the bottom of the file. (Mainly at the bottom)

…ing if the user doesn't have a mongo connection string

…ser's understand how everything is packaged in a release.

Zalgo2462 · 2019-02-14T22:14:05Z

I ran the installer in a vm and noticed the following issues:

Installing ipfix-rita in /opt/ipfix-rita
net.core.rmem_max = 67108864
dirname: missing operand
Try 'dirname --help' for more information.
basename: missing operand
Try 'basename --help' for more information.
basename: missing operand
Try 'basename --help' for more information.

This might stem from the ipfix-rita wrapper script.

EDIT: Indeed it does.

The copy of ubuntu 16 I am using to test does not symlink /etc/localtime, instead it copies the needed file there. In that case export TZ="$(basename $(dirname $(readlink /etc/localtime)))/$(basename $(readlink /etc/localtime))" does not work. Instead we should use something like timedatectl status | grep zone | cut -d':' -f2 | cut -d' ' -f2. This should be done in a separate PR.

…output is sent to stderr

…t installed

Merged master into feature branch in order to bring in changes necessary for testing.

Zalgo2462

Running the installer and just hitting enter breaks since the recommended MongoDB URI is deleted before it gets saved into the converter config.

Zalgo2462 · 2019-03-12T22:04:50Z

install-scripts/install_ipfix-rita.sh

+	echo "" >&2
+
+	echo "What MongoDB URI should IPFIX-RITA use to contact the RITA database [$RITA_MONGO_URI]: " >&2
+	read RITA_MONGO_URI <&2


This deletes the default RITA_MONGO_URI for the docker host.

The original code reads the response into the default REPLY variable to avoid this.

read -p "What MongoDB URI should IPFIX-RITA use to contact the RITA database [\$RITA_MONGO_URI]: " -r if [ -n "\$REPLY" ]; then RITA_MONGO_URI="\$REPLY" fi

Zalgo2462 · 2019-03-12T22:07:30Z

install-scripts/install_ipfix-rita.sh

+					#TODO, this cases issues, we have mongodb://127.0.0.1:27017 then we
+					# have ritauser@mongodb://127.0.0.1:27017 need to detect if they have
+					# mongodb and the port before trying to validate
+	        rita_system=`validated_ssh_target rita "$rita_system"`


We don't need to access the rita system with ssh in order to install ipfix-rita. (Unless we plan on installing rita as well)

Samuel Carroll added 15 commits January 4, 2019 13:37

Removing version from build script, and adding MongoDB restart comman…

2a45e6e

…d (might need to rearrange so it pops more)

Work in progress, I will still need to test this but good groundwork …

719dba9

…for breaking out IPFIX-RITA's installer to something nicer

Moving code to different sections to enable a non-interactive version…

4f736fb

… of the installer. Still in the testing/debugging stages

Made some modifications to the install script, still needs debugging …

719f578

…and additional work

Adding changes to make the install interactive functional. Need to ad…

1373dc5

…d mongo check and test/debug non-interactive commands

Adding changes to install script, works fine for non-interactive. Sti…

64d1038

…ll having awk(ward) problems

Finished the noninteractive flag debugging

cefbb03

Removing version from build script, and adding MongoDB restart comman…

5db2f7f

…d (might need to rearrange so it pops more)

Work in progress, I will still need to test this but good groundwork …

ea01e15

…for breaking out IPFIX-RITA's installer to something nicer

Moving code to different sections to enable a non-interactive version…

dbfc51f

… of the installer. Still in the testing/debugging stages

Made some modifications to the install script, still needs debugging …

a94e7dc

…and additional work

Adding changes to make the install interactive functional. Need to ad…

6e56f06

…d mongo check and test/debug non-interactive commands

Adding changes to install script, works fine for non-interactive. Sti…

e588ef8

…ll having awk(ward) problems

Finished the noninteractive flag debugging

5aa93e8

Resolving conflict with master

26361b0

Zalgo2462 suggested changes Feb 7, 2019

View reviewed changes

Samuel Carroll added 3 commits February 13, 2019 11:10

Modified script to read and write to &2, also some slight error check…

c1f6a79

…ing if the user doesn't have a mongo connection string

Adding a directory structure to start of the make-release script so u…

2edc981

…ser's understand how everything is packaged in a release.

Commenting out the system check for now, will add back in

b137746

Logan Lembke and others added 2 commits February 14, 2019 17:00

Fix bug where config value was sent to stderr. Ensure sysctl command …

cec22b2

…output is sent to stderr

Pull mongo image in case the user running make-release doesn't have i…

c12e04a

…t installed

Zalgo2462 force-pushed the 54-1-7-19-Fixes branch from c126074 to c12e04a Compare March 12, 2019 20:52

Merge branch 'master' into 54-1-7-19-Fixes

b73bb6f

Merged master into feature branch in order to bring in changes necessary for testing.

Zalgo2462 suggested changes Mar 12, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

54 Initial Fixes #82

54 Initial Fixes #82

SamuelCarroll commented Feb 6, 2019

Zalgo2462 left a comment •

edited

Loading

Zalgo2462 Feb 7, 2019

Zalgo2462 Feb 7, 2019

Zalgo2462 Feb 7, 2019

Zalgo2462 Feb 7, 2019

Zalgo2462 left a comment

Zalgo2462 Feb 7, 2019

SamuelCarroll Feb 8, 2019

SamuelCarroll Feb 14, 2019

Zalgo2462 Feb 7, 2019

SamuelCarroll Feb 8, 2019

Zalgo2462 Feb 7, 2019

Zalgo2462 Feb 7, 2019

SamuelCarroll Feb 8, 2019

Zalgo2462 Feb 7, 2019

Zalgo2462 Feb 7, 2019

Zalgo2462 Feb 7, 2019

Zalgo2462 Feb 7, 2019

SamuelCarroll Feb 8, 2019

Zalgo2462 Feb 7, 2019

Zalgo2462 commented Feb 14, 2019 •

edited

Loading

Zalgo2462 left a comment

Zalgo2462 Mar 12, 2019

Zalgo2462 Mar 12, 2019




		do_system_tests () {

54 Initial Fixes #82

Are you sure you want to change the base?

54 Initial Fixes #82

Conversation

SamuelCarroll commented Feb 6, 2019

Zalgo2462 left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zalgo2462 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zalgo2462 commented Feb 14, 2019 • edited Loading

Zalgo2462 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Zalgo2462 left a comment •

edited

Loading

Zalgo2462 commented Feb 14, 2019 •

edited

Loading