Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Specify env folder #53

Merged
merged 3 commits into from
Sep 15, 2019
Merged

Specify env folder #53

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6283750
Allow users to specify environments folder
sechitwood Sep 12, 2019
aff96e9
Update documentation
sechitwood Sep 12, 2019
e003e8d
Respond to PR comments
sechitwood Sep 13, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,15 +52,15 @@ $ whirl --help
The default action is to start the DAG in your current directory. It expects an environment to be configured. You can pass this as a command line argument or you can configure it in a `.whirl.env` file. (See #Configuring environment variables.) The environment refers to a directory with the same name in the `envs` directory located near the _whirl_ script.

```bash
$ whirl [start] [-e <environment>]
$ whirl [start] [-d <directory>] [-e <environment>]
```

Specifying the `start` command line argument is a more explicit way to start _whirl_.

#### Stopping whirl

```bash
$ whirl stop [-e <environment>]
$ whirl stop [-d <directory>] [-e <environment>]
```
Stops the configured environment.

Expand Down
33 changes: 32 additions & 1 deletion whirl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
AIRFLOW_UI_PORT=5000
SCRIPT_DIR=$( dirname ${BASH_SOURCE[0]} )

set -ev
# load and export environment variables
# priority order:
# .whirl.env in home directory
Expand Down Expand Up @@ -33,14 +34,38 @@ function export_environment_vars() {
echo "Found WHIRL_ENVIRONMENT in ${DAG_FOLDER}/.whirl.env";
WHIRL_ENVIRONMENT=$(grep -oRE "^(\s+)?WHIRL_ENVIRONMENT=(.*)" "${DAG_FOLDER}/.whirl.env" | \
sed -e 's/.*=\(.*\)/\1/g')
echo "environment is ${WHIRL_ENVIRONMENT}"
fi
fi
else
WHIRL_ENVIRONMENT=${WHIRL_ENVIRONMENT_ARG}
fi

ENVIRONMENT_FOLDER=${SCRIPT_DIR}/envs/${WHIRL_ENVIRONMENT}
# determine whether to use the environment dir set at the commandline or
# in the DAG FOLDER .whirl.env
echo ""
if [ -z "${WHIRL_ENVIRONMENT_DIR_ARG}" ]; then
if [ -f ${DAG_FOLDER}/.whirl.env ]; then
grep -qRE "^(\s+)?WHIRL_ENVIRONMENT_DIR=(.*)" "${DAG_FOLDER}/.whirl.env"
if grep -qRE "^(\s+)?WHIRL_ENVIRONMENT_DIR=(.*)" "${DAG_FOLDER}/.whirl.env"; then
echo "Found WHIRL_ENVIRONMENT_DIR in ${DAG_FOLDER}/.whirl.env";

grep -oRE "^(\s+)?WHIRL_ENVIRONMENT_DIR=(.*)" "${DAG_FOLDER}/.whirl.env";
grep -oRE "^(\s+)?WHIRL_ENVIRONMENT_DIR=(.*)" "${DAG_FOLDER}/.whirl.env" | sed -e 's/.*=\(.*\)/\1/g';
WHIRL_ENVIRONMENT_DIR=$(grep -oRE "^(\s+)?WHIRL_ENVIRONMENT_DIR=(.*)" "${DAG_FOLDER}/.whirl.env" | \
sed -e 's/.*=\(.*\)/\1/g')
echo "environment directory is ${WHIRL_ENVIRONMENT_DIR}"
fi
fi
else
WHIRL_ENVIRONMENT_DIR=${WHIRL_ENVIRONMENT_DIR_ARG}
fi

if [[ -z "${WHIRL_ENVIRONMENT_DIR}" ]]; then
WHIRL_ENVIRONMENT_DIR=${SCRIPT_DIR}/envs
fi
ENVIRONMENT_FOLDER=${WHIRL_ENVIRONMENT_DIR}/${WHIRL_ENVIRONMENT};

if [[ -z "${WHIRL_ENVIRONMENT}" || ! -d ${ENVIRONMENT_FOLDER} ]]; then
echo "No valid environment '${WHIRL_ENVIRONMENT}' specified"
exit 2;
Expand Down Expand Up @@ -120,6 +145,7 @@ usage() {
echo "usage: ${BASH_SOURCE[0]} [-h|--help] [-e|--environment env] [start|stop|ci]"
echo " -h|--help display usage"
echo " -e|--environment environment specify environment to use"
echo " -d|--directory environment_folder specify the folder that contains the environments (defaults to SCRIPT_DIR)"
echo " start|stop start or stop all"
echo " ci runs in daemonized mode and awaits dag run completion"
exit 1
Expand All @@ -135,6 +161,11 @@ function read_arguments() {
shift # past argument
shift # past value
;;
-d|--directory)
WHIRL_ENVIRONMENT_DIR_ARG="${2}"
shift # past argument
shift # past value
;;
start)
CI_MODE=false
shift
Expand Down