[GH-1604] Add error handling to Slack service

[okotsopoulos]

Purpose

• https://broadinstitute.atlassian.net/browse/GH-1553 Slack integration test didn't pass in GitHub Actions

The Slack token is stored in Vault. We now read it into environment variable WFL_SLACK_TOKEN.

Atlantis syncs the value from Vault into GitHub Secrets, and we use it to set the environment variable as needed within GitHub Actions: https://github.com/broadinstitute/terraform-ap-deployments/pull/554

Updated documentation.

• https://broadinstitute.atlassian.net/browse/GH-1604 Slack not working in WFL dev or prod

Catch throwables within slack service, and add unit test.

System Tests

Start a local server with Slack notifications enabled:

$ WFL_SLACK_ENABLED=enabled ./ops/server.sh 

Run system test known to emit Slack message off of local server:

$ cd /path/to/wfl/api;
$ WFL_WFL_URL=http://localhost:3000 clojure -M:test --focus wfl.system.v1-endpoint-test/test-workload-sink-outputs-to-tdr

Resulting Slack message, and test passes:

--- system (clojure.test) ---------------------------
wfl.system.v1-endpoint-test
  test-workload-sink-outputs-to-tdr

1 tests, 7 assertions, 0 failures.

Then ran the full suite, all pass:

$ make TARGET=system
export CPCACHE=/Users/okotsopo/wfl/api/.cpcache;            \
	export WFL_WFL_URL=http://localhost:3000; \
	clojure  -M:parallel-test wfl.system.v1-endpoint-test | \
	tee /Users/okotsopo/wfl/derived/api/system.log
WARNING: Specified path is external to project: ../derived/api/src
WARNING: Specified path is external to project: ../derived/api/resources

Ran 33 tests containing 374 assertions.
0 failures, 0 errors.
api system finished on Mon Feb 28 14:07:14 EST 2022
docs system finished on Mon Feb 28 14:07:14 EST 2022
functions/aou system finished on Mon Feb 28 14:07:14 EST 2022
functions/sg system finished on Mon Feb 28 14:07:14 EST 2022
helm system finished on Mon Feb 28 14:07:14 EST 2022
ui system finished on Mon Feb 28 14:07:14 EST 2022

Review Instructions

• Take a look at the modified / added automated tests as a guide to slack usage, expected behavior.
• Please read through the Atlantis documentation I added, and let me know if it feels sufficient if you were working in that space.

[tbl3rd]

Still poking around, but not sure when I'll get back to it.