Skip to content

Server Configuration

Miro Kubicek edited this page Nov 20, 2019 · 7 revisions

There are two main system properties that titanoboa server checks during its startup:

Property Description
boa.server.dependencies.path clj file containing a map with external dependencies that need to be loaded - it may contain maven repositories, coordinates and items that need to be required or imported; the dependencies are loaded before the server configuration is which allows for great extensibility, e.g. if proper dependencies are added to external dependencies file then in the server configuration a user may define additional systems for 3rd party RDBS or MQ connection pooling etc...
boa.server.config.path server configuration clj file - this clj file will be evaluated in titanoboa.server namespace and allows for very flexible server configuration since additional code may be executed. Primarily this clj file may alter var titanoboa.server/server-config

External dependencies

External dependencies file sample:

{:coordinates [[amazonica "0.3.117" :exclusions [com.amazonaws/aws-java-sdk
                                                 com.amazonaws/amazon-kinesis-client
                                                 com.taoensso/nippy]]
               [com.amazonaws/aws-java-sdk-core "1.11.237"]
               [com.amazonaws/aws-java-sdk-sqs "1.11.237"]
               [com.amazonaws/aws-java-sdk-sns "1.11.237"]
               [com.amazonaws/aws-java-sdk-s3 "1.11.237"]
               [com.amazonaws/aws-java-sdk-ses "1.11.237"]
               [com.amazonaws/aws-java-sdk-ec2 "1.11.237"]
               [clj-pdf "2.2.33"]]
 :require [[amazonica.aws.s3]
           [amazonica.aws.s3transfer]
           [amazonica.aws.ec2]
           [amazonica.aws.simpleemail]
           [clj-pdf.core]]
 :import nil
 :repositories {"central" "https://repo1.maven.org/maven2/"
                "clojars" "https://clojars.org/repo"}}

This file is also visible via titanoboa's GUI (under systems tab) and can be edited during runtime. Its changes will trigger automatic reload (in clustered environment all nodes will perform the reload).

Server properties

Server properties sample:

(in-ns 'titanoboa.server)
(log/info "Hello, I am core.async server-config and I am being loaded...")
(defonce archival-queue-local (clojure.core.async/chan (clojure.core.async/dropping-buffer 1024)))
(alter-var-root #'server-config
                (constantly {:systems-catalogue
                                              {:core  {:system-def #'titanoboa.system.local/local-core-system
                                                       :worker-def #'titanoboa.system.local/local-worker-system
                                                       :autostart  true}}
                             :job-folder-path "job-folders/"
                             :enable-cluster false
                             :jetty {:join? false
                                     :port 3000}
                             :auth? false
                             :systems-config {:core
                                              {:new-jobs-chan (clojure.core.async/chan (clojure.core.async/dropping-buffer 1024))
                                               :jobs-chan (clojure.core.async/chan (clojure.core.async/dropping-buffer 1024))
                                               :finished-jobs-chan archival-queue-local
                                               :eviction-interval (* 1000 60 5)
                                               :eviction-age (* 1000 60 60)}}}))

This is probably the most minimalist server configuration. It contains Systems catalogue with just one core system. This core system just uses local (in-memory) core.async [job channel] (https://github.com/mikub/titanoboa/wiki#job-channel). There is no other non-core system defined - none to archive jobs (these end up in another core.async channel) or to secure the server etc.

Following sample is more extensive as it contains an archival system (that archives processed jobs into a postgres DB) and a security system (that verifies user credentials against a database). It also contains yet another system "test-db" that presumably contains another DB connection pool that is probably used from some workflow:

(in-ns 'titanoboa.server)
(log/info "Hello, I am core.async server-config and I am being loaded...")
(defonce archival-queue-local (clojure.core.async/chan (clojure.core.async/dropping-buffer 1024)))
(alter-var-root #'server-config
                (constantly {:systems-catalogue
                                               {:core  {:system-def #'titanoboa.system.local/local-core-system
                                                        :worker-def #'titanoboa.system.local/local-worker-system
                                                        :worker-count 2
                                                        :autostart  true}
                                                :test-db {:system-def #'titanoboa.system.jdbc/jdbc-pool
                                                          :autostart  true}
                                                :archival-system {:system-def #'titanoboa.system.local/archival-system
                                                                  :autostart  true}
                                                :auth-system {:system-def #'titanoboa.system.auth/auth-system
                                                              :autostart  true}}
                             :jobs-repo-path "repo/"
                             :steps-repo-path "step-repo/"
                             :job-folder-path "job-folders/"
                             :log-file-path "titanoboa.log"
                             :enable-cluster   false
                             :jetty {:ssl-port 443
                                     :join? false
                                     :ssl? true
                                     :http? false
                                     :keystore "keystore.jks"
                                     :key-password  "somepwdgoeshere"}
                             :archive-ds-ks [:archival-system :system :db-pool]
                             :auth-ds-ks [:auth-system :system :db-pool]
                             :auth? true
                             :auth-conf {:privkey "auth_privkey.pem"
                                         :passphrase "Ilovetitanoboa!"
                                         :pubkey  "auth_pubkey.pem"}
                             :systems-config {:core
                                              {:new-jobs-chan (clojure.core.async/chan (clojure.core.async/dropping-buffer 1024))
                                               :jobs-chan (clojure.core.async/chan (clojure.core.async/dropping-buffer 1024))
                                               :finished-jobs-chan archival-queue-local}
                                              :archival-system
                                              {:jdbc-url "jdbc:postgresql://localhost:5432/mydb?currentSchema=titanoboa"
                                               :user "postgres"
                                               :password "postgres"
                                               :finished-jobs-chan archival-queue-local}}}))

Security configuration

Titanoboa supports JWT authentication. It can be easily configured as a part of the server-config under :auth-conf key:

                             :auth? true
                             :auth-conf {:privkey "auth_privkey.pem"
                                         :passphrase "Ilovetitanoboa!"
                                         :pubkey  "auth_pubkey.pem"}
                             :auth-ds-ks [:auth-system :system :db-pool]

:auth-conf contains information regarding private/public key used for signing and :auth-ds-ks should provide link to a system property that provides a connection to the user database (more specifically to a datasource).

In this particular case it is the :auth-system defined in :systems-catalogue and the datasource is :db-pool so the whole keyword sequence is [:auth-system :system :db-pool].

Security extensions for webhooks

Each job definition has its own webhook URI that can be used to start a job. Naturally not all consumers of webhooks will support JWT - and so titanoboa makes it possible to define custom authentication function for any URI. In case an extension function is defined for give URI and no JWT token is found, this function is then used to authenticate. It takes the request as an argument and it is supposed to return a transformed request (e.g. to add :auth-user to the request or to return {:status 401} instead etc.)

Github example

For example Github can call webhooks, but it does not support JWT, instead it can provide a HMAC hex digest of the request body as a signature. To unsign webhook requests from Github, one could use following custom security extension in server-config:

:auth-conf       {:privkey    "auth_privkey.pem"
                  :passphrase "Ilovetitanoboa!"
                  :pubkey     "auth_pubkey.pem"
                  :extensions {"/systems/%3Acore/jobs/github2slack" (fn [request]
                                                                      (let [x-hub-signature (get-in request [:headers "x-hub-signature"])
                                                                            payload (slurp (:body request))]
                                                                        (let [payload-signature (str "sha1=" (pandect.algo.sha1/sha1-hmac payload "qnscAdgRlkIhAUPY44oiexBKtQbGY0orf7OV1I50"))]
                                                                          (if (crypto.equality/eq? payload-signature x-hub-signature)
                                                                            (assoc request :auth-user "github")
                                                                            {:status 401 :body "x-hub-signature does not match"}))))}}

This extension is simple and would just need following dependencies added to the External dependencies file:

{:coordinates [[pandect "0.6.1"]
               [crypto-equality "1.0.0"]]
 :require [[pandect.algo.sha1]
           [crypto.equality]]}

This way all requests going to URI /systems/%3Acore/jobs/github2slack (i.e. all requests starting a job "github2slack") will be authenticated using this custom security extension.

You can’t perform that action at this time.