diff --git a/.gitignore b/.gitignore index 5d7f1d3..6becf3c 100644 --- a/.gitignore +++ b/.gitignore @@ -3,4 +3,5 @@ node_modules/* dist/*.js dist/*.js.map -carta-controller-*.tgz \ No newline at end of file +carta-controller-*.tgz +docs/_build diff --git a/README.md b/README.md index 341844c..e348b45 100644 --- a/README.md +++ b/README.md @@ -4,74 +4,8 @@ ![last commit](https://img.shields.io/github/last-commit/CARTAvis/carta-controller) ![commit activity](https://img.shields.io/github/commit-activity/m/CARTAvis/carta-controller) -## Introduction - The CARTA controller provides a simple dashboard which authenticates users and allows them to manage their CARTA backend processes. It also serves static frontend code to clients, and dynamically redirects authenticated client connections to the appropriate backend processes. The controller can either handle authentication itself, or delegate it to an external OAuth2-based authentication server. -## Dependencies - -To allow the controller to serve CARTA sessions, you must give it access to an executable CARTA backend, which can be either a compiled executable or a container. If you want to use a non-standard version of the CARTA frontend, you must also build it, and adjust the controller configuration to point to it. You should use the `v1.4.0` tag of [`carta-backend`](https://github.com/CARTAvis/carta-backend). - -By default, the controller runs on port 8000. It should be run behind a proxy, so that it can be accessed via HTTP and HTTPS. - -MongoDB is required for storing user preferences, layouts and (in the near future) controller metrics. You also need a working [NodeJS LTS](https://github.com/nvm-sh/nvm#long-term-support) installation with NPM. Use `npm install` to install all Node dependencies. - -## Installation - -You can install the CARTA controller from NPM by running `npm install -g carta-controller` and then running `carta-controller`. -You can also install the controller from GitHub by cloning this repository, running `npm install` and then `npm run start`. - -## Authentication support - -The CARTA controller supports three modes for authentication. All three modes use refresh and access tokens, as described in the [OAuth2 Authorization flow](https://tools.ietf.org/html/rfc6749#section-1.3.1), stored in [JWT](https://jwt.io/) format. The modes are: -- **LDAP-based authentication**: An existing LDAP server is used for user authentication. After the user's username and password configuration are validated by the LDAP server, `carta-controller` returns a long-lived refresh token, signed with a private key, which can be exchanged by the CARTA dashboard or the CARTA frontend client for a short-lived access token. -- **Google authentication**: Google's authentication libraries are used for handling authentication. You must create a new web application in the [Google API console](https://console.developers.google.com/apis/credentials). You will then use the client ID provided by this application in a number of places during the configuration. -- **External authentication**: This allows users to authenticate with some external OAuth2-based authentication system. This requires a fair amount of configuration, and has not been well-tested. It is assumed that the refresh token passed by the authentication system is stored as an `HttpOnly` cookie. - -You can generate a private/public key pair in PEM format using `openssl`: -```shell script -openssl genrsa -out carta_private.pem 4096 -openssl rsa -in carta_private.pem -outform PEM -pubout -out carta_public.pem -``` - -## Controller Configuration -Controller configuration is handled by a configuration file in JSON format, adhering to the [CARTA config schema](config/config_schema.json). Additional details can be found in the auto-generated config documentation in the `docs` folder, or the [example config](config/example_config.json). By default, the controller assumes the config file is located at `/etc/carta/config.json`, but you can change this with the `--config` or `-c` command line argument when running the controller. - -For external authentication systems, you may need to translate a unique ID (such as email or username) from the authenticated user information to the system user. You can do this by providing a [user lookup table](config/usertable.txt.stub), which is watched by the controller and reloaded whenever it is updated. - -You can alter the controller's dashboard appearance by adjusting the `dashboard` field in the config file. You can change the banner image and background, and add login instructions or institutional notices. - -## System Configuration - -The user under which the CARTA controller is running (assumed to be `carta`) must be given permission to use `sudo` to start `carta_backend` processes as any authenticated user and stop running `carta_backend` processes belonging to authenticated users. We provide a [kill script](scripts/carta_kill_script.sh) which is only able to kill processes matching the name `carta_backend`. This makes it possible to restrict what processes the `carta` user is permitted to kill. - -To provide the `carta` user with these privileges, you must make modifications to the [sudoers configuration](https://www.sudo.ws/man/1.9.0/sudoers.man.html). An [example sudoers config](config/example_sudoers_conf.stub) is provided. This example allows the `carta` user to run `carta_backend` only as users belonging to a specific group (assumed to be `carta-users`), in order to deny access to unauthorized accounts. - -**Please only edit your sudoers configuration with `visudo` or equivalent.** - -We strongly suggest serving over HTTPS and redirecting HTTP traffic to HTTPS, especially if handling authentication internally. If you use [nginx](https://www.nginx.com/) as a proxy, you can use [this configuration example](config/example_nginx.conf.stub) as a starting point to redirect incoming traffic from port 443 to port 8000. - -You can also use other HTTP servers, such as Apache. Please ensure that they are set up to forward both standard HTTP requests and WebSocket traffic to the correct port. - -By default, the controller attempts to write log files to the `/var/log/carta` directory. Please ensure that this directory exists and that the `carta` user has write permission. - -## Running the controller - -- Checkout and build [carta-backend](https://github.com/CARTAvis/carta-backend) using the `v1.4.0` tag (or create the appropriate container). Detailed instructions for Ubuntu 20.04 are available [here](docs/ubuntu_focal_detailed_install.md). -- Edit the controller configuration file at `/etc/carta/config.json` -- Perform system configuration: - - Ensure `/var/log/carta` exists and is writeable by the appropriate user - - Adjust the sudoers configuration - - Redirect traffic to port 8000 - -After you have built the backend and edited the controller configuration, you can start the controller with `npm run start` (if cloning from the git repository) or just running `carta-controller` (if installing from NPM). You can use a utility such as [forever](https://github.com/foreversd/forever) or [pm2](https://pm2.keymetrics.io/) to keep the controller running. - -## Getting help - -If you encounter a problem with the controller or documentation, please submit an issue in this repo. If you need assistance in configuration or deployment, please contact the [CARTA helpdesk](mailto:carta_helpdesk@asiaa.sinica.edu.tw). - -## TODO +For installation and configuration instructions, and more detailed information about the controller's features, please consult [the full documentation on ReadTheDocs](https://carta-controller.readthedocs.io/en/1.4/). -Still to be implemented: -- Better error feedback -- More flexibility with external auth +If you encounter a problem with the controller or documentation, please submit an issue in the controller repo. If you need assistance in configuration or deployment, please contact the [CARTA helpdesk](mailto:carta_helpdesk@asiaa.sinica.edu.tw). diff --git a/config/config_schema.json b/config/config_schema.json index 7a534d5..bbf3eb6 100644 --- a/config/config_schema.json +++ b/config/config_schema.json @@ -1,5 +1,5 @@ { - "$id": "https://example.com/person.schema.json", + "$id": "carta_config", "$schema": "http://json-schema.org/draft-07/schema#", "title": "CARTA configuration schema", "description": "Schema defining configuration options for the CARTA server", @@ -459,4 +459,4 @@ } } } -} \ No newline at end of file +} diff --git a/config/example_config.json b/config/example_config.json index f516be1..a99ab64 100644 --- a/config/example_config.json +++ b/config/example_config.json @@ -16,7 +16,7 @@ "databaseName": "CARTA" }, "serverAddress": "https://my-carta-server.com", - "processCommand": "/usr/local/bin/carta_backend", + "processCommand": "/usr/bin/carta_backend", "killCommand": "/usr/local/bin/carta_kill_script.sh", "rootFolderTemplate": "/home/{username}", "baseFolderTemplate": "/home/{username}", @@ -28,4 +28,4 @@ "loginText": "Click Sign in to log in via Google.", "footerText": "If you would like to access the CARTA server, or have any problems, comments or suggestions, please contact us." } -} \ No newline at end of file +} diff --git a/config/example_sudoers_conf.stub b/config/example_sudoers_conf.stub index ebafc78..f0c27be 100644 --- a/config/example_sudoers_conf.stub +++ b/config/example_sudoers_conf.stub @@ -1,7 +1,7 @@ -# customise this file to fit your environment using visudo /etc/sudoers.d/carta_server +# customise this file to fit your environment using visudo /etc/sudoers.d/carta_controller # carta user can run the carta_backend command as any user in the carta-users group without entering password -carta ALL=(%carta-users) NOPASSWD:SETENV: /usr/local/bin/carta_backend +carta ALL=(%carta-users) NOPASSWD:SETENV: /usr/bin/carta_backend # carta user can run the kill script as any user in the carta-users group without entering password -carta ALL=(%carta-users) NOPASSWD: /usr/local/bin/carta_kill_script.sh \ No newline at end of file +carta ALL=(%carta-users) NOPASSWD: /usr/local/bin/carta_kill_script.sh diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..c7c0696 --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,20 @@ +# Minimal makefile for Sphinx documentation +# + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = src +BUILDDIR = _build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/config_schema.html b/docs/config_schema.html deleted file mode 100644 index c9022ab..0000000 --- a/docs/config_schema.html +++ /dev/null @@ -1,57 +0,0 @@ - CARTA configuration schema

CARTA configuration schema

Type: object No Additional Properties

Schema defining configuration options for the CARTA server

Type: string

Reference to configuration schema file

Type: object

Configuration option for authentication providers

Type: object No Additional Properties

Google AuthProvider

Type: string

Google application client ID

 Must match regular expression: ^\S+.apps.googleusercontent.com$
Example:

"my-app-id.apps.googleusercontent.com"
-

Type: string

Valid domains to accept. If this is empty or undefined, all domains are accepted. Domain specified by hd field in Google authentication configuration.
Examples:

"gmail.com"
-
"my-google-domain.com"
-
""
-

Type: boolean Default: true

Whether to use the email field as a unique identifier
Examples:

true
-
false
-

Type: string

Path of user lookup table as text file in format . Example table given in usertable.txt.stub
Example:

"/etc/carta/userlookup.txt"
-

Type: object No Additional Properties

LDAP AuthProvider

Type: string

Path to public key (in PEM format) used for verifying JWTs
Example:

"/etc/carta/carta_public.pem"
-

Type: string

Path to private key (in PEM format) used for signing JWTs
Example:

"/etc/carta/carta_private.pem"
-

Type: enum (of string) Default: "RS256"

Algorithm used for public/private keys

Must be one of:

  • "HS256"
  • "HS384"
  • "HS512"
  • "RS256"
  • "RS384"
  • "RS512"
  • "ES256"
  • "ES384"
  • "ES512"
  • "PS256"
  • "PS384"
  • "PS512"

Type: string

Issuer field for JWT
Example:

"my-carta-server"
-

Type: string Default: "1w"

Lifetime of refresh tokens
Examples:

"1w"
-
"15h"
-
"2d"
-

Type: string Default: "15m"

Lifetime of access tokens
Examples:

"90s"
-
"1h"
-
"15m"
-

Type: object No Additional Properties

Options to path through to the LDAP auth instance

Type: string

LDAP connection URI

 Must match regular expression: ^ldaps?://

Type: string

Search base

Type: string Default: "uid={{username}}"

Search filter to use

Type: boolean Default: true

Whether to start TLS when making a connection

Type: boolean Default: true

Whether to automatically reconnect to LDAP

Type: object No Additional Properties

External AuthProvider

Type: array of string

List of valid issuers in JWT field

Must contain a minimum of 1 items

Each item of this array must be:

Type: string

Example:

[
-    "my-auth-server",
-    "my-other-auth-server"
-]
-

Type: string

Path to public key (in PEM format) used for verifying JWTs
Example:

"/etc/carta/my_auth_server_public_key.pem"
-

Type: enum (of string) Default: "RS256"

Algorithm used for public/private keys

Must be one of:

  • "HS256"
  • "HS384"
  • "HS512"
  • "RS256"
  • "RS384"
  • "RS512"
  • "ES256"
  • "ES384"
  • "ES512"
  • "PS256"
  • "PS384"
  • "PS512"

Type: string

Name of unique field to use as user ID
Examples:

"user"
-
"sub"
-
"user_id"
-

Type: string

Route for refreshing access tokens

 Must match regular expression: ^https?://

Type: string

Route for logging out

 Must match regular expression: ^https?://

Type: string

Path of user lookup table as text file in format . If no user lookup is needed, this should be omitted. Example table given in usertable.txt.stub
Example:

"/etc/carta/userlookup.txt"
-

Type: object No Additional Properties

Database configuration

Type: string Default: "mongodb://localhost:27017"

MongoDB connection URI used to connect to a MongoDB deployment

 Must match regular expression: ^mongodb://

Type: string Default: "CARTA"

Default database to connect to

Type: integer Default: 8000

Port to listen on. It is advised to listen on a port other than 80 or 443, behind an SSL proxy

Value must be greater or equal to 0 and lesser or equal to 65535

Type: string

Public-facing server address

 Must match regular expression: ^https?://

Type: string

Optional parameter for explicitly configuring a custom dashboard address

 Must match regular expression: ^https?://

Type: string

Optional parameter for explicitly configuring a custom API base address

 Must match regular expression: ^https?://

Type: string

Path to the built frontend folder. If no path is provided, the packaged version will be used

Type: object Default: {"min": 3003, "max": 3500} No Additional Properties

Port range to use for the CARTA backend process

Type: integer

Value must be greater or equal to 1024 and lesser or equal to 65535

Type: integer

Value must be greater or equal to 1024 and lesser or equal to 65535

Type: string

Path to CARTA backend executable
Example:

"/usr/local/bin/carta_backend"
-

Type: string

Path to CARTA kill script
Example:

"/usr/local/bin/carta_kill_script.sh"
-

Type: string

Top-level path of directories accessible to CARTA. The {username} placeholder will be replaced with the username
Examples:

"/home/{username}"
-
"/"
-

Type: string

Starting directory of CARTA. Must be a subfolder of rootFolderTemplate. The {username} placeholder will be replaced with the username
Examples:

"/home/{username}/CARTA"
-
"/data"
-
"/"
-

Type: string Default: "/var/log/carta/{username}_{datetime}_{pid}.log"

Location of log file. The {username}, {pid} and {datetime} placeholders will be replaced with the username, process ID. and dat/time formatted as YYYYMMDD.h_mm_ss respectively
Examples:

"/var/log/carta/{username}_{pid}.log"
-
"/home/{username}/CARTA/log/{datatime}_{pid}.log"
-

Type: array of string

Additional arguments to be passed to the backend process, defined as an array of strings. See backend documentation for details.

Each item of this array must be:

Type: string

Example:

[
-    "-threads",
-    "4",
-    "-omp_threads",
-    "4",
-    "-init_exit_after",
-    "30",
-    "-exit_after",
-    "0"
-]
-

Type: integer Default: 250

Wait time before checking whether started process is still running and sending a response to the connecting client

Value must be greater or equal to 0

Type: object

Dashboard appearance configuration

Type: string Default: "#f6f8fa"

Background color for the dashboard
Examples:

"red"
-
"rgb(171 66 66)"
-
"#ff11ee"
-

Type: string Default: "#606f7e"

Background color for the institutional logo banner
Examples:

"red"
-
"rgb(171 66 66)"
-
"#ff11ee"
-

Type: string

Path to institutional logo in PNG or SVG format

Type: string

Text displayed before and after sign in. Plain text or HTML
Examples:

"Welcome to the server"
-
"<span>Welcome to <b>the</b> server</span>"
-

Type: string

Text displayed before sign-in only. Plain text or HTML
Examples:

"Please enter your username and password"
-
"<span>Click <b>Sign in</b> to log in via Google</span>"
-

Type: string

Footer text. Plain text or HTML
Examples:

"Please contact the CARTA helpdesk for more information"
-
"<span>If you would like to access the server, or have any problems, comments or suggestions, please <a href='mailto:test@test.com'>contact us.</a></span>"
-
\ No newline at end of file diff --git a/docs/requirements.txt b/docs/requirements.txt new file mode 100644 index 0000000..ee76eaf --- /dev/null +++ b/docs/requirements.txt @@ -0,0 +1,3 @@ +sphinx=3.5.3 +sphinx-jsonschema==1.16.7 +sphinx-rtd-theme==0.5.1 diff --git a/docs/schema_doc.css b/docs/schema_doc.css deleted file mode 100644 index c0de87c..0000000 --- a/docs/schema_doc.css +++ /dev/null @@ -1,175 +0,0 @@ -body { - font: 16px/1.5em "Overpass", "Open Sans", Helvetica, sans-serif; - color: #333; - font-weight: 300; - padding: 40px; -} - -.btn.btn-link { - font-size: 18px; -} - -.jsfh-animated-property { - animation: eclair; - animation-iteration-count: 1; - animation-fill-mode: forwards; - animation-duration: .75s; - -} - -@keyframes eclair { - 0%,100% { - transform: scale(1); - } - 50% { - transform: scale(1.03); - } -} - -.btn.btn-primary { - margin: 10px; -} - -.btn.example-show.collapsed:before { - content: "show" -} - -.btn.example-show:before { - content: "hide" -} - -.description.collapse:not(.show) { - max-height: 100px !important; - overflow: hidden; - - display: -webkit-box; - -webkit-line-clamp: 2; - -webkit-box-orient: vertical; -} - -.description.collapsing { - min-height: 100px !important; -} - -.collapse-description-link.collapsed:after { - content: '+ Read More'; -} - -.collapse-description-link:not(.collapsed):after { - content: '- Read Less'; -} - -.badge { - font-size: 100%; - margin-bottom: 0.5rem; -} - -.badge.value-type { - font-size: 120%; - margin-right: 5px; - margin-bottom: 10px; -} - - -.badge.default-value { - font-size: 120%; - margin-left: 5px; - margin-bottom: 10px; -} - -.badge.restriction { - display: inline-block; -} - -.badge.required-property,.badge.deprecated-property,.badge.pattern-property,.badge.no-additional { - font-size: 100%; - margin-left: 10px; -} - -.accordion div.card:only-child { - border-bottom: 1px solid rgba(0, 0, 0, 0.125); -} - -.examples { - padding: 1rem !important; -} - -.highlight.jumbotron { - padding: 1rem !important; -} - -.generated-by-footer { - margin-top: 1em; - text-align: right; -} - -/* From https://github.com/richleland/pygments-css/blob/master/friendly.css, see https://github.com/trentm/python-markdown2/wiki/fenced-code-blocks */ -.highlight { background: #e9ecef; } /* Changed from #f0f0f0 in the original style to be the same as bootstrap's jumbotron */ -.highlight .hll { background-color: #ffffcc } -.highlight .c { color: #60a0b0; font-style: italic } /* Comment */ -.highlight .err { border: 1px solid #FF0000 } /* Error */ -.highlight .k { color: #007020; font-weight: bold } /* Keyword */ -.highlight .o { color: #666666 } /* Operator */ -.highlight .ch { color: #60a0b0; font-style: italic } /* Comment.Hashbang */ -.highlight .cm { color: #60a0b0; font-style: italic } /* Comment.Multiline */ -.highlight .cp { color: #007020 } /* Comment.Preproc */ -.highlight .cpf { color: #60a0b0; font-style: italic } /* Comment.PreprocFile */ -.highlight .c1 { color: #60a0b0; font-style: italic } /* Comment.Single */ -.highlight .cs { color: #60a0b0; background-color: #fff0f0 } /* Comment.Special */ -.highlight .gd { color: #A00000 } /* Generic.Deleted */ -.highlight .ge { font-style: italic } /* Generic.Emph */ -.highlight .gr { color: #FF0000 } /* Generic.Error */ -.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */ -.highlight .gi { color: #00A000 } /* Generic.Inserted */ -.highlight .go { color: #888888 } /* Generic.Output */ -.highlight .gp { color: #c65d09; font-weight: bold } /* Generic.Prompt */ -.highlight .gs { font-weight: bold } /* Generic.Strong */ -.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */ -.highlight .gt { color: #0044DD } /* Generic.Traceback */ -.highlight .kc { color: #007020; font-weight: bold } /* Keyword.Constant */ -.highlight .kd { color: #007020; font-weight: bold } /* Keyword.Declaration */ -.highlight .kn { color: #007020; font-weight: bold } /* Keyword.Namespace */ -.highlight .kp { color: #007020 } /* Keyword.Pseudo */ -.highlight .kr { color: #007020; font-weight: bold } /* Keyword.Reserved */ -.highlight .kt { color: #902000 } /* Keyword.Type */ -.highlight .m { color: #40a070 } /* Literal.Number */ -.highlight .s { color: #4070a0 } /* Literal.String */ -.highlight .na { color: #4070a0 } /* Name.Attribute */ -.highlight .nb { color: #007020 } /* Name.Builtin */ -.highlight .nc { color: #0e84b5; font-weight: bold } /* Name.Class */ -.highlight .no { color: #60add5 } /* Name.Constant */ -.highlight .nd { color: #555555; font-weight: bold } /* Name.Decorator */ -.highlight .ni { color: #d55537; font-weight: bold } /* Name.Entity */ -.highlight .ne { color: #007020 } /* Name.Exception */ -.highlight .nf { color: #06287e } /* Name.Function */ -.highlight .nl { color: #002070; font-weight: bold } /* Name.Label */ -.highlight .nn { color: #0e84b5; font-weight: bold } /* Name.Namespace */ -.highlight .nt { color: #062873; font-weight: bold } /* Name.Tag */ -.highlight .nv { color: #bb60d5 } /* Name.Variable */ -.highlight .ow { color: #007020; font-weight: bold } /* Operator.Word */ -.highlight .w { color: #bbbbbb } /* Text.Whitespace */ -.highlight .mb { color: #40a070 } /* Literal.Number.Bin */ -.highlight .mf { color: #40a070 } /* Literal.Number.Float */ -.highlight .mh { color: #40a070 } /* Literal.Number.Hex */ -.highlight .mi { color: #40a070 } /* Literal.Number.Integer */ -.highlight .mo { color: #40a070 } /* Literal.Number.Oct */ -.highlight .sa { color: #4070a0 } /* Literal.String.Affix */ -.highlight .sb { color: #4070a0 } /* Literal.String.Backtick */ -.highlight .sc { color: #4070a0 } /* Literal.String.Char */ -.highlight .dl { color: #4070a0 } /* Literal.String.Delimiter */ -.highlight .sd { color: #4070a0; font-style: italic } /* Literal.String.Doc */ -.highlight .s2 { color: #4070a0 } /* Literal.String.Double */ -.highlight .se { color: #4070a0; font-weight: bold } /* Literal.String.Escape */ -.highlight .sh { color: #4070a0 } /* Literal.String.Heredoc */ -.highlight .si { color: #70a0d0; font-style: italic } /* Literal.String.Interpol */ -.highlight .sx { color: #c65d09 } /* Literal.String.Other */ -.highlight .sr { color: #235388 } /* Literal.String.Regex */ -.highlight .s1 { color: #4070a0 } /* Literal.String.Single */ -.highlight .ss { color: #517918 } /* Literal.String.Symbol */ -.highlight .bp { color: #007020 } /* Name.Builtin.Pseudo */ -.highlight .fm { color: #06287e } /* Name.Function.Magic */ -.highlight .vc { color: #bb60d5 } /* Name.Variable.Class */ -.highlight .vg { color: #bb60d5 } /* Name.Variable.Global */ -.highlight .vi { color: #bb60d5 } /* Name.Variable.Instance */ -.highlight .vm { color: #bb60d5 } /* Name.Variable.Magic */ -.highlight .il { color: #40a070 } /* Literal.Number.Integer.Long */ \ No newline at end of file diff --git a/docs/schema_doc.min.js b/docs/schema_doc.min.js deleted file mode 100644 index 0d9c788..0000000 --- a/docs/schema_doc.min.js +++ /dev/null @@ -1 +0,0 @@ -function flashElement(t){myElement=document.getElementById(t),myElement.classList.add("jsfh-animated-property"),setTimeout(function(){myElement.classList.remove("jsfh-animated-property")},1e3)}function setAnchor(t){history.pushState({},"",t)}function anchorOnLoad(){let t=window.location.hash.split("?")[0].split("&")[0];"#"===t[0]&&(t=t.substr(1)),t.length>0&&anchorLink(t)}function anchorLink(t){$("#"+t).parents().addBack().filter(".collapse:not(.show), .tab-pane, [role='tab']").each(function(t){if($(this).hasClass("collapse"))$(this).collapse("show");else if($(this).hasClass("tab-pane")){const t=$("a[href='#"+$(this).attr("id")+"']");t&&t.tab("show")}else"tab"===$(this).attr("role")&&$(this).tab("show")}),setTimeout(function(){let e=document.getElementById(t);e&&(e.scrollIntoView({block:"center",behavior:"smooth"}),setTimeout(function(){flashElement(t)},500))},1e3)}$(document).on("click",'a[href^="#"]',function(t){t.preventDefault(),history.pushState({},"",this.href)}); \ No newline at end of file diff --git a/docs/src/_static/config b/docs/src/_static/config new file mode 120000 index 0000000..101d543 --- /dev/null +++ b/docs/src/_static/config @@ -0,0 +1 @@ +../../../config \ No newline at end of file diff --git a/docs/src/_static/css/custom.css b/docs/src/_static/css/custom.css new file mode 100644 index 0000000..cb129fc --- /dev/null +++ b/docs/src/_static/css/custom.css @@ -0,0 +1,16 @@ +/* override table width restrictions -- this is needed for the Read The Docs theme */ +@media screen and (min-width: 767px) { + + .wy-nav-content { + max-width: 100% !important; + } + + .wy-table-responsive table td, .wy-table-responsive table th { + white-space: normal !important;; + } + + .wy-table-responsive { + overflow: visible !important; + max-width: 100%; + } +} diff --git a/docs/src/_static/scripts b/docs/src/_static/scripts new file mode 120000 index 0000000..bbaa944 --- /dev/null +++ b/docs/src/_static/scripts @@ -0,0 +1 @@ +../../../scripts \ No newline at end of file diff --git a/docs/src/conf.py b/docs/src/conf.py new file mode 100644 index 0000000..b09b3f8 --- /dev/null +++ b/docs/src/conf.py @@ -0,0 +1,61 @@ +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a full +# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + + +# -- Project information ----------------------------------------------------- + +project = 'CARTA Controller' +copyright = '2021, Angus Comrie, Adrianna Pińska and Robert Simmonds' +author = 'Angus Comrie, Adrianna Pińska and Robert Simmonds' + +# The full version, including alpha/beta/rc tags +release = '1.4' + + +# -- General configuration --------------------------------------------------- + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + 'sphinx_rtd_theme', + 'sphinx-jsonschema', +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] + + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'sphinx_rtd_theme' + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] + +html_css_files = [ + 'css/custom.css', +] diff --git a/docs/src/configuration.rst b/docs/src/configuration.rst new file mode 100644 index 0000000..45a89a5 --- /dev/null +++ b/docs/src/configuration.rst @@ -0,0 +1,84 @@ +.. _configuration: + +Configuration +============= + +.. _config-system: + +System Configuration +-------------------- + +.. _config-backend: + +CARTA backend permissions +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The user under which the CARTA controller is running (assumed to be ``carta``) must be given permission to use ``sudo`` to start ``carta_backend`` processes as any authenticated user and stop running ``carta_backend`` processes belonging to authenticated users. We provide a `kill script <_static/scripts/carta_kill_script.sh>`_ which is only able to kill processes matching the name ``carta_backend``. This makes it possible to restrict what processes the ``carta`` user is permitted to kill: + +.. literalinclude:: _static/scripts/carta_kill_script.sh + :language: shell + +To provide the ``carta`` user with these privileges, you must make modifications to the `sudoers configuration `_. An `example sudoers config <_static/config/example_sudoers_conf.stub>`_ is provided. This example allows the ``carta`` user to run ``carta_backend`` only as users belonging to a specific group (assumed to be ``carta-users``), in order to deny access to unauthorized accounts: + +.. literalinclude:: _static/config/example_sudoers_conf.stub + :language: cfg + :name: example_sudoers + +.. warning:: + Please only edit your sudoers configuration with ``visudo`` or equivalent. + +.. _config-authentication: + +Authentication +~~~~~~~~~~~~~~ + +When configured to use LDAP authentication, the controller signs and validates refresh and access tokens with SSL keys. You can generate a private/public key pair in PEM format using ``openssl``: + +.. code-block:: shell + + cd /etc/carta + openssl genrsa -out carta_private.pem 4096 + openssl rsa -in carta_private.pem -outform PEM -pubout -out carta_public.pem + +.. _config-nginx: + +Nginx +~~~~~ + +We strongly suggest serving over HTTPS and redirecting HTTP traffic to HTTPS, especially if handling authentication internally. If you use `nginx `_ as a proxy, you can use `this configuration example <_static/config/example_nginx.conf.stub>`_ as a starting point to redirect incoming traffic from port 443 to port 8000: + +.. literalinclude:: _static/config/example_nginx.conf.stub + :language: nginx + :emphasize-lines: 7-15 + :name: example_nginx + +You can also use other HTTP servers, such as Apache. Please ensure that they are set up to forward both standard HTTP requests and WebSocket traffic to the correct port. + +.. _config-dirs: + +Directories +~~~~~~~~~~~ + +By default, the controller attempts to write log files to the ``/var/log/carta`` directory. Please ensure that this directory exists and that the ``carta`` user has write permission. + +.. _config-controller: + +Controller Configuration +------------------------ + +Controller configuration is handled by a configuration file in JSON format, adhering to the :ref:`CARTA configuration schema`. An `example configuration file <_static/config/example_config.json>`_ is provided: + +.. literalinclude:: _static/config/example_config.json + :language: json + :name: example_config + +By default, the controller assumes the config file is located at ``/etc/carta/config.json``, but you can change this with the ``--config`` or ``-c`` command line argument when running the controller. + +The controller automatically executes the backend with the ``--no_http`` flag, to suppress the backend's built-in HTTP server. If the ``logFileTemplate`` configuration option is set, ``--no_log`` is also used to suppress user-level logs. ``--port`` is used to override the default port. ``--top_level_folder`` and a positional argument are used to set the top-level and starting data directories for the user, as specified in the ``rootFolderTemplate`` and ``baseFolderTemplate`` options, respectively. Additional backend flags may be specified with ``additionalArgs``. + +If you use an external :ref:`authentication` system, you may need to translate a unique ID (such as email or username) from the authenticated external user information to an internal system user. You can do this by providing a `user lookup table <_static/config/usertable.txt.stub>`_, which is watched by the controller and reloaded whenever it is updated: + +.. literalinclude:: _static/config/usertable.txt.stub + :language: cfg + +You can alter the controller's dashboard appearance by adjusting the ``dashboard`` field in the config file. You can change the banner image and background, and add login instructions or institutional notices. diff --git a/docs/src/index.rst b/docs/src/index.rst new file mode 100644 index 0000000..1212c05 --- /dev/null +++ b/docs/src/index.rst @@ -0,0 +1,37 @@ +.. CARTA Controller documentation master file, created by + sphinx-quickstart on Wed Mar 10 15:04:08 2021. + You can adapt this file completely to your liking, but it should at least + contain the root `toctree` directive. + +CARTA Controller +================ + +|backend-github| |npm-package| |last-commit| |commit-activity| + +CARTA is the Cube Analysis and Rendering Tool for Astronomy. This document describes the installation and configuration process for the controller component. + +Detailed :ref:`step-by-step instructions` are provided for Ubuntu 20.04 (Focal Fossa). + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + introduction + installation + configuration + ubuntu_focal_instructions + schema + +.. |backend-github| image:: https://img.shields.io/badge/CARTA%20Version-1.4.0-brightgreen + :alt: View this backend version on GitHub + :target: https://github.com/CARTAvis/carta-backend/releases/tag/v1.4.0 + +.. |npm-package| image:: http://img.shields.io/npm/v/carta-controller.svg?style=flat + :alt: View this project on npm + :target: https://npmjs.org/package/carta-controller + +.. |last-commit| image:: https://img.shields.io/github/last-commit/CARTAvis/carta-controller + :alt: Last commit + +.. |commit-activity| image:: https://img.shields.io/github/commit-activity/m/CARTAvis/carta-controller + :alt: Commit activity diff --git a/docs/src/installation.rst b/docs/src/installation.rst new file mode 100644 index 0000000..4e4c04f --- /dev/null +++ b/docs/src/installation.rst @@ -0,0 +1,38 @@ +.. _installation: + +Installation +============ + +.. _install_backend: + +Installing the backend +---------------------- + +We provide `binary Debian packages `_ of the latest development and release versions of the CARTA backend for Ubuntu 20.04 (Focal Fossa) and Ubuntu 18.04 (Bionic Beaver). You can install the release version with all dependencies by adding our PPA to your system and running ``apt-get install carta-backend``. Please refer to our :ref:`Ubuntu Focal instructions` for more details. + +.. note:: + + The ``casacore-data`` package is recommended by the backend package, but installing it is optional. The packages in our PPA should be compatible both with the ``casacore-data`` package in the core Ubuntu repositories and with the package provided by the `Kern PPAs `_. You may also wish to manage the required data files without using a package. + +To install the backend on a different host system, or to install a custom version, you can build it from source from the `backend repository `_ on GitHub. + +.. _install_frontend: + +Installing the frontend +----------------------- + +If you install the controller from NPM, the corresponding packaged version of the frontend will also be installed automatically. If you wish to install the controller from source, or would like to use a custom frontend version, you can install it from the `frontend repository `_ on GitHub. + +.. _install_controller: + +Installing the controller +------------------------- + +You can install the release version of the CARTA controller from NPM by running ``npm install -g carta-controller``, or from GitHub by cloning the `controller repository `_, checking out the ``release/1.4`` branch, and running ``npm install``. + +.. _run_controller: + +Running the controller +---------------------- + +After you have installed the backend and the controller and edited the controller :ref:`configuration`, you can start the controller with ``npm run start`` (if installing from the source on GitHub) or just by running ``carta-controller`` (if installing the package from NPM). You can use a utility such as `forever `_ or `pm2 `_ to keep the controller running. It is also possible to create `a pm2 startup script `_ which will automatically start the controller when the system is rebooted. diff --git a/docs/src/introduction.rst b/docs/src/introduction.rst new file mode 100644 index 0000000..800d92a --- /dev/null +++ b/docs/src/introduction.rst @@ -0,0 +1,47 @@ +.. _introduction: + +Introduction +============ + +The CARTA controller provides a simple dashboard which authenticates users and allows them to manage their CARTA backend processes. It also serves static frontend code to clients, and dynamically redirects authenticated client connections to the appropriate backend processes. The controller can either handle authentication itself, or delegate it to an external OAuth2-based authentication server. + +.. _dependencies: + +Dependencies +------------ + +To allow the controller to serve CARTA sessions, you must give it access to an executable CARTA backend, which can be either a compiled executable or a container. If you want to use a non-standard version of the CARTA frontend, you must also build it, and adjust the controller configuration to point to it. You should use the ``v1.4.0`` tag of `the CARTA backend `_. + +By default, the controller runs on port 8000. It should be run behind a proxy, so that it can be accessed via HTTP and HTTPS. + +MongoDB is required for storing user preferences, layouts and (in the near future) controller metrics. + +You also need a working `NodeJS LTS `_ installation with NPM. Use ``npm install`` to install all Node dependencies. + +.. _authentication: + +Authentication support +---------------------- + +The CARTA controller supports three modes for authentication. All three modes use refresh and access tokens, as described in the `OAuth2 Authorization flow `_, stored in `JWT `_ format. The modes are: + +* **LDAP-based authentication**: An existing LDAP server is used for user authentication. After the user's username and password configuration are validated by the LDAP server, ``carta-controller`` returns a long-lived refresh token, signed with a private key, which can be exchanged by the CARTA dashboard or the CARTA frontend client for a short-lived access token. +* **Google authentication**: Google's authentication libraries are used for handling authentication. You must create a new web application in the `Google API console `_. You will then use the client ID provided by this application in a number of places during the configuration. +* **External authentication**: This allows users to authenticate with some external OAuth2-based authentication system. This requires a fair amount of configuration, and has not been well-tested. It is assumed that the refresh token passed by the authentication system is stored as an ``HttpOnly`` cookie. + +.. _getting_help: + +Getting help +------------ + +If you encounter a problem with the controller or documentation, please submit an issue in the controller repo. If you need assistance in configuration or deployment, please contact the `CARTA helpdesk `_. + +.. _future_work: + +Future work +----------- + +Features still to be implemented: + +* Better error feedback +* More flexibility with external auth diff --git a/docs/src/schema.rst b/docs/src/schema.rst new file mode 100644 index 0000000..789cd90 --- /dev/null +++ b/docs/src/schema.rst @@ -0,0 +1,7 @@ +.. _schema: + +.. jsonschema:: _static/config/config_schema.json + :lift_description: + :lift_definitions: + :auto_target: + :auto_reference: diff --git a/docs/src/ubuntu_focal_instructions.rst b/docs/src/ubuntu_focal_instructions.rst new file mode 100644 index 0000000..f0081ba --- /dev/null +++ b/docs/src/ubuntu_focal_instructions.rst @@ -0,0 +1,104 @@ +.. _focal_instructions: + +Step-by-step instructions for Ubuntu 20.04.2 (Focal Fossa) +========================================================== + +Dependencies +------------ + +Install the CARTA backend and other required packages +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. code-block:: shell + + # Add CARTA PPA + sudo add-apt-repository ppa:cartavis-team/carta + sudo apt-get update + + # Install the release backend package with all dependencies + sudo apt-get install carta-backend + + # Install additional packages + sudo apt-get install nginx curl g++ mongodb make + +Set up directories and permissions +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Ensure that all users who should have access to CARTA belong to a group that identifies them (assumed here to be called ``carta-users``). Also ensure that LDAP has been set up correctly. + +.. code-block:: shell + + # create a 'carta' user to run the controller + sudo adduser --disabled-login --gecos "" carta + + # log directory owned by carta + sudo mkdir -p /var/log/carta + sudo chown carta: /var/log/carta + + # config directory owned by carta + sudo mkdir -p /etc/carta + sudo chown carta: /etc/carta + + # edit sudoers file to allow passwordless sudo execution of + # /usr/local/bin/carta_kill_script.sh and /usr/bin/carta_backend + # by the carta user + sudo visudo -f /etc/sudoers.d/carta_controller + +An :ref:`example sudoers configuration` is provided in the configuration section. + +Configure nginx +~~~~~~~~~~~~~~~ + +A :ref:`sample configuration file` is provided in the configuration section. This should be adapted to your server configuration. The relevant part of the config is for forwarding ``/`` to port 8000. + +Install CARTA controller +------------------------ + +.. code-block:: shell + + # Most of these commands should be executed as the carta user + sudo su - carta + + # Install NVM and NPM + cd ~ + curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.37.2/install.sh | bash + source .bashrc + nvm install --lts + nvm install-latest-npm + + # Install carta-controller (includes frontend config) + npm install -g carta-controller + + # For security reasons, copy the kill script to a system bin directory + cp ${NVM_BIN}/../lib/node_modules/carta-controller/scripts/carta_kill_script.sh ~ + exit + sudo mv /home/carta/carta_kill_script.sh /usr/local/bin/ + sudo chown root: /usr/local/bin/carta_kill_script.sh + sudo su - carta + + # Generate private/public keys + cd /etc/carta + openssl genrsa -out carta_private.pem 4096 + openssl rsa -in carta_private.pem -outform PEM -pubout -out carta_public.pem + +Configure controller +~~~~~~~~~~~~~~~~~~~~ + +Edit ``/etc/carta/config.json`` to customise the appearance of the dashboard and other options. A :ref:`sample configuration file` is provided in the configuration section. + +Run controller +~~~~~~~~~~~~~~ + +This should be executed as the ``carta`` user. + +.. code-block:: shell + + # Install PM2 node service + npm install -g pm2 + pm2 start carta-controller + +Create pm2 startup script +~~~~~~~~~~~~~~~~~~~~~~~~~ + +This service will start the controller automatically after a reboot. Please refer to the `pm2 documentation `_ for detailed instructions. You should run ``pm2 startup`` as ``carta``, execute the generated command as a user with ``sudo`` access, and finally run ``pm2 save`` as ``carta`` to save the running controller process. + diff --git a/docs/ubuntu_focal_detailed_install.md b/docs/ubuntu_focal_detailed_install.md deleted file mode 100644 index 11f4fe9..0000000 --- a/docs/ubuntu_focal_detailed_install.md +++ /dev/null @@ -1,128 +0,0 @@ -# Detailed instructions for setting up `carta-controller` on Ubuntu 20.04.1 (Focal Fossa) - -## Dependencies: -### Install required packages -```shell script -# assuming user "ubuntu" with home directory /home/ubuntu -sudo apt update -sudo apt install -y git software-properties-common cmake pkg-config subversion g++ gfortran libzstd-dev \ -libfmt-dev libprotobuf-dev protobuf-compiler libhdf5-dev libtbb-dev libssl-dev libcurl4-openssl-dev \ -libgsl-dev nginx build-essential libncurses5-dev libreadline-dev flex bison libblas-dev \ -liblapacke-dev libcfitsio-dev wcslib-dev mongodb libgrpc++-dev protobuf-compiler-grpc libpugixml-dev -``` - -### Compile other dependencies -```shell script -mkdir -p repos - -# ZFP -cd ~/repos -git clone --branch 0.5.5 https://github.com/LLNL/zfp.git -cd zfp -mkdir -p build && cd build -cmake -DCMAKE_BUILD_TYPE=Release .. -make -j`nproc` && sudo make install - -# uWebSockets -cd ~/repos -git clone --branch v0.14.8 https://github.com/uNetworking/uWebSockets.git -cd uWebSockets -make -sudo make install -# /usr/lib64 is not in the path in 20.04, manually install -sudo cp libuWS.so /usr/local/lib/ -sudo ldconfig - -# cascore-data (From KERN-7 PPA) -sudo add-apt-repository -y ppa:kernsuite/kern-7 -sudo apt update -sudo apt install -y casacore-data - -# carta-casacore (To be replaced by PPA package soon) -cd ~/repos -git clone https://github.com/CARTAvis/carta-casacore.git -cd carta-casacore -git submodule init && git submodule update -cd casa6 -git submodule init && git submodule update -cd ../ -mkdir -p build && cd build -cmake -DUSE_FFTW3=ON -DUSE_HDF5=ON -DUSE_THREADS=ON -DUSE_OPENMP=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DBUILD_PYTHON=OFF -DUseCcache=1 -DHAS_CXX11=1 -DDATA_DIR=/usr/share/casacore/data .. -# Use as many cores as you can here -make -j`nproc` -sudo make install -``` - -# Additional directories and permissions -```shell script -# log directory owned by carta -sudo mkdir -p /var/log/carta -sudo chown carta: /var/log/carta - -# config directory owned by carta -sudo mkdir -p /etc/carta -sudo chown carta: /etc/carta - -# edit sudoers file to allow passwordless sudo execution of -# /home/carta/bin/carta_kill_script.sh and /home/carta/bin/carta_backend -# by the carta user -sudo visudo -f /etc/sudoers.d/carta_server -``` - -## CARTA Backend install -Assuming this runs as user `carta` - -```shell script -cd ~ -mkidr -p repos && cd repos -git clone --branch v1.4.0-beta.1 https://github.com/CARTAvis/carta-backend.git -cd carta-backend -git submodule init && git submodule update -mkdir -p build && cd build -cmake -DCMAKE_BUILD_TYPE=Release -DEnableAvx=On ../ -make -j`nproc` -mkdir -p ~/bin -cp carta_backend ~/bin/ -``` - -## nginx server config -This bit should be entirely adapted to fit your server configuration. The relevant part of the config is for forwarding `/` to port 8000. -```nginx -location / { - proxy_set_header X-Forwarded-For $remote_addr; - proxy_pass http://localhost:8000/; - proxy_http_version 1.1; - proxy_set_header Upgrade $http_upgrade; - proxy_set_header Connection 'upgrade'; - proxy_set_header Host $host; - proxy_cache_bypass $http_upgrade; - } - -``` - -## CARTA Server install -Assuming this runs as user `carta` -```shell script -cd ~ -curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash -source .bashrc -nvm install --lts -nvm install-latest-npm - -# Install carta-controller (includes frontend config) -npm install -g carta-controller -cp ${NVM_BIN}/../lib/node_modules/carta-controller/scripts/carta_kill_script.sh - -# ensure bin folder is added to path -source ~/.profile - -# Generate JWT keys and edit config -cd /etc/carta -openssl genrsa -out carta_private.pem 4096 -openssl rsa -in carta_private.pem -outform PEM -pubout -out carta_public.pem -nano config.json - -# Install PM2 node service -npm install -g pm2 -pm2 start carta-controller -``` \ No newline at end of file diff --git a/scripts/generate_schema_html.sh b/scripts/generate_schema_html.sh deleted file mode 100755 index 13de316..0000000 --- a/scripts/generate_schema_html.sh +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/bash -cd "${0%/*}" -# Requires python package json-schema-for-humans: https://github.com/coveooss/json-schema-for-humans -# Install using "pip install json-schema-for-humans" -generate-schema-doc --minify --no-link-to-reused-ref --expand-buttons ../config/config_schema.json ../docs/config_schema.html