Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. Weโ€™ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jupyter server #230

Merged
merged 29 commits into from
Aug 31, 2021
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
e017edc
jupyter server extension application:
oliver-sanders May 11, 2021
2dffb0d
cli: add entry points for server commands
oliver-sanders May 17, 2021
449635b
patch authorisation interface
oliver-sanders May 18, 2021
e0631d0
default url to remove unnecessary redirect
oliver-sanders May 19, 2021
a2bcfe8
uis: add shudown hook
oliver-sanders May 21, 2021
7fa2f97
userprofile: add an owner field
oliver-sanders May 21, 2021
b2fb384
cylc.uiserver.main -> cylc.uiserver.app
oliver-sanders May 21, 2021
d3bfa88
logging: convert bespoke logging to jupyter_server logging
oliver-sanders May 21, 2021
da35266
gui app: fix userprofile
oliver-sanders May 21, 2021
fb88a33
app: tidy application
oliver-sanders May 21, 2021
e9f3ac9
project setup
oliver-sanders Jul 26, 2021
49b096a
fix new log entries
oliver-sanders Jul 26, 2021
a521675
fixed tests
oliver-sanders Jul 26, 2021
b73c6c3
authorisation: reinstate decorator
oliver-sanders Jul 28, 2021
5166b0b
mypy: upgrade to 0.900
oliver-sanders Jul 29, 2021
eb30dc0
flake8: add plugins
oliver-sanders Jul 29, 2021
75160b2
pytest: enable doctests
oliver-sanders Jul 30, 2021
e19717a
update README
oliver-sanders Jul 30, 2021
c1916f7
changelog
oliver-sanders Jul 30, 2021
edf3fa1
remove dev cylc_gui namespace
oliver-sanders Jul 30, 2021
1ec12e5
licencing
oliver-sanders Aug 2, 2021
cd5e9c4
setup: bump deps
oliver-sanders Aug 12, 2021
aa6a095
setup: get build working
oliver-sanders Aug 13, 2021
3b0b51b
uis: restore CYLC_SITE_CONF_PATH functionality
oliver-sanders Aug 13, 2021
8b0c544
tidy
oliver-sanders Aug 17, 2021
72e05c5
jupyter_server: upgrade to >=1.10.2
oliver-sanders Aug 24, 2021
5da6044
authenticate the static handler
oliver-sanders Aug 24, 2021
4c63763
add version endpoint
oliver-sanders Aug 24, 2021
acc20b1
feedback
oliver-sanders Aug 24, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 2 additions & 4 deletions .coveragerc
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,7 @@ omit =
venv/*
cylc/uiserver/tests/*
cylc/uiserver/websockets/*
cylc/uiserver/config_defaults.py
setup.py
cylc/uiserver/jupyter*_config.py
parallel = True
plugins=
include=
Expand All @@ -53,8 +52,7 @@ omit =
venv/*
cylc/uiserver/tests/*
cylc/uiserver/websockets/*
cylc/uiserver/config_defaults.py
setup.py
cylc/uiserver/jupyter*_config.py
precision=2
show_missing=False
skip_covered=False
Expand Down
11 changes: 11 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,17 @@ creating a new release entry be sure to copy & paste the span tag with the
`actions:bind` attribute, which is used by a regex to find the text to be
updated. Only the first match gets replaced, so it's fine to leave the old
ones in. -->

-------------------------------------------------------------------------------
## __cylc-uiserver-0.6.0 (<span actions:bind='release-date'>Released 2021-??-??</span>)__

Multi-user functionality implented.
oliver-sanders marked this conversation as resolved.
Show resolved Hide resolved

### Enhancements

[#230](https://github.com/cylc/cylc-uiserver/pull/230) -
Convert the UI Server to a jupyter_server extension

-------------------------------------------------------------------------------
## __cylc-uiserver-0.5.0 (<span actions:bind='release-date'>Released 2021-07-28</span>)__

Expand Down
154 changes: 119 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,22 +1,21 @@
[![Anaconda-Server Badge](https://anaconda.org/conda-forge/cylc-uiserver/badges/version.svg)](https://anaconda.org/conda-forge/cylc-uiserver)
[![Anaconda-Server Badge](https://anaconda.org/conda-forge/cylc-uiserver/badges/downloads.svg)](https://anaconda.org/conda-forge/cylc-uiserver)
[![PyPI](https://img.shields.io/pypi/v/cylc-uiserver.svg?color=yellow)](https://pypi.org/project/cylc-uiserver/)
[![forum](https://img.shields.io/discourse/https/cylc.discourse.group/posts.svg)](https://cylc.discourse.group/)
[![Test](https://github.com/cylc/cylc-uiserver/actions/workflows/test.yml/badge.svg?branch=master&event=push)](https://github.com/cylc/cylc-uiserver/actions/workflows/test.yml)
[![codecov](https://codecov.io/gh/cylc/cylc-uiserver/branch/master/graph/badge.svg)](https://codecov.io/gh/cylc/cylc-uiserver)


# Cylc UI Server

This project contains the Cylc UI Server. A JupyterHub-compatible application,
This project contains the Cylc UI Server which provides the Cylc GUI
used to serve the Cylc UI, and to communicate with running Cylc Schedulers.

[Cylc Website](https://cylc.org/) |
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://cylc.org/ currently forwards to https://niwa.co.nz/
I'm guessing this will be fixed by production time but thought I'd flag it now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ping @hjoliver

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://cylc.org/ currently forwards to https://niwa.co.nz/

Oh, that's changed recently it used to redirect to GH pages, yikes!

Response.text: Body has already been consumed.

Hmmm, I've seen this error before, I think my rebase of the cylc-ui branch might not have been fully successful.

[Contributing](CONTRIBUTING.md) |
[Developing](DEVELOPING.md)

## Contents
[Developing](#Developing) |
[Forum](https://cylc.discourse.group/)

- [Installation](#installation)
- [Introduction](#introduction)
- [Copyright](#copyright-and-terms-of-use)

## Introduction

Expand All @@ -25,24 +24,37 @@ interface.

This repository provides the following components of the Cylc system.

* The UI
* **The UI**

This is the Cylc web app that provides control and monitoring functionalities
for Cylc workflows.

* The UI Server
> The UI is developed in a separate repository https://github.com/cylc/cylc-ui

* **The UI Server**

This is a web server which serves the Cylc web UI. It connects to running
workflows and workflow databases to provide the information the UI displays.
It is a [Jupyter Server](https://github.com/jupyter-server/jupyter_server).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A Jupyter Server application, or a Jupyter web Application? I'm reading their docs, from what I understood, Jupyter Server is the utility code for creating applications or extensions. Applications being complete web applications that use the Jupyter Server, and Extensions only extend the REST API.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Huh, so looks like everything is an extension, but there's a special ExtensionApp that makes an extension that behaves like the old applications (like notebook, and current UI Server I think?). So everything is an extension? ๐Ÿ˜„

Copy link
Member Author

@oliver-sanders oliver-sanders Aug 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there's some similar but different terminology here which makes things confusing.

The Cylc UI Server (as of this PR) is a "Jupyter Server extension applicaiton".

  • The "extension" part is because it is extending Jupyter Server.
  • The application part is because it is a configurable traitlets application.

Extension applications can:

Jupyter server also offers non-configurable extensions (ones which aren't applications), these are cruder.

This page helps show the difference - https://jupyter-server.readthedocs.io/en/latest/developers/extensions.html

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @oliver-sanders !


* The Hub
* **The Hub**

In multi-user setups this launches UI Servers, provides a proxy for running
server and handles authentication. It is a
[JupyterHub](https://github.com/jupyterhub/jupyterhub) server.

This launches UI Servers, provides a proxy for running server and handles
authentication. It is a JupyterHub server.

## Installation

For production:
[![Anaconda-Server Badge](https://anaconda.org/conda-forge/cylc-uiserver/badges/version.svg)](https://anaconda.org/conda-forge/cylc-uiserver)
[![PyPI](https://img.shields.io/pypi/v/cylc-uiserver.svg?color=yellow)](https://pypi.org/project/cylc-uiserver/)
![Conda-Platforms](https://img.shields.io/conda/pn/conda-forge/cylc-uiserver)

For more information on the Cylc components and full-stack Cylc installations
see the
[Cylc documentation](https://cylc.github.io/cylc-doc/latest/html/installation.html).

### For Single-User Setups

```console
# via conda (preferred)
Expand All @@ -52,13 +64,56 @@ $ conda install cylc-uiserver
$ pip install cylc-uiserver
```

## Running
### For Multi-User Setups

```console
$ cylc hub
# via conda (preferred)
$ conda install cylc-uiserver-hub

# via pip (consult jupyterhub documentation)
$ pip install cylc-uiserver[hub]
```


## Running

The Cylc UIServer is a
[Jupyter Server](https://github.com/jupyter-server/jupyter_server)
extension (like [JupyterLab](https://github.com/jupyterlab/jupyterlab)).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I guess it's an extension then, I thought it was an application ๐Ÿ‘

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If splitting hairs it is technically an "extension application"!

From the users perspective the difference between an "extension" and an "extension application" is invisible.

Copy link
Member

@kinow kinow Aug 16, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Brilliant, thanks @oliver-sanders ! It was more so I knew the correct term in case I needed to report an issue upstream, or talk to someone else about Jupyter Server. But extension application is quite a mouthful, I'll probably just use jupyter server, or jupyter server app, or whichever my ๐Ÿง  picks first. Leaving unresolved in case others are interested in this answer too.


### For Single-User Setups

Run as a standalone server using a URL token for authentication:

```bash
# launch the Cylc GUI and open a browser tab
cylc gui

# alternatively the same app can be opened with the jupyter command
$ jupyter cylc
```

The default URL is [http://localhost:8000](http://localhost:8000).
> By default authentication is provided by the URL token, alternatively a
> password can be configured (see Jupyter Server docs).
>
> There is no per-user authorisation so anyone who has the URL token has full
> access to the server.

### For Multi-User Setups

Run a central [JupyterHub](https://github.com/jupyterhub/jupyterhub) server
under a user account with the privileges required to spawn `cylc` processes as
other users.

```bash
# launch the Cylc Hub
# (the default URL is http://localhost:8000)
cylc hub
```

> Users then authenticate with the hub which launches and manages their UI
> Server.


## Configuring

Expand All @@ -70,20 +125,20 @@ The Cylc Hub will load the following files in order:

These are the Cylc defaults which are hardcoded within the repository.

(`<python-installation>/cylc/uiserver/config_defaults.py`)
(`<python-installation>/cylc/uiserver/jupyter_config.py`)

2) Site Config

This file configures the Hub/UIS for all users. The default path can be
changed by the ``CYLC_SITE_CONF_PATH`` environment variable.

(`/etc/cylc/hub/config.py`)
(`/etc/cylc/hub/jupyter_config.py`)

3) User Config

This file

(`~/.cylc/hub/config.py`)
(`~/.cylc/hub/jupyter_config.py`)

Alternatively a single config file can be provided on the command line.

Expand All @@ -96,30 +151,57 @@ $ cylc hub --config
> If specifying a config file on the command line the system config containing
> the hardcoded Cylc default will **not** be loaded.

> **Note:**
>
> The hub can also be run using the ``jupyterhub`` command, however, you
> must source the configuration files manually on the command line.

See the Jupyterhub documentation for details on configuration options.

### UI Server

The UI Server is (currently) also configured from the same configuration file(s)
as the hub using the
`UIServer` namespace.
The UI Server is also configured from the same configuration file(s) as the hub
using the `UIServer` namespace.

Currently the UI Server accepts these configurations:

* `c.UIServer.ui_build_dir`
* `c.UIServer.ui_version`
* `c.UIServer.logging_config`
* `c.UIServer.scan_iterval`
* `c.CylcUIServer.ui_build_dir`
* `c.CylcUIServer.ui_version`
* `c.CylcUIServer.scan_iterval`

See the `cylc.uiserver.app.UIServer` file for details.

The Cylc UI Server is a
[Jupyter Server](https://github.com/jupyter-server/jupyter_server) extension.
Jupyter Server can run multiple extensions. To control the extensions that
are run use the `ServerApp.jpserver_extensions` configuration, see the
[Jupyter Server configuration documentation](https://jupyter-server.readthedocs.io/en/latest/other/full-config.html#other-full-config).

### UI

The UI can be configured via the "Settings" option in the Dashboard.

Currently these configurations are stored in the web browser so won't travel
around a network and might not persist.

See the `cylc.uiserver.main.UIServer` file for details.

## Developing

1) Read the [Contributing](CONTRIBUTING.md) page.
[![Contributors](https://img.shields.io/github/contributors/cylc/cylc-uiserver.svg?color=9cf)](https://github.com/cylc/cylc-uiserver/graphs/contributors)
[![Commit activity](https://img.shields.io/github/commit-activity/m/cylc/cylc-uiserver.svg?color=yellowgreen)](https://github.com/cylc/cylc-uiserver/commits/master)
[![Last commit](https://img.shields.io/github/last-commit/cylc/cylc-uiserver.svg?color=ff69b4)](https://github.com/cylc/cylc-uiserver/commits/master)

Contributions welcome:

2) Fork and clone this repo.
* Read the [contributing](CONTRIBUTING.md) page.
* Development setup instructions are in the
[developer docs](https://cylc.github.io/cylc-admin/#cylc-8-developer-docs).
* Involved change proposals can be found in the
[admin pages](https://cylc.github.io/cylc-admin/#change-proposals).
* Touch base in the
[developers chat](https://matrix.to/#/#cylc-general:matrix.org).

3) Install from source into your Python environment:
1) Install from source into your Python environment:

```console
$ pip install -e .[all]
Expand All @@ -130,12 +212,14 @@ See the `cylc.uiserver.main.UIServer` file for details.
> If you want to run with a development copy of Cylc Flow you must install
> it first else `pip` will download the latest version from PyPi.

4) For UI development set the following configuration to use your UI build
2) For UI development follow the developer instructions for the
[cylc-ui](https://github.com/cylc/cylc-ui) project, then
set the following configuration so Cylc uses your UI build
(rather than the default bundled UI build):

```python
# ~/.cylc/hub/config.py
c.UIServer.ui_build_dir = '~/cylc-ui/dist' # path to build
# ~/.cylc/hub/jupyter_config.py
c.CylcUIServer.ui_build_dir = '~/cylc-ui/dist' # path to build
```

## Copyright and Terms of Use
Expand All @@ -148,7 +232,7 @@ either version 3 of the License, or (at your option) any later version.

Cylc is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE. See the GNU General Public License for more details.
PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with
Cylc. If not, see [GNU licenses](http://www.gnu.org/licenses/).
Cylc. If not, see [GNU licenses](http://www.gnu.org/licenses/).
19 changes: 7 additions & 12 deletions cylc/uiserver/scripts/uis.py โ†’ conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,18 +12,13 @@
#
# You should have received a copy of the GNU General Public License
# along with this program. If not, see <http://www.gnu.org/licenses/>.
"""cylc uis

Launch a Cylc UI Server.
pytest_plugins = [
'jupyter_server.pytest_plugin'
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This pulls in the jupyter_server testing framework.

]

This is an internal command invoked by the Cylc Hub via the configured
Jupyterhub spawner.
"""

from cylc.uiserver.main import main as uis_main

INTERNAL = True


def main(*_):
uis_main()
def pytest_ignore_collect(path):
# --doctest-modules seems to ignore the value if configured in pyproject
if 'jupyter_config.py' in str(path):
return True
15 changes: 15 additions & 0 deletions cylc/uiserver/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -14,3 +14,18 @@
# along with this program. If not, see <http://www.gnu.org/licenses/>.

__version__ = "0.5.0"

from cylc.uiserver.app import CylcUIServer


def _jupyter_server_extension_points():
"""
Returns a list of dictionaries with metadata describing
where to find the `_load_jupyter_server_extension` function.
"""
return [
{
"module": "cylc.uiserver.app",
'app': CylcUIServer
}
]
Loading