📦️(project) improve dependencies management

Ralph can be used as a CLI, as a server, or as a library. To handle this
complexity, we decided to split dependencies to core and extra
requirements:

- the core package only contains dependencies to use Ralph as a library
  (mostly to use models in a python project),
- the `cli` extra installs Click-related dependencies,
- the `backends` extra installs all supported backends; it might also
  correspond to a library usage,
- the `lrs` extra installs FastAPI-related dependencies for the LRS
  server implementation.

A full installation command would look like:

```
pip install ralph-malph[backends,cli,lrs]
```

While a library installation could be:

```
pip install ralph-malph
```

or

```
pip install ralph-malph[backends]
```

.circleci/config.yml

@@ -137,7 +137,7 @@ jobs:
             - v1-dependencies-{{ .Revision }}
       - run:
           name: Install development dependencies
-          command: pip install --user .[dev]
+          command: pip install --user .[backend-es,backend-ldp,backend-mongo,backend-swift,backend-ws,cli,dev,lrs]
       - save_cache:
           paths:
             - ~/.local
@@ -205,6 +205,32 @@ jobs:
               -timeout 60s \
               ~/.local/bin/pytest
 
+  test-library:
+    docker:
+      - image: cimg/python:3.9
+        auth:
+          username: $DOCKER_HUB_USER
+          password: $DOCKER_HUB_PASSWORD
+    working_directory: ~/fun
+    steps:
+      - checkout
+      - run:
+          name: Install development dependencies
+          command: pip install --user .
+
+      - run:
+          name: Test library usage
+          command: |
+            for module in $(\
+              find src/ralph/models/**/*.py | \
+              sed "s|^src/\(.*\)\.py|\1|g" | \
+              tr "/" "." | \
+              sed "s/\.__init__//g" | \
+              sort \
+            ); do
+              python -c "from ${module} import *"
+            done
+
   # ---- Tray jobs (k8s) ----
   tray:
     machine:
@@ -489,6 +515,10 @@ workflows:
           filters:
             tags:
               only: /.*/
+      - test-library:
+          filters:
+            tags:
+              only: /.*/
 
       # Docs
       #
@@ -516,6 +546,7 @@ workflows:
             - build-docs
             - lint
             - test
+            - test-library
           filters:
             tags:
               only: /.*/

CHANGELOG.md

@@ -26,15 +26,12 @@ and this project adheres to
 ### Changed
 
 - Migrate to `python-legacy` handler for `mkdocstrings` package
-- Upgrade `bclick` to `3.2.2`
 - Upgrade `click` to `8.1.3`
 - Upgrade `elasticsearch` to `8.3.3`
 - Upgrade `fastapi` to `0.79.1`
-- Upgrade `Jinja2` to `3.1.2`
 - Upgrade `ovh` to `1.0.0`
 - Upgrade `pydantic` to `1.9.2`
 - Upgrade `pymongo` to `4.2.0`
-- Upgrade `pyparsing` to `3.0.9`
 - Upgrade `python-keystoneclient` to `5.0.0`
 - Upgrade `python-swiftclient` to `4.0.1`
 - Upgrade `requests` to `2.28.1`
@@ -44,6 +41,7 @@ and this project adheres to
 - Make backends yield results instead of writing to standard streams (BC)
 - Use pydantic settings management instead of global variables in defaults.py
 - Rename backend and parser parameter environment variables (BC)
+- Make project dependencies management more modular for library usage
 
 ### Removed
 

Dockerfile

@@ -26,7 +26,7 @@ RUN apt-get update && \
         python-dev && \
     rm -rf /var/lib/apt/lists/*
 
-RUN python setup.py install
+RUN pip install .[backend-es,backend-ldp,backend-mongo,backend-swift,backend-ws,cli,lrs]
 
 
 # -- Core --

docs/index.md

@@ -39,14 +39,26 @@ Ralph package can be installed from PyPI using the `pip` tool:
 $ python3.9 -m venv venv
 $ source venv/bin/activate
 
-# Install the package (in a virtualenv)
-(venv) $ pip install ralph-malph
+# Install the full package (in a virtualenv)
+(venv) $ pip install ralph-malph[backend-es,backend-ldp,backend-mongo,backend-swift,backend-ws,cli,lrs]
+
+# Install only the core package with the Elasticsearch backend and the LRS (in a virtualenv)
+(venv) $ pip install ralph-malph[backend-es,lrs]
 
 # Test the ralph command (it should be in your PATH)
 (venv) $ ralph --help
 ```
 
-Alternatively, Docker users can pull the latest ralph image and start playing
+If you only need to integrate `ralph` models in your project, you don't need to
+install the `backends`, `cli` or `lrs` extra dependencies, the core package is
+what you need:
+
+```sh
+# Install the core library (in a virtualenv)
+(venv) $ pip install ralph-malph
+```
+
+Alternatively, Docker users can pull the latest `ralph` image and start playing
 with it:
 
 ```sh

setup.cfg

@@ -23,39 +23,33 @@ classifiers =
 [options]
 include_package_data = True
 install_requires =
-    bcrypt==3.2.2
-    click==8.1.3
-    click-option-group==0.5.3
-    elasticsearch==8.3.3
-    fastapi==0.79.1
-    gunicorn==20.1.0
-    ; We temporary pin `h11` to avoid pip downloading the latest version to solve a
-    ; dependency conflict caused by `httpx` which requires httpcore>=0.15.0,<0.16.0 and
-    ; `httpcore` depends on h11>=0.11,<0.13.
-    ; See: https://github.com/encode/httpx/issues/2244
-    h11==0.12.0
-    httpx==0.23.0
-    Jinja2==3.1.2
+    ; By default, we only consider core dependencies required to use Ralph as a
+    ; library (mostly models).
     langcodes==3.3.0
-    ovh==1.0.0
     pydantic[dotenv,email]==1.9.2
-    pymongo[srv]==4.2.0
-    pyparsing==3.0.9
-    python-keystoneclient==5.0.0
-    python-swiftclient==4.0.1
-    requests==2.28.1
     rfc3987==1.3.8
-    sentry_sdk==1.9.5
-    uvicorn[standard]==0.18.2
-    watchgod==0.8.2
-    websockets==10.3
 package_dir =
     =src
 packages = find:
 zip_safe = True
 python_requires = >= 3.9
 
 [options.extras_require]
+backend-es =
+    elasticsearch==8.3.3
+backend-ldp =
+    ovh==1.0.0
+    requests==2.28.1
+backend-mongo =
+    pymongo[srv]==4.2.0
+backend-swift =
+    python-keystoneclient==5.0.0
+    python-swiftclient==4.0.1
+backend-ws =
+    websockets==10.3
+cli =
+    click==8.1.3
+    click-option-group==0.5.3
 dev =
     bandit==1.7.4
     black==22.6.0
@@ -79,6 +73,17 @@ dev =
     pytest-cov==3.0.0
 ci =
     twine==4.0.1
+lrs =
+    bcrypt==3.2.2
+    fastapi==0.79.1
+    ; We temporary pin `h11` to avoid pip downloading the latest version to solve a
+    ; dependency conflict caused by `httpx` which requires httpcore>=0.15.0,<0.16.0 and
+    ; `httpcore` depends on h11>=0.11,<0.13.
+    ; See: https://github.com/encode/httpx/issues/2244
+    h11==0.12.0
+    httpx==0.23.0
+    sentry_sdk==1.9.5
+    uvicorn[standard]==0.18.2
 
 [options.packages.find]
 where = src

src/ralph/conf.py

@@ -4,10 +4,18 @@
 from pathlib import Path
 from typing import Literal, Union
 
-from click import get_app_dir
+try:
+    from click import get_app_dir
+except ImportError:
+    # If we use Ralph as a library and Click is not installed, we consider the
+    # application directory to be the current directory. For non-CLI usage, it
+    # has no consequences.
+    from unittest.mock import Mock
+
+    get_app_dir = Mock(return_value=".")
 from pydantic import AnyUrl, BaseModel, BaseSettings
 
-from ralph.utils import import_string
+from .utils import import_string
 
 MODEL_PATH_SEPARATOR = "__"