docs: improve documentation (DSP-1693) (#62)

* fix error in documentation

* fix errors, clarify things, fix styling issues

* remove doc from disallowed prefixes in the check-pr-title action

* Update documentation

* Update documentation

* Update documentation

* Update docs/dsp-tools-create.md

Co-authored-by: Balduin Landolt <33053745+BalduinLandolt@users.noreply.github.com>

* add suggestions of code review

* split index into several subpages

Co-authored-by: Balduin Landolt <33053745+BalduinLandolt@users.noreply.github.com>

.github/workflows/ckeck-pr-title.yml

@@ -13,8 +13,8 @@ jobs:
       - uses: deepakputhraya/action-pr-title@master
         with:
           regex: '([a-z])+(\(([a-z\-_ ])+\))?!?: [a-z]([a-zA-Z-\.\d \(\)\[\]#_])+$' # Regex the title should match.
-          allowed_prefixes: "fix,refactor,feat,docs,chore,style,test" # title should start with the given prefix
-          disallowed_prefixes: "feature,hotfix,doc" # title should not start with the given prefix
+          allowed_prefixes: 'fix,refactor,feat,docs,chore,style,test' # title should start with the given prefix
+          disallowed_prefixes: 'feature,hotfix' # title should not start with the given prefix
           prefix_case_sensitive: true # title prefix are case insensitive
           min_length: 7 # Min length of the title
           max_length: 120 # Max length of the title

docs/dsp-tools-excel.md

@@ -1,13 +1,17 @@
-[![PyPI version](https://badge.fury.io/py/knora.svg)](https://badge.fury.io/py/knora)
+[![PyPI version](https://badge.fury.io/py/dsp-tools.svg)](https://badge.fury.io/py/dsp-tools)
 
-# DSP tools to use Excel-files for data modelling and data import
-Dsp-tools is able to directly read and process excel files and output the appropriate JSON and/or XML-files for
-importing data to the dsp-repository.
+# Excel files for data modelling and data import
+dsp-tools is able to process Excel files and output the appropriate JSON or XML file. The JSON/XML file can then
+be used to create the ontology on the DSP server or import data to the DSP repository. dsp-tools can also be used to
+create a list from an Excel file.
 
-## Flat and hierarchical lists
-Lists or "controlled vocabularies" are sets of fixed terms that are used to characterize something. Hierarchical lists
-correspond to classifications or taxonomies.
-
-The format of the excel is described  [here](./dsp-tools-create.md#lists-from-excel).
+## Create the data model JSON from an Excel file
+[not yet implemented]
 
+## Create a DSP-conform XML file from an Excel file
+[not yet implemented]
 
+## Create flat or hierarchical lists from an Excel file
+Lists or controlled vocabularies are sets of fixed terms that are used to characterize objects. Hierarchical lists
+correspond to classifications or taxonomies. With dsp-tools a list can be created from an Excel file. The expected
+format of the Excel file is described [here](./dsp-tools-create.md#lists-from-excel).

docs/dsp-tools-information-for-developers.md

@@ -0,0 +1,92 @@
+[![PyPI version](https://badge.fury.io/py/dsp-tools.svg)](https://badge.fury.io/py/dsp-tools)
+
+There is a `Makefile` for all the following tasks (and more). Type `make` to print the available targets.
+
+## Install from source
+
+To install from source run:
+```bash
+python3 setup.py install
+```
+
+## Install the requirements
+
+To install the requirements run:
+
+```bash
+pip3 install -r requirements.txt
+```
+
+To generate a requirements file (usually requirements.txt), that you commit with your project, run:
+
+```bash
+pip3 freeze > requirements.txt
+```
+
+## Testing
+Please note that testing requires launching the complete DSP API stack which is based on docker images. Therefore, we
+recommend installing the [docker desktop client](https://www.docker.com/products).
+
+To run the complete test suite:
+
+```bash
+make test
+```
+
+## Publishing
+
+Publishing is automated with GitHub Actions and should _not_ be done manually. Please follow the
+[Pull Request Guidelines](https://docs.dasch.swiss/developers/dsp/contribution/#pull-request-guidelines). If done
+correctly, when merging a pull request into `main`, the `release-please` action will create or update a pull request for
+a release. This pull request will follow semantic versioning and update the change log. Once all desired features are
+merged, the release can be executed by merging this release pull request into `main`. This will trigger actions that
+create a release on GitHub, on PyPI and the docs.
+
+Please ensure you have only one pull request per feature.
+
+## Publishing manually
+
+Publishing is automated with GitHub Actions and should _not_ be done manually. If you still need to do it, follow the
+steps below.
+
+Generate the distribution package. Make sure you have the latest versions of `setuptools` and `wheel` installed:
+
+```bash
+python3 -m pip install --upgrade pip setuptools wheel
+python3 setup.py sdist bdist_wheel
+```
+
+You can install the package locally from the dist:
+
+```bash
+python3 -m pip ./dist/some_name.whl
+```
+
+Upload package works also with `make`:
+
+```bash
+make dist
+make upload
+```
+
+For local development:
+
+```bash
+python3 setup.py develop
+```
+
+## Contributing to the documentation
+
+The documentation is a collection of [markdown](https://en.wikipedia.org/wiki/Markdown) files in the `docs` folder.  
+After updates of the files, build and check the result with the following commands:
+
+```bash
+make build-docs
+make serve-docs 
+```
+
+To update the changes to the official documentation pages run:
+
+```bash
+make publish-docs
+```

docs/dsp-tools-usage.md

@@ -0,0 +1,108 @@
+[![PyPI version](https://badge.fury.io/py/dsp-tools.svg)](https://badge.fury.io/py/dsp-tools)
+
+## Installation
+
+To install the latest version run:
+
+```
+pip3 install dsp-tools
+```
+
+To update to the latest version run:
+
+```
+pip3 install --upgrade dsp-tools
+```
+
+## Create a data model on a DSP server
+
+```bash
+dsp-tools create [options] data_model_definition.json
+```
+
+The following options are available:
+
+- `-s` | `--server` _server_: URL of the DSP server (default: localhost:3333)
+- `-u` | `--user` _username_: username used for authentication with the DSP API (default: root@example.com)
+- `-p` | `--password` _password_: password used for authentication with the DSP API (default: test)
+- `-V` | `--validate`: If set, only the validation of the JSON file is performed.
+- `-l` | `--lists`: If set, only the lists are created using a [simplified schema](./dsp-tools-create.md#lists). Please note
+  that in this case the project must already exist.
+- `-v` | `--verbose`: If set, some information about the progress is printed to the console.
+
+The command is used to read the definition of a data model (provided in a JSON file) and create it on the
+DSP server. The following example shows how to load the ontology defined in `data_model_definition.json` onto the DSP
+server `https://api.dsl.server.org` provided with the `-s` option. The username `root@example.com` and the password
+  `test` are used.
+
+```bash
+dsp-tools create -s https://api.dsl.server.org -u root@example.com -p test data_model_definition.json
+```
+
+The description of the expected JSON format can be found [here](./dsp-tools-create.md). 
+
+## Get a data model from a DSP server
+
+```bash
+dsp-tools get [options] output_file.json
+```
+
+The following options are available:
+
+- `-s` | `--server` _server_: URL of the DSP server (default: localhost:3333)
+- `-u` | `--user` _username_: username used for authentication with the DSP API (default: root@example.com)
+- `-p` | `--password` _password_: password used for authentication with the DSP API (default: test)
+- `-P` | `--project` _shortcode_ | _shortname_ | _iri_: shortcode, shortname or
+  [IRI](https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier) of the project
+- `-v` | `--verbose`: If set, some information about the progress is printed to the console.
+
+The command is used to get the definition of a data model from a DSP server and write it into a JSON file. This JSON file
+could then be used to upload the data model to another DSP server. The following example shows how to get the data model
+from a DSP server `https://api.dsl.server.org` provided with the `-s` option. The username `root@example.com` and the
+password `test` are used. The data model is saved into the output file `output_file.json`.
+
+```bash
+dsp-tools get -s https://api.dsl.server.org -u root@example.com -p test output_file.json
+```
+
+## Upload data to a DSP server
+
+```bash
+dsp-tools xmlupload [options] xml_data_file.xml
+```
+
+The following options are available:
+
+- `-s` | `--server` _server_: URL of the DSP server (default: localhost:3333)
+- `-u` | `--user` _username_: username used for authentication with the DSP API (default: root@example.com)
+- `-p` | `--password` _password_: password used for authentication with the DSP API (default: test)
+- `-i` | `--imgdir` _dirpath_: path to the directory where the bitstream objects are stored (default: .)
+- `-S` | `--sipi` _SIPIserver_: URL of the SIPI IIIF server (default: http://0.0.0.0:1024)
+
+The command is used to upload data defined in an XML file onto a DSP server. The following example shows how to upload
+data from an XML file `xml_data_file.xml` onto the DSP server `https://api.dsl.server.org` provided with the `-s` option.
+The username `root@example.com` and the password `test` are used. The interface for the SIPI IIIF server is provided
+with the `-S` option (`https://iiif.dsl.server.org`).
+
+```bash
+dsp-tools xmlupload -s https://api.dsl.server.org -u root@example.com -p test -S https://iiif.dsl.server.org xml_data_file.xml
+```
+
+The description of the expected XML format can be found [here](./dsp-tools-xmlupload.md). 
+
+## Convert an Excel file into a JSON file that is compatible with dsp-tools
+
+```bash
+dsp-tools excel [options] excel_file.xlsx output_file.json
+```
+
+The following options are available:
+
+- `-S` | `--sheet` _sheetname_: name of the Excel worksheet to use (default: Tabelle1)
+- `-s` | `--shortcode` _shortcode_: shortcode of the project (required)
+- `-l` | `--listname` _listname_: name to be used for the list and the list definition file (required)
+- `-L` | `--label` _label_: label to be used for the list (required)
+- `-x` | `--lang` _lang_: language used for the list labels and commentaries (default: en)
+- `-v` | `--verbose`: If set, some information about the progress is printed to the console.
+
+The description of the expected Excel format can be found [here](./dsp-tools-create.md#lists-from-excel).