Skip to content
Permalink
Browse files
docs: improve documentation (DSP-1693) (#62)
* fix error in documentation

* fix errors, clarify things, fix styling issues

* remove doc from disallowed prefixes in the check-pr-title action

* Update documentation

* Update documentation

* Update documentation

* Update docs/dsp-tools-create.md

Co-authored-by: Balduin Landolt <33053745+BalduinLandolt@users.noreply.github.com>

* add suggestions of code review

* split index into several subpages

Co-authored-by: Balduin Landolt <33053745+BalduinLandolt@users.noreply.github.com>
  • Loading branch information
irinaschubert and BalduinLandolt committed Jun 7, 2021
1 parent 69772f4 commit 591b5ad46f8c4b6aecd023ff1448953a7c6c7d45
Showing with 1,094 additions and 949 deletions.
  1. +2 −2 .github/workflows/ckeck-pr-title.yml
  2. +396 −356 docs/dsp-tools-create.md
  3. +13 −9 docs/dsp-tools-excel.md
  4. +92 −0 docs/dsp-tools-information-for-developers.md
  5. +108 −0 docs/dsp-tools-usage.md
  6. +458 −372 docs/dsp-tools-xmlupload.md
  7. +21 −208 docs/index.md
  8. +4 −2 mkdocs.yml
@@ -13,8 +13,8 @@ jobs:
- uses: deepakputhraya/action-pr-title@master
with:
regex: '([a-z])+(\(([a-z\-_ ])+\))?!?: [a-z]([a-zA-Z-\.\d \(\)\[\]#_])+$' # Regex the title should match.
allowed_prefixes: "fix,refactor,feat,docs,chore,style,test" # title should start with the given prefix
disallowed_prefixes: "feature,hotfix,doc" # title should not start with the given prefix
allowed_prefixes: 'fix,refactor,feat,docs,chore,style,test' # title should start with the given prefix
disallowed_prefixes: 'feature,hotfix' # title should not start with the given prefix
prefix_case_sensitive: true # title prefix are case insensitive
min_length: 7 # Min length of the title
max_length: 120 # Max length of the title

Large diffs are not rendered by default.

@@ -1,13 +1,17 @@
[![PyPI version](https://badge.fury.io/py/knora.svg)](https://badge.fury.io/py/knora)
[![PyPI version](https://badge.fury.io/py/dsp-tools.svg)](https://badge.fury.io/py/dsp-tools)

# DSP tools to use Excel-files for data modelling and data import
Dsp-tools is able to directly read and process excel files and output the appropriate JSON and/or XML-files for
importing data to the dsp-repository.
# Excel files for data modelling and data import
dsp-tools is able to process Excel files and output the appropriate JSON or XML file. The JSON/XML file can then
be used to create the ontology on the DSP server or import data to the DSP repository. dsp-tools can also be used to
create a list from an Excel file.

## Flat and hierarchical lists
Lists or "controlled vocabularies" are sets of fixed terms that are used to characterize something. Hierarchical lists
correspond to classifications or taxonomies.

The format of the excel is described [here](./dsp-tools-create.md#lists-from-excel).
## Create the data model JSON from an Excel file
[not yet implemented]

## Create a DSP-conform XML file from an Excel file
[not yet implemented]

## Create flat or hierarchical lists from an Excel file
Lists or controlled vocabularies are sets of fixed terms that are used to characterize objects. Hierarchical lists
correspond to classifications or taxonomies. With dsp-tools a list can be created from an Excel file. The expected
format of the Excel file is described [here](./dsp-tools-create.md#lists-from-excel).
@@ -0,0 +1,92 @@
[![PyPI version](https://badge.fury.io/py/dsp-tools.svg)](https://badge.fury.io/py/dsp-tools)

There is a `Makefile` for all the following tasks (and more). Type `make` to print the available targets.

## Install from source

To install from source run:
```bash
python3 setup.py install
```

## Install the requirements

To install the requirements run:

```bash
pip3 install -r requirements.txt
```

To generate a requirements file (usually requirements.txt), that you commit with your project, run:

```bash
pip3 freeze > requirements.txt
```

## Testing
Please note that testing requires launching the complete DSP API stack which is based on docker images. Therefore, we
recommend installing the [docker desktop client](https://www.docker.com/products).

To run the complete test suite:

```bash
make test
```

## Publishing

Publishing is automated with GitHub Actions and should _not_ be done manually. Please follow the
[Pull Request Guidelines](https://docs.dasch.swiss/developers/dsp/contribution/#pull-request-guidelines). If done
correctly, when merging a pull request into `main`, the `release-please` action will create or update a pull request for
a release. This pull request will follow semantic versioning and update the change log. Once all desired features are
merged, the release can be executed by merging this release pull request into `main`. This will trigger actions that
create a release on GitHub, on PyPI and the docs.

Please ensure you have only one pull request per feature.

## Publishing manually

Publishing is automated with GitHub Actions and should _not_ be done manually. If you still need to do it, follow the
steps below.

Generate the distribution package. Make sure you have the latest versions of `setuptools` and `wheel` installed:

```bash
python3 -m pip install --upgrade pip setuptools wheel
python3 setup.py sdist bdist_wheel
```

You can install the package locally from the dist:

```bash
python3 -m pip ./dist/some_name.whl
```

Upload package works also with `make`:

```bash
make dist
make upload
```

For local development:

```bash
python3 setup.py develop
```

## Contributing to the documentation

The documentation is a collection of [markdown](https://en.wikipedia.org/wiki/Markdown) files in the `docs` folder.
After updates of the files, build and check the result with the following commands:

```bash
make build-docs
make serve-docs
```

To update the changes to the official documentation pages run:

```bash
make publish-docs
```
@@ -0,0 +1,108 @@
[![PyPI version](https://badge.fury.io/py/dsp-tools.svg)](https://badge.fury.io/py/dsp-tools)

## Installation

To install the latest version run:

```
pip3 install dsp-tools
```

To update to the latest version run:

```
pip3 install --upgrade dsp-tools
```

## Create a data model on a DSP server

```bash
dsp-tools create [options] data_model_definition.json
```

The following options are available:

- `-s` | `--server` _server_: URL of the DSP server (default: localhost:3333)
- `-u` | `--user` _username_: username used for authentication with the DSP API (default: root@example.com)
- `-p` | `--password` _password_: password used for authentication with the DSP API (default: test)
- `-V` | `--validate`: If set, only the validation of the JSON file is performed.
- `-l` | `--lists`: If set, only the lists are created using a [simplified schema](./dsp-tools-create.md#lists). Please note
that in this case the project must already exist.
- `-v` | `--verbose`: If set, some information about the progress is printed to the console.

The command is used to read the definition of a data model (provided in a JSON file) and create it on the
DSP server. The following example shows how to load the ontology defined in `data_model_definition.json` onto the DSP
server `https://api.dsl.server.org` provided with the `-s` option. The username `root@example.com` and the password
`test` are used.

```bash
dsp-tools create -s https://api.dsl.server.org -u root@example.com -p test data_model_definition.json
```

The description of the expected JSON format can be found [here](./dsp-tools-create.md).

## Get a data model from a DSP server

```bash
dsp-tools get [options] output_file.json
```

The following options are available:

- `-s` | `--server` _server_: URL of the DSP server (default: localhost:3333)
- `-u` | `--user` _username_: username used for authentication with the DSP API (default: root@example.com)
- `-p` | `--password` _password_: password used for authentication with the DSP API (default: test)
- `-P` | `--project` _shortcode_ | _shortname_ | _iri_: shortcode, shortname or
[IRI](https://en.wikipedia.org/wiki/Internationalized_Resource_Identifier) of the project
- `-v` | `--verbose`: If set, some information about the progress is printed to the console.

The command is used to get the definition of a data model from a DSP server and write it into a JSON file. This JSON file
could then be used to upload the data model to another DSP server. The following example shows how to get the data model
from a DSP server `https://api.dsl.server.org` provided with the `-s` option. The username `root@example.com` and the
password `test` are used. The data model is saved into the output file `output_file.json`.

```bash
dsp-tools get -s https://api.dsl.server.org -u root@example.com -p test output_file.json
```

## Upload data to a DSP server

```bash
dsp-tools xmlupload [options] xml_data_file.xml
```

The following options are available:

- `-s` | `--server` _server_: URL of the DSP server (default: localhost:3333)
- `-u` | `--user` _username_: username used for authentication with the DSP API (default: root@example.com)
- `-p` | `--password` _password_: password used for authentication with the DSP API (default: test)
- `-i` | `--imgdir` _dirpath_: path to the directory where the bitstream objects are stored (default: .)
- `-S` | `--sipi` _SIPIserver_: URL of the SIPI IIIF server (default: http://0.0.0.0:1024)

The command is used to upload data defined in an XML file onto a DSP server. The following example shows how to upload
data from an XML file `xml_data_file.xml` onto the DSP server `https://api.dsl.server.org` provided with the `-s` option.
The username `root@example.com` and the password `test` are used. The interface for the SIPI IIIF server is provided
with the `-S` option (`https://iiif.dsl.server.org`).

```bash
dsp-tools xmlupload -s https://api.dsl.server.org -u root@example.com -p test -S https://iiif.dsl.server.org xml_data_file.xml
```

The description of the expected XML format can be found [here](./dsp-tools-xmlupload.md).

## Convert an Excel file into a JSON file that is compatible with dsp-tools

```bash
dsp-tools excel [options] excel_file.xlsx output_file.json
```

The following options are available:

- `-S` | `--sheet` _sheetname_: name of the Excel worksheet to use (default: Tabelle1)
- `-s` | `--shortcode` _shortcode_: shortcode of the project (required)
- `-l` | `--listname` _listname_: name to be used for the list and the list definition file (required)
- `-L` | `--label` _label_: label to be used for the list (required)
- `-x` | `--lang` _lang_: language used for the list labels and commentaries (default: en)
- `-v` | `--verbose`: If set, some information about the progress is printed to the console.

The description of the expected Excel format can be found [here](./dsp-tools-create.md#lists-from-excel).
Loading

0 comments on commit 591b5ad

Please sign in to comment.