Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat new server implementation #131

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions .dockerignore

This file was deleted.

1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,4 @@ __pycache__
.env
dist/
.python-version
.pytest_cache
24 changes: 0 additions & 24 deletions .pre-commit-config.yaml

This file was deleted.

124 changes: 46 additions & 78 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,114 +2,82 @@

`tdp-server` provides a server to interact with [`tdp-lib`](https://github.com/tOSIT-IO/tdp-lib).

A full beginers guide to launch a testing server is provided at [`docs/quick-start.md`](./docs/quick-start.md).

## Requirements

The server is made with Python. The following is required:

- [Python](https://www.python.org/) `^3.6`
- [Poetry](https://python-poetry.org/) `1.1.15`
- A RDBMS, to store deployment history (e.g. [PostgresQL](https://www.postgresql.org/))
- An identity management system, for authentication purposes (e.g. [Keycloak](https://www.keycloak.org/))
- [Python](https://www.python.org/) `3.9.5`
- [Poetry](https://python-poetry.org/) `1.7.1`

### Dev environment
## Installation

A dev/testing environment using Keycloak and PostgreSQL is provided through `docker-compose`:
Install the `tdp-server` dependencies in the pyproject.toml as follows:

```bash
docker-compose -f dev/docker-compose.yml up -d
poetry install
```

## Installation
## Usage

1. Use Poetry to download and install the required Python dependencies:
```bash
poetry install
```
1. Define the required environment variables in an `.env` file. An example file is provided in `dev/.env-dev`:
```bash
cp dev/.env-dev .env
```

In particular:

- `DATABASE_DSN`: the data source name of the RDBMS.
- `TDP_COLLECTION_PATH`: the path to one or more TDP collection, separated by `:` (as [`tdp-collection`](https://github.com/TOSIT-IO/tdp-collection) and [`tdp-collection-extras`](https://github.com/TOSIT-IO/tdp-collection-extras)).
- `TDP_VARS`: the path to an empty directory where the `tdp_vars` will be stored and versioned.
- `TDP_RUN_DIRECTORY`: the path to the directory where the Ansible command will be launched (as [`tdp-getting-started`](https://github.com/tOSIT-IO/tdp-getting-started) for example).
_Note: the `ansible.cfg` file of the working directory must contain the path of the `tdp_vars` directory defined previously._
1. Initialize the database and the `tdp_vars` directory:
```bash
python tdp_server/initialize_database.py
python tdp_server/initialize_tdp_vars.py
```
To see the specification:

## Usage
```sh
# Launch the server
poetry run uvicorn tdp_server.main:app --reload
```

Start the server using:
Specification is available at <http://localhost:8000/docs>.

```bash
uvicorn tdp_server.main:app --reload
```
## Discussion

## Build Docker Container
- Endpoint prefix `/api/v1` as base path has been chosen.

```bash
docker build -t tdp_server -f docker/Dockerfile .
```
[Zalando API guidelines](https://opensource.zalando.com/restful-api-guidelines/#urls) recommends directly starting it from `/`, however the [Ambari server](https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/components.md) choses this endpoint prefix and it explicits what the user is using. Moreover the [Naming Conventions of the Australian government API](https://api.gov.au/sections/naming-conventions.html) states that the version must be specified.

## Run Docker Container
- For versioning `v1` has been chosen for first version.

Executing the container with the minimal configuration variables:
`v1` has been chosen like in the Ambari server, the [Naming Conventions of the Australian government API](https://api.gov.au/sections/naming-conventions.html) give an example with the version `v1`, however, the [Zalando API guidelines](https://opensource.zalando.com/restful-api-guidelines/#116) state that we must chose semantic versioning.

```bash
docker run \
-e TDP_COLLECTION_PATH="/tdp/ops/tdp/ansible/ansible_collections/tosit/tdp" \
-e TDP_RUN_DIRECTORY="/tdp/ops" \
-e TDP_VARS="/tdp/ops/inventory/tdp_vars" \
-e DATABASE_DSN=sqlite:////tdp/sqlite.db \
-e OPENID_CONNECT_DISCOVERY_URL="http://host.docker.internal:8080/auth/realms/tdp_server/.well-known/openid-configuration" \
-e OPENID_CLIENT_ID=tdp_server \
-e OPENID_CLIENT_SECRET=secret \
-v "..../sqlite.db:/tdp/sqlite.db" \
-v"..../tdp-ops:/tdp/ops" \
-p 8000:8000 \
tdp_server
```
- All resources are pluralized except `plan`.

N.B.: Mounting a sqlite database is not the recommended way to persist the server's data.
Since all resources consist of several items (configurations, services, components, deployments), the resource name is in plural form as stated as a must requirement in the [Zalando API guideline](https://opensource.zalando.com/restful-api-guidelines/#134) and in the [Naming Conventions of the Australian government API](https://api.gov.au/sections/naming-conventions.html). Only the `\plan` resource is singular since there is only one plan before it can be deployed and also to be consistent with TDP lib.

### Accessing the REST API
- All resource names are written in lower-case.

A token must be provided to access the API. Tokens can be obtained using the `get_token.py` script.
As stated in the [Naming Conventions of the Australian government API](https://api.gov.au/sections/naming-conventions.html) all resource names must be written in lower-case.

For example, using `curl`:
- Query parmeters in URL are written using snake_case.

```bash
token=$(python get_token.py)
curl -H "Authorization: Bearer $token" http://localhost:8000/api/v1/service/
```
The [Naming Conventions of the Australian government API](https://api.gov.au/sections/naming-conventions.html) recommend using snake_case or camelcase for query parameters, however in the [FastAPI documentation](https://fastapi.tiangolo.com/tutorial/query-params/) snake_case is used and it is also the way to declare variables in python. Query parameters whch are used as path parameters in the URL are therefore also in snake_case although they shouldn't be according to the first document but is performed in the FastAPI documentation.

### Accessing the API documentation pages
- Kebab-case will be used for path segments except query parameters.

Documentation pages of the API are available at:
For now, no resource name or action is a combination of two words. However, if it is the case, kebab-case must be used as stated in the [Naming Conventions of the Australian government API](https://api.gov.au/sections/naming-conventions.html) and the [Zalando API guideline](https://opensource.zalando.com/restful-api-guidelines/#129).

- OpenAPI UI <http://localhost:8000/docs>
- ReDoc UI <http://localhost:8000/redoc>
- Verbs have been chosen where actions are done.

## Contributing
The [Zalando API guidelines](https://opensource.zalando.com/restful-api-guidelines/#urls) state that it MUST be verb-free and the same is mentioned in the[dreamfactory blog](https://blog.dreamfactory.com/best-practices-for-naming-rest-api-endpoints/). The reason is that with the path we should access ressources and the actions are defined with the GET, PUT, POST, PATCH, DELETE methods. However, we are not using a REST API and with `/plan` we perform different actions such as `resume` or `reconfigure` with the same post method and both actions are not performed on different resources so introducing a verb is the best way of description. For the `deploy` endpoint the question is more debatable, however the clearest wording is favored and therefore we chose to maintain the verb.

`tdp-server` uses [Git Hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) to enforce consistency in the code and commit messages.
- No empty path segments and trailing slashes.

Use Poetry to install the hooks:
The [Zalando API guidelines](https://opensource.zalando.com/restful-api-guidelines/#136) state that the URL must not have any empty path segments or trailing slashes. The [FastAPI documentation](https://fastapi.tiangolo.com/tutorial/query-params/) give an example where there is a trailing slash, however to be as clean as possible and avoid any confusion no trailing slash occurs in any URL.

```bash
poetry run pre-commit install --hook-type pre-commit
poetry run pre-commit install --hook-type commit-msg
```
- Used domain specific resource names.

Each resource name is domain specific which represent either the elements in TDP (`/services` and `/components`) or actions in TDP Manager as stated as a must requirement in the [Zalando API guidelines](https://opensource.zalando.com/restful-api-guidelines/#142).

- Resources and sub-resources are identified via path segments.

The `components` resource is a sub-resource of `services` which is a sub-resource of `configurations`. The path segment for a component looks the following `/api/v1/configurations/services/{service_id}/components/{component_id}`. This has been chosen to improve consumer experience while following the setup of TDP. The [Zalando API guidelines](https://opensource.zalando.com/restful-api-guidelines/#143) state this practice as a must requirement.

- Cursor-based pagination instead of offset-based has been chosen.

This is stated as a SHOULD requirement in the [Zalando API guidelines](https://opensource.zalando.com/restful-api-guidelines/#pagination).

- JSON is used as response body.

The JSON format is used as response body of every endpoint so that each function returns a JSON object as stated as a must requirement in the [Zalando API guidelines](https://opensource.zalando.com/restful-api-guidelines/#167). Schemas are written using the Pydantic BaseModel which will read the body as JSON. The [FastAPI documentation](https://fastapi.tiangolo.com/tutorial/body/) shows this method to declare request bodies.

The following environment variables can be used to ease development:
- Enum values are declared as UPPER_CASE.

- `DO_NOT_USE_IN_PRODUCTION_DISABLE_TOKEN_CHECK`
- `MOCK_DEPLOY`
As recommended by the [Zalando API guidelines](https://opensource.zalando.com/restful-api-guidelines/#240) enum values should be written as UPPER_SNAKE_CASE.
108 changes: 0 additions & 108 deletions alembic.ini

This file was deleted.

17 changes: 0 additions & 17 deletions dev/.env-dev

This file was deleted.

18 changes: 0 additions & 18 deletions dev/config/init_script.sql

This file was deleted.

13 changes: 0 additions & 13 deletions dev/config/pg_servers.json

This file was deleted.

Loading
Loading