Skip to content

Commit

Permalink
feature: import targets (#195)
Browse files Browse the repository at this point in the history 
* feature: import targets

Implements the feature import target `rstuf admin import-targets`.

This feature gives the RSTUF administrator the functionality to
import a large number of existent targets. It helps to roll out and
deploy the RSTUF in existing repositories.

This feature is detailed and explained at these links
- repository-service-tuf/repository-service-tuf#188
- BDD Feature: repository-service-tuf/repository-service-tuf#218

Some changes in the ceremony was done to reuse in the code:

  - The `ceremony._check_server` was converted to
  `helpers.api_client.get_headers`
  - The `ceremony._bootstrap_state` was converted to
  `helpers.api_client.task_status`

Added 100% coverage to `helpsers.api_client`

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

* Fix the sentences

Co-authored-by: Martin Vrachev <martin.vrachev@gmail.com>

* Make the optional dependencies clear and docs

The optional dependencies `psycopg2` and `sqlalchemy` are optional.
This is required only when the user aim to use the `import-targets`
feature.

Add to documentation the new feature

Add details about the usage and CSV

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

* Add typing notation

- add typing notation
- make the `_check_csv_files` more explicity

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

* fix tests for changed commited to the ceremony

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

* fix the bootstrap check for ceremony

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

* add unit tests for import-targets

- Added some small refactoring in the implementation
- 100% coverage for the UT
- Fix some comments from the review

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

* Removed unused Mock from test_import_targets

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

* fix linting

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

* Apply suggestions from code review

Co-authored-by: Martin Vrachev <martin.vrachev@gmail.com>

* Add missing asserts

Co-authored-by: Martin Vrachev <martin.vrachev@gmail.com>

* fix import-targets params, remove duplicate checks

- move parameters to use double dash `--{names}`
- remove duplicate check that is done by `is_logged`
- fix tests

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

* fix bug in is_logged, typing and typo

- fix `is_logged` bug in case of no data with 200 status code
- fix typing for the `task_status` response
- fix typo/wording for task status

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

* remove `**kw` from `api_client.Login` lambdas

- remove login keyword parameters from lambda

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

* update documentation using `--` to the parameters

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>

---------

Signed-off-by: Kairo de Araujo <kdearaujo@vmware.com>
Co-authored-by: Martin Vrachev <martin.vrachev@gmail.com>
  • Loading branch information
@MVrachev
Kairo Araujo and MVrachev committed Feb 16, 2023
1 parent 913e1d8 commit 6282adc
Show file tree
Hide file tree
Showing 13 changed files with 1,462 additions and 410 deletions.
2 changes: 2 additions & 0 deletions Pipfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ requests = "*"
tuf = "==2.0.0"
dynaconf = {extras = ["ini"], version = "*"}
isort = "*"
sqlalchemy = "*"
psycopg2 = "*"

[dev-packages]
black = "*"
Expand Down
238 changes: 196 additions & 42 deletions Pipfile.lock
View file

Large diffs are not rendered by default.

76 changes: 75 additions & 1 deletion docs/source/guide/index.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -448,4 +448,78 @@ Show token detailed information.
"expiration": "2022-09-04T08:42:44"
},
"message": "Token information"
}
}
Import Targets (``import-targets``)
-----------------------------------
This feature imports a large number of targets directly to RSTUF Database.
RSTUF doesn't recommend using this feature for regular flow, but in case you're
onboarding an existent repository that contains a large number of targets.
This feature requires extra dependencies:
.. code:: shell
pip install repository-service-tuf[psycopg2,sqlachemy]
To use this feature, you need to create CSV files with the content to be imported
by RSTUF CLI.
This content requires the following data:
- `path <https://theupdateframework.github.io/specification/latest/#targetpath>`_: The target path
- `size <https://theupdateframework.github.io/specification/latest/#targets-obj-length>`_: The target size
- `hash-type <https://theupdateframework.github.io/specification/latest/#targets-obj-length>`_: The defined hash as a metafile. Example: blak2b-256
- `hash <https://theupdateframework.github.io/specification/latest/#targets-obj-length>`_: The hash
The CSV must use a semicolon as the separator, following the format
``path;size;hash-type;hash`` without a header.
See the below CSV file example:
.. code::
relaxed_germainv1.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a
vigilant_keldyshv2.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a
adoring_teslav3.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a
funny_greiderv4.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a
clever_ardinghelliv5.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a
pbeat_galileov6.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a
wonderful_gangulyv7.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a
sweet_ardinghelliv8.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a
brave_mendelv9.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a
nice_ridev10.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a
.. code:: shell
❯ rstuf admin import-targets -h
Usage: rstuf admin import-targets [OPTIONS]
Import targets to RSTUF from exported CSV file.
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ * --metadata-url TEXT RSTUF Metadata URL i.e.: http://127.0.0.1 . [required] │
│ * --db-uri TEXT RSTUF DB URI. i.e.: postgresql://postgres:secret@127.0.0.1:5433 [required] │
│ * --csv TEXT CSV file to import. Multiple --csv parameters are allowed. See rstuf CLI guide for more details. [required] │
│ --skip-publish-targets Skip publishing targets in TUF Metadata. │
│ --help -h Show this message and exit. │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
❯ rstuf admin import-targets --db-uri postgresql://postgres:secret@127.0.0.1:5433 --csv targets-1of2.csv --csv targets-2of2.csv --metadata-url http://127.0.0.1:8080/
Import status: Loading data from ../repository-service-tuf/tests/data/targets-1of2.csv
Import status: Importing ../repository-service-tuf/tests/data/targets-1of2.csv data
Import status: ../repository-service-tuf/tests/data/targets-1of2.csv imported
Import status: Loading data from ../repository-service-tuf/tests/data/targets-2of2.csv
Import status: Importing ../repository-service-tuf/tests/data/targets-2of2.csv data
Import status: ../repository-service-tuf/tests/data/targets-2of2.csv imported
Import status: Commiting all data to the RSTUF database
Import status: All data imported to RSTUF DB
Import status: Submitting action publish targets
Import status: Publish targets task id is dd1cbf2320ad4df6bda9ca62cdc0ef82
Import status: task STARTED
Import status: task SUCCESS
Import status: Finished.
4 changes: 4 additions & 0 deletions pyproject.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -46,5 +46,9 @@ ignore_missing_imports = true
[tool.hatch.version]
path = "repository_service_tuf/__version__.py"

[project.optional-dependencies]
psycopg2 = ["psycopg2>=2.9.5"] # required by import-targets sub-command
sqlalchemy = ["sqlalchemy>=2.0.1"] # required by import-targets sub-command

[project.scripts]
rstuf = "repository_service_tuf.cli:rstuf"
95 changes: 6 additions & 89 deletions repository_service_tuf/cli/admin/ceremony.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
#
import json
import os
import time
from dataclasses import asdict, dataclass
from enum import Enum
from typing import Any, Dict, Optional
Expand All @@ -29,8 +28,9 @@
from repository_service_tuf.helpers.api_client import (
URL,
Methods,
is_logged,
get_headers,
request_server,
task_status,
)
from repository_service_tuf.helpers.tuf import (
RolesKeysInput,
Expand Down Expand Up @@ -426,37 +426,6 @@ def _configure_keys(rolename: str, role: RolesKeysInput) -> None:
key_count += 1


def _check_server(settings) -> dict[str, str]:
server = settings.get("SERVER")
token = settings.get("TOKEN")
if server and token:
token_access_check = is_logged(server, token)
if token_access_check.state is False:
raise click.ClickException(
f"{str(token_access_check.data)}"
"\n\nTry re-login: 'Repository Service for TUF admin login'"
)

expired_admin = token_access_check.data.get("expired")
if expired_admin is True:
raise click.ClickException(
"Token expired. Run 'Repository Service for TUF admin login'"
)
else:
headers = {"Authorization": f"Bearer {token}"}
response = request_server(
server, URL.bootstrap.value, Methods.get, headers=headers
)
if response.status_code != 200 and (
response.json().get("bootstrap") is True or None
):
raise click.ClickException(f"{response.json().get('detail')}")
else:
raise click.ClickException("Login first. Run 'rstuf-cli admin login'")

return headers


def _bootstrap(server, headers, json_payload) -> Optional[str]:
task_id = None
response = request_server(
Expand Down Expand Up @@ -486,58 +455,6 @@ def _bootstrap(server, headers, json_payload) -> Optional[str]:
return task_id


def _bootstrap_state(task_id, server, headers) -> None:
received_state = []
while True:
state_response = request_server(
server, f"{URL.task.value}{task_id}", Methods.get, headers=headers
)

if state_response.status_code != 200:
raise click.ClickException(
f"Unexpected response {state_response.text}"
)

data = state_response.json().get("data")

if data:
if state := data.get("state"):
if state not in received_state:
console.print(f"Bootstrap status: {state}")
received_state.append(state)

if state == "SUCCESS":
try:
result = data.get("result")
bootstrap_result = result.get("details").get(
"bootstrap"
)
except AttributeError:
bootstrap_result = False

if bootstrap_result is not True:
raise click.ClickException(
f"Something went wrong, result: {result}"
)

console.print("[green]Bootstrap finished.[/]")
break

elif state == "FAILURE":
raise click.ClickException(
f"Failed: {state_response.text}"
)
else:
raise click.ClickException(
f"No state in data received {state_response.text}"
)
else:
raise click.ClickException(
f"No data received {state_response.text}"
)
time.sleep(2)


@admin.command()
@click.option(
"-b",
Expand Down Expand Up @@ -599,20 +516,20 @@ def ceremony(context, bootstrap, file, upload, save) -> None:

settings = context.obj["settings"]
if bootstrap:
headers = _check_server(settings)
headers = get_headers(settings)
bs_response = request_server(
settings.SERVER, URL.bootstrap.value, Methods.get, headers=headers
)
bs_data = bs_response.json()
if bs_response.status_code == 404:
raise click.ClickException(
f"Server {settings.SERVER} doesn't allow bootstrap"
)
if bs_response.status_code != 200:
raise click.ClickException(
f"Error {bs_response.status_code} {bs_data.get('detail')}"
f"Error {bs_response.status_code} {bs_response.text}"
)

bs_data = bs_response.json()
if bs_data.get("bootstrap") is True or None:
raise click.ClickException(f"{bs_data.get('message')}")

Expand Down Expand Up @@ -775,6 +692,6 @@ def ceremony(context, bootstrap, file, upload, save) -> None:
if task_id is None:
raise click.ClickException("task id wasn't received")

_bootstrap_state(task_id, settings.SERVER, headers)
task_status(task_id, settings.SERVER, headers, "Bootstrap status: ")

console.print("\nCeremony done. 🔐 🎉")

0 comments on commit 6282adc

Please sign in to comment.