Task: CLI artifact add command rstuf artifact add

[kairoaraujo]

Implement to the rstuf CLI the functionality to add artifacts.

Two specific features to add targets:

Add Artifact (Feature)

This target adds the target using the RSTUF REST API. This feature is an interface to add targets using the command line.

1. Simple one artifact file

Note: the user can use the token (-t/--token) when API requires authentication/authorization, if the user uses rstuf --auth it ignores the token parameter.

Giving the artifact

rstuf artifact add <file> [-p/--path <file-path>, -c/--custom, -t/--token <token>]

Example: rstuf artifact add file-v1.0.3.tar.gz -p projectA/download/

It gets from the file the hash, size and add a custom path in the metadata projectA/download/file-v1.0.3.tar.gz

Without file, just the information

rstuf artifact add -p/--path [file-path], -l/lenght [length] -h/hashes [b2sum] -c/--custom

2. From JSON

rstuf artifact add -d @file.json

{
  "targets": [
    {
      "info": {
        "length": "int",
        "hashes": {
          "blake2b-256": "str"
        },
        "custom": {
          "key": "value"
        }
      },
      "path": "str"
    },
    {
      "info": {
        "length": "int",
        "hashes": {
          "blake2b-256": "str"
        },
        "custom": {
          "key": "value"
        }
      },
      "path": "str"
    },
  ]
}

It should return the task id.

Import targets (Feature)

• [DONE] feature: import targets #195

This feature adds a huge amount of targets, importing from CSV directly to RSTUF Database
The feature description is detailed in these links

• Feature: data load for migrations repository-service-tuf#188
• BDD Feature: Feature: FT for import targets to RSTUF DB repository-service-tuf#218

[kairoaraujo]

• Add a feature rstuf admin add-target option to import directly to DB. It is related to Feature: data load for migrations repository-service-tuf#188

CLI Implementation

The RSTUF user will be required to create a CSV file that contains the following information
path;size;hash algorithm;hash;
Example:
test/optimistic_lamport-v1.0.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a;

The CSV file limit will be half-million of targets (500,000 lines)

The CLI will implement as an admin command, and it will require some parameters

metadata-url: Address where the CLI can download the RSTUF TUF Metadata. i.e.: http://127.0.0.1:8080

db-uri: the RSTUF SQL DB URI. i.e.: postgresql://postgres:secret@127.0.0.1:5433
csv the CSV path. i.e targets-1of2.csv
Example:

rstuf admin add-targets -metadata-url http://127.0.0.1:8080 -db-uri postgresql://postgres:secret@127.0.0.1:5433 -csv ../repository-service-tuf/tests/data/targets-1of2.csv
The CLI will insert all the targets to the RSTUF SQL DB directly.
Running in a Macbook Pro (2019) 2,4 GHz 8-Core Intel Core i9/32GB 2667 MHz DDR4

import time: ~15 seconds

[kairoaraujo]

Requires for implementation:

Add targets (Feature)

• Requires a Feature: BDD test implementation for refinement

Import targets (Feature)

• Feature: FT for import targets to RSTUF DB repository-service-tuf#218 Approved for implementation.
• Feature: Add endpoint to trigger publish_targets in the Rest API repository-service-tuf-api#242
• feature: import targets #195 (comment)

[kairoaraujo]

I updated the Issue description and the comment above.

[MVrachev]

I think we can close that one, am I right @kairoaraujo?

[kairoaraujo]

We added only part of this.

• add targets
• import targets

[MVrachev]

Sorry, but the comment you are referring to has three todo items and all of them are marked as complete.
What do I miss?

[kairoaraujo]

Sorry, but the comment you are referring to has three todo items and all of them are marked as complete.
What do I miss?

There are two feauteres there (subtitles) there: Add targets (Feature) and Import targets (Feature)

[kairoaraujo]

@KAUTH

I suggest breaking down this feature into three sub-tasks

• Implement rstuf artifact add giving a file (Task: Implement rstuf artifact add giving a file #343)
• Implement rstuf artifact add giving file information (Task: Implement rstuf artifact add giving file information #402)
• Implement rstuf artifact add giving JSON file

So we can delivery small features and discuss more details in each.

[KAUTH]

@kairoaraujo good point! I was thinking of breaking it similarly as well.

I was also considering that giving file information can also be substituted with giving JSON file with information for only one file. Adding an actual file I see the immediate value, but do you think we need both JSON file and file info as well?

[kairoaraujo]

The JSON file is for adding a list of files, but it is a low priority.

[KAUTH]

The JSON file is for adding a list of files, but it is a low priority.

Yeah, my point is one file is also a list of one. So, do we want both JSON file and passing file information?
In any case, I am working on "Implement rstuf artifact add giving a file. I will create tasks for each one and link them to this issue.