Feature: data load for migrations

[kairoaraujo]

What is the feature about?

There is a use case of RSTUF where a repository already has a lot of existing data.
Example: PyPI has 7+ million packages, NPM has 1.2+ million packages, etc.

If a repository decides to adopt RSTUF, loading all this pre-existent using the process of making requests to the API, waiting it to be processed by the worker can take a long time.

A suggestion is to implement to RSTUF CLI a new feature for data migration/load.
it could be faster if the data can be submitted in a structured way directly to RSTUF SQL DB.

1. The data can be imported from a generic format, for example CSV (path;size;hash;custom;)
2. The CLI already gets the delegated hash bin role based on the path (The hash bins size is available in API /api/v1/config)
3. CLI can be executed from a host with access to RSTUF SQL DB and add all the data in batches, for example by delegated hash bin role name
4. CLI sends a task publish_targets and waits for it to be processed.

This might be related to:

• Task: CLI artifact add command rstuf artifact add repository-service-tuf-cli#39

A prototype + feature is required.

Services it relates to

repository-service-for-tuf-cli

Related tasks

No response

References

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[kairoaraujo]

My proposal for this Feature is based on a prototype done with CLI to investigate the required changes in the Worker and some information about the benchmark.

Context:

• I'm considering millions of existent packages to be migrated/added to RSTUF during the deployment
• I'm considering the RSTUF setup has a big number of hash bins delegated roles: 16,384

CLI Implementation

• The RSTUF user will be required to create a CSV file that contains the following information
  path;size;hash algorithm;hash;
  Example:
  test/optimistic_lamport-v1.0.tar.gz;12345;blake2b-256;716f6e863f744b9ac22c97ec7b76ea5f5908bc5b2f67c61510bfc4751384ea7a;

• The CSV file limit will be half-million of targets (500,000 lines)

• The CLI will implement as an admin command, and it will require some parameters

  • metadata-url: Address where the CLI can download the RSTUF TUF Metadata. i.e.: http://127.0.0.1:8080
  • db-uri: the RSTUF SQL DB URI. i.e.: postgresql://postgres:secret@127.0.0.1:5433
  • csv the CSV path. i.e targets-1of2.csv

  Example:

  rstuf admin add-targets -metadata-url http://127.0.0.1:8080 -db-uri postgresql://postgres:secret@127.0.0.1:5433 -csv ../repository-service-tuf/tests/data/targets-1of2.csv

The CLI will insert all the targets to the RSTUF SQL DB directly.
Running in a Macbook Pro (2019) 2,4 GHz 8-Core Intel Core i9/32GB 2667 MHz DDR4

• import time: ~15 seconds

Worker:

I identified some changes (I will file separate task issues)

1. [must] The automatic publish_targets job should be deactivated or removed.
   Loading a huge amount of data will take a long time to process, and an automatic job in between will also try to update the existing/unpublished targets from the database. I will file an issue about it where we can discuss the best approach: do we need this job as each add/remove files its publish_target task? What happens if a failure occurs and it is not published? ...

2. [must] Improve the database update for published target files. We do it one by one instead using a query/update in CRUD. Also the place where we update the target from the database.

3. [nice to have] We can get the bin roles with unpublished targets ordered by role name.

4. [nice to have] We could skip the settings refresh for a sub-task, as the main task already updated the settings.

5. [maybe must] We need to have a way to trigger the publish_targets from the CLI (through REST API).
   I added it as maybe must, as It depends on the decision on item 1.

I was doing it manually.

❯ docker exec -it repository-service-tuf-worker_repository-service-tuf-worker_1 /bin/bash
root@1509b2901ab3:/opt/repository-service-tuf-worker# python
Python 3.10.7 (main, Sep 13 2022, 01:53:53) [GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import app
14:42:09 DEBUG STORAGE_BACKEND is defined as <class 'repository_service_tuf_worker.services.storage.local.LocalStorage'>
14:42:09 DEBUG KEYVAULT_BACKEND is defined as <class 'repository_service_tuf_worker.services.keyvault.local.LocalKeyVault'>
14:42:09 DEBUG STORAGE_BACKEND is defined as <class 'repository_service_tuf_worker.services.storage.local.LocalStorage'>
14:42:09 DEBUG KEYVAULT_BACKEND is defined as <class 'repository_service_tuf_worker.services.keyvault.local.LocalKeyVault'>
>>> app.repository._send_publish_targets_task('import')
14:42:33 DEBUG Start from server, version: 0.9, properties: {'capabilities': {'publisher_confirms': True, 'exchange_exchange_bindings': True, 'basic.nack': True, 'consumer_cancel_notify': True, 'connection.blocked': True, 'consumer_priorities': True, 'authentication_failure_close': True, 'per_consumer_qos': True, 'direct_reply_to': True}, 'cluster_name': 'rabbit@f789dd71edf0', 'copyright': 'Copyright (c) 2007-2022 VMware, Inc. or its affiliates.', 'information': 'Licensed under the MPL 2.0. Website: https://rabbitmq.com', 'platform': 'Erlang/OTP 25.0.4', 'product': 'RabbitMQ', 'version': '3.10.7'}, mechanisms: [b'PLAIN', b'AMQPLAIN'], locales: ['en_US']
14:42:33 DEBUG using channel_id: 1
14:42:33 DEBUG Channel open
>>> 

On my basic worker prototype:
Running in a Macbook Pro (2019) 2,4 GHz 8-Core Intel Core i9/32GB 2667 MHz DDR4:

• Publishing 500,000 targets: ~40 minutes
  1. Loaded 1 file with 500,000 targets
  2. Call the publish_targets to publish all targets
• Publishing 1,000,000 targets: ~55 minutes
  1. Loaded 1 of 2 file with 500,000 targets
  2. Loaded 2 of 2 file with 500,000 targets
  3. Call the publish_targets to publish all targets

The possible bottleneck/processing is publishing all targets sequentially.

In the second prototype, I also tried to improve this algorithm using Celery Canvas, creating an asynchronous group of new tasks.
But it didn't improve the performance for publishing the targets. It was ~40 minutes as well

[MVrachev]

Sounds like a reasonable plan.
Good breakdown of the feature.

[kairoaraujo]

Related tasks/issues

• Feature: FT for import targets to RSTUF DB #218
• Task: Is the automatic publish_targets job is required? repository-service-tuf-worker#175 (feedback task)
• Task: Improve database update for published targets repository-service-tuf-worker#176
• Task: [nice to have] order the delegated hash bin targets roles by name repository-service-tuf-worker#177
• Task: [nice to have] We could skip the settings refresh for a sub-task repository-service-tuf-worker#178
• Task: CLI artifact add command rstuf artifact add repository-service-tuf-cli#39
• Task: Document how to import existent targets to RSTUF #226
• Feature: Add endpoint to trigger publish_targets in the Rest API repository-service-tuf-api#242
• Enhancement: Improve the task monitoring the sub-task, to have a better update for task repository-service-tuf-worker#190 [not required]

[kairoaraujo]

The implementation load data feature in CLI is implemented. I'm checking as done in the above list of required tasks

[kairoaraujo]

All tasks for this feature are completed. I am closing this issue/feature.