diff --git a/CHANGES.rst b/CHANGES.rst index a936ff23..311b2e21 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -141,7 +141,3 @@ Misc - `#4592 `_, `#5491 `_, `#5492 `_, `#5580 `_, `#5633 `_, `#5693 `_, `#5867 `_, `#6035 `_ - ----- - - diff --git a/CHANGES/6145.doc b/CHANGES/6145.doc new file mode 100644 index 00000000..86e85e94 --- /dev/null +++ b/CHANGES/6145.doc @@ -0,0 +1 @@ +Moved README to readthedocs website. diff --git a/README.md b/README.md index 5ecc4443..ec86d90d 100644 --- a/README.md +++ b/README.md @@ -5,203 +5,6 @@ Supported plugins: - Pulp 2 ISO can be migrated into Pulp 3 File. - Pulp 2 Docker can be migrated into Pulp 3 Container. - RPM plugin migration is planned and currently in development. - -### Requirements - -* /var/lib/pulp is shared from Pulp 2 machine -* access to Pulp 2 database - -### Configuration -On Pulp 2 machine: - -1. Make sure MongoDB listens on the IP address accesible outside, it should be configured as -one of the `bindIP`s in /etc/mongod.conf. - -2. Make sure /var/lib/pulp is on a shared filesystem. - - -On Pulp 3 machine: -1. Mount /var/lib/pulp to your Pulp 3 storage location. By default, it's /var/lib/pulp. - -2. Configure your connection to MongoDB in /etc/pulp/settings.py. You can use the same configuration - as you have in Pulp 2 (only seeds might need to be different, it depends on your setup). - -E.g. -```python -PULP2_MONGODB = { - 'name': 'pulp_database', - 'seeds': ':27017', - 'username': '', - 'password': '', - 'replica_set': '', - 'ssl': False, - 'ssl_keyfile': '', - 'ssl_certfile': '', - 'verify_ssl': True, - 'ca_path': '/etc/pki/tls/certs/ca-bundle.crt', -} -``` - -### Installation - -Clone the repository and install it. -``` -$ git clone https://github.com/pulp/pulp-2to3-migration.git -$ pip install -e pulp-2to3-migration -``` - -Or add it to [the ansible installer](https://github.com/pulp/ansible-pulp) configuration like any - other pulp plugin. - - -### User Guide - -#### Migration Plan - -To configure what to migrate to Pulp 3, one needs to define a Migration Plan (MP). -A MP defines which plugins to migrate and how. -Migration plugin is declarative, fully migrated content and repositories in Pulp 3 will - correspond to the most recent MP which has been run. - Type of a plugin specified in a MP is a Pulp 2 plugin. E.g. one needs to specify `iso` to - migrate content and repositories into Pulp 3 File plugin. - Examples of the migration plan: - - - Migrate all for a specific plugin - -```json -{ - "plugins": [ - { - "type": "iso" - } - ] -} -``` - - - Migrate a pulp 2 repository `file` (into a Pulp 3 repository with one repository version - ), using an importer from a pulp 2 - repository `file2`, and - distributors from a repository `file` and `file2` - -```json -{ - "plugins": [ - { - "type": "iso", - "repositories": [ - { - "name": "file", - "repository_versions": [ - { - "pulp2_repository_id": "file", - "pulp2_distributor_repository_ids": [ - "file", "file2" - ] - } - ], - "pulp2_importer_repository_id": "file2" - } - ] - } - ] -} - -``` - -#### Workflow - -All the commands should be run on Pulp 3 machine. - -1. Create a Migration Plan -``` -$ # migrate content for Pulp 2 ISO plugin -$ http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "iso"}]}' - -HTTP/1.1 201 Created -{ - "pulp_created": "2019-07-23T08:18:12.927007Z", - "pulp_href": "/pulp/api/v3/migration-plans/59f8a786-c7d7-4e2b-ad07-701479d403c5/", - "plan": "{ \"plugins\": [{\"type\": \"iso\"}]}" -} - -``` - -2. Use the ``pulp_href`` of the created Migration Plan to run the migration -``` -$ http POST :24817/pulp/api/v3/migration-plans/59f8a786-c7d7-4e2b-ad07-701479d403c5/run/ - -HTTP/1.1 202 Accepted -{ - "task": "/pulp/api/v3/tasks/55db2086-cf2e-438f-b5b7-cd0dbb7c8cf4/" -} - -``` - -3. For listing the mapping for Pulp 2 to Pulp 3 -``` -$ http :24817/pulp/api/v3/pulp2repositories/ - -HTTP/1.1 200 OK -{ - "count": 1, - "next": null, - "previous": null, - "results": [ - { - "is_migrated": true, - "not_in_plan": false, - "pulp2_object_id": "5dbc478c472f68283ad8e6bd", - "pulp2_repo_id": "file-large", - "pulp3_distribution_hrefs": [], - "pulp3_publication_href": [], - "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/ca0e505e-51c2-46e1-be40-3762d372f9b2/", - "pulp3_repository_version": null, - "pulp_created": "2019-11-01T14:59:04.648920Z", - "pulp_href": "/pulp/api/v3/pulp2repositories/92c6d1c8-718b-4ea9-8a23-b2386849c2c5/" - } - ] -} - -``` - -### Plugin Writer's Guide - -If you are extending this migration tool to be able to migrate the plugin of your interest -from Pulp 2 to Pulp 3, here are some guidelines. - - -1. Create a migrator class (subclass the provided `Pulp2to3PluginMigrator` class). There should be - one migrator class per plugin. Define all the necessary attributes and methods for it (see - `Pulp2to3PluginMigrator` for more details) - -2. Discovery of the plugins is done via entry_points. Add your migrator class defined in step 1 - to the list of the "migrators" entry_points in setup.py. - -3. Add a Content model to communicate with Pulp 2. - - It has to have a field `TYPE_ID` which will correspond to the `_content_type_id` of your Content - in Pulp 2. Don't forget to add it to `pulp2_content_models` in step 1. - -4. Add a Content model to pre-migrate Pulp2 content to (subclass the provided `Pulp2to3Content` -class). It has to have: - - a field `pulp2_type` which will correspond to the `_content_type_id` of your Content in Pulp 2. - - on a Meta class a `default_related_name` set to `_detail_model` - - a classmethod `pre_migrate_content_detail` (see `Pulp2to3Content` for more details) - - a coroutine `create_pulp3_content` (see `Pulp2to3Content` for more details) - - Don't forget to add this Content model to your migrator class. - - If your content has one artifact and if you are willing to use the default implementation of the - first stage of DeclarativeContentMigration, on your Content model you also need: - - an `expected_digests` property to provide expected digests for artifact creation/validation - - an `expected_size` property to provide the expected size for artifact creation/validation - - a `relative_path_for_content_artifact` property to provide the relative path for content - artifact creation. - -5. Subclass the provided `Pulp2to3Importer` class and define `migrate_to_pulp3` method which -creates a plugin Remote instance based on the provided pre-migrated `Pulp2Importer`. - -6. Subclass the provided `Pulp2to3Distributor` class and define `migrate_to_pulp3` method which -creates a plugin Publication and/or Distribution instance (depends on the plugin) based on the -provided pre-migrated `Pulp2Distributor`. +For more information, please see the [documentation](docs/index.rst). diff --git a/docs/Makefile b/docs/Makefile index d3837963..6fb6cd79 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -53,7 +53,7 @@ diagrams: html: mkdir -p $(BUILDDIR)/html - curl -o _build/html/api.json "http://localhost:24817/pulp/api/v3/docs/api.json?plugin=pulp_2to3_migration" + curl -o _static/api.json "http://localhost:24817/pulp/api/v3/docs/api.json?plugin=pulp_2to3_migration" $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html @echo @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." diff --git a/docs/_static/api.json b/docs/_static/api.json new file mode 100644 index 00000000..99d17cf4 --- /dev/null +++ b/docs/_static/api.json @@ -0,0 +1 @@ +{"swagger": "2.0", "info": {"title": "Pulp 3 API", "license": {"name": "GPLv2+"}, "logo": {"url": "https://pulp.plan.io/attachments/download/517478/pulp_logo_word_rectangle.svg"}, "version": "v3"}, "host": "localhost:24817", "schemes": ["http"], "basePath": "/", "consumes": ["application/json"], "produces": ["application/json"], "securityDefinitions": {"Basic": {"type": "basic"}}, "security": [{"Basic": []}], "paths": {"/pulp/api/v3/migration-plans/": {"get": {"operationId": "migration-plans_list", "summary": "List migration plans", "description": "MigrationPlan ViewSet.", "parameters": [{"name": "limit", "in": "query", "description": "Number of results to return per page.", "required": false, "type": "integer"}, {"name": "offset", "in": "query", "description": "The initial index from which to return the results.", "required": false, "type": "integer"}, {"name": "fields", "in": "query", "description": "A list of fields to include in the response.", "required": false, "type": "string"}, {"name": "exclude_fields", "in": "query", "description": "A list of fields to exclude from the response.", "required": false, "type": "string"}], "responses": {"200": {"description": "", "schema": {"required": ["count", "results"], "type": "object", "properties": {"count": {"type": "integer"}, "next": {"type": "string", "format": "uri", "x-nullable": true}, "previous": {"type": "string", "format": "uri", "x-nullable": true}, "results": {"type": "array", "items": {"$ref": "#/definitions/pulp_2to3_migration.MigrationPlan"}}}}}}, "tags": ["migration-plans"]}, "post": {"operationId": "migration-plans_create", "summary": "Create a migration plan", "description": "MigrationPlan ViewSet.", "parameters": [{"name": "data", "in": "body", "required": true, "schema": {"$ref": "#/definitions/pulp_2to3_migration.MigrationPlan"}}], "responses": {"201": {"description": "", "schema": {"$ref": "#/definitions/pulp_2to3_migration.MigrationPlan"}}}, "tags": ["migration-plans"]}, "parameters": []}, "{migration_plan_href}": {"get": {"operationId": "migration-plans_read", "summary": "Inspect a migration plan", "description": "MigrationPlan ViewSet.", "parameters": [{"name": "fields", "in": "query", "description": "A list of fields to include in the response.", "required": false, "type": "string"}, {"name": "exclude_fields", "in": "query", "description": "A list of fields to exclude from the response.", "required": false, "type": "string"}], "responses": {"200": {"description": "", "schema": {"$ref": "#/definitions/pulp_2to3_migration.MigrationPlan"}}}, "tags": ["migration-plans"]}, "delete": {"operationId": "migration-plans_delete", "summary": "Delete a migration plan", "description": "MigrationPlan ViewSet.", "parameters": [], "responses": {"204": {"description": ""}}, "tags": ["migration-plans"]}, "parameters": [{"name": "migration_plan_href", "in": "path", "description": "URI of Migration Plan. e.g.: /pulp/api/v3/migration-plans/1/", "required": true, "type": "string"}]}, "{migration_plan_href}run/": {"post": {"operationId": "migration-plans_run", "summary": "Run migration plan", "description": "Trigger an asynchronous task to run a migration from Pulp 2.", "parameters": [{"name": "data", "in": "body", "required": true, "schema": {"$ref": "#/definitions/MigrationPlanRun"}}], "responses": {"202": {"description": "", "schema": {"$ref": "#/definitions/AsyncOperationResponse"}}}, "tags": ["migration-plans"]}, "parameters": [{"name": "migration_plan_href", "in": "path", "description": "URI of Migration Plan. e.g.: /pulp/api/v3/migration-plans/1/", "required": true, "type": "string"}]}, "/pulp/api/v3/pulp2content/": {"get": {"operationId": "pulp2content_list", "summary": "List pulp2 contents", "description": "ViewSet for Pulp2Content model.", "parameters": [{"name": "pulp2_id", "in": "query", "description": "", "required": false, "type": "string"}, {"name": "pulp2_id__in", "in": "query", "description": "Filter results where pulp2_id is in a comma-separated list of values", "required": false, "type": "string"}, {"name": "pulp2_content_type_id", "in": "query", "description": "", "required": false, "type": "string"}, {"name": "pulp2_content_type_id__in", "in": "query", "description": "Filter results where pulp2_content_type_id is in a comma-separated list of values", "required": false, "type": "string"}, {"name": "pulp2_last_updated__lt", "in": "query", "description": "Filter results where pulp2_last_updated is less than value", "required": false, "type": "number"}, {"name": "pulp2_last_updated__lte", "in": "query", "description": "Filter results where pulp2_last_updated is less than or equal to value", "required": false, "type": "number"}, {"name": "pulp2_last_updated__gt", "in": "query", "description": "Filter results where pulp2_last_updated is greater than value", "required": false, "type": "number"}, {"name": "pulp2_last_updated__gte", "in": "query", "description": "Filter results where pulp2_last_updated is greater than or equal to value", "required": false, "type": "number"}, {"name": "pulp2_last_updated__range", "in": "query", "description": "Filter results where pulp2_last_updated is between two comma separated values", "required": false, "type": "number"}, {"name": "pulp3_content", "in": "query", "description": "Foreign Key referenced by HREF", "required": false, "type": "string"}, {"name": "pulp2_last_updated", "in": "query", "description": "ISO 8601 formatted dates are supported", "required": false, "type": "string"}, {"name": "limit", "in": "query", "description": "Number of results to return per page.", "required": false, "type": "integer"}, {"name": "offset", "in": "query", "description": "The initial index from which to return the results.", "required": false, "type": "integer"}, {"name": "fields", "in": "query", "description": "A list of fields to include in the response.", "required": false, "type": "string"}, {"name": "exclude_fields", "in": "query", "description": "A list of fields to exclude from the response.", "required": false, "type": "string"}], "responses": {"200": {"description": "", "schema": {"required": ["count", "results"], "type": "object", "properties": {"count": {"type": "integer"}, "next": {"type": "string", "format": "uri", "x-nullable": true}, "previous": {"type": "string", "format": "uri", "x-nullable": true}, "results": {"type": "array", "items": {"$ref": "#/definitions/pulp_2to3_migration.Pulp2Content"}}}}}}, "tags": ["pulp2content"]}, "parameters": []}, "{pulp2_content_href}": {"get": {"operationId": "pulp2content_read", "summary": "Inspect a pulp2 content", "description": "ViewSet for Pulp2Content model.", "parameters": [{"name": "fields", "in": "query", "description": "A list of fields to include in the response.", "required": false, "type": "string"}, {"name": "exclude_fields", "in": "query", "description": "A list of fields to exclude from the response.", "required": false, "type": "string"}], "responses": {"200": {"description": "", "schema": {"$ref": "#/definitions/pulp_2to3_migration.Pulp2Content"}}}, "tags": ["pulp2content"]}, "parameters": [{"name": "pulp2_content_href", "in": "path", "description": "URI of Pulp2 Content. e.g.: /pulp/api/v3/pulp2content/1/", "required": true, "type": "string"}]}, "/pulp/api/v3/pulp2repositories/": {"get": {"operationId": "pulp2repositories_list", "summary": "List pulp2 repositorys", "description": "ViewSet for Pulp2Repositories model.", "parameters": [{"name": "pulp2_repo_id", "in": "query", "description": "", "required": false, "type": "string"}, {"name": "pulp2_repo_id__in", "in": "query", "description": "Filter results where pulp2_repo_id is in a comma-separated list of values", "required": false, "type": "string"}, {"name": "is_migrated", "in": "query", "description": "", "required": false, "type": "string"}, {"name": "not_in_plan", "in": "query", "description": "", "required": false, "type": "string"}, {"name": "limit", "in": "query", "description": "Number of results to return per page.", "required": false, "type": "integer"}, {"name": "offset", "in": "query", "description": "The initial index from which to return the results.", "required": false, "type": "integer"}, {"name": "fields", "in": "query", "description": "A list of fields to include in the response.", "required": false, "type": "string"}, {"name": "exclude_fields", "in": "query", "description": "A list of fields to exclude from the response.", "required": false, "type": "string"}], "responses": {"200": {"description": "", "schema": {"required": ["count", "results"], "type": "object", "properties": {"count": {"type": "integer"}, "next": {"type": "string", "format": "uri", "x-nullable": true}, "previous": {"type": "string", "format": "uri", "x-nullable": true}, "results": {"type": "array", "items": {"$ref": "#/definitions/pulp_2to3_migration.Pulp2Repository"}}}}}}, "tags": ["pulp2repositories"]}, "parameters": []}, "{pulp2_repository_href}": {"get": {"operationId": "pulp2repositories_read", "summary": "Inspect a pulp2 repository", "description": "ViewSet for Pulp2Repositories model.", "parameters": [{"name": "fields", "in": "query", "description": "A list of fields to include in the response.", "required": false, "type": "string"}, {"name": "exclude_fields", "in": "query", "description": "A list of fields to exclude from the response.", "required": false, "type": "string"}], "responses": {"200": {"description": "", "schema": {"$ref": "#/definitions/pulp_2to3_migration.Pulp2Repository"}}}, "tags": ["pulp2repositories"]}, "parameters": [{"name": "pulp2_repository_href", "in": "path", "description": "URI of Pulp2 Repository. e.g.: /pulp/api/v3/pulp2repositories/1/", "required": true, "type": "string"}]}}, "definitions": {"pulp_2to3_migration.MigrationPlan": {"required": ["plan"], "type": "object", "properties": {"pulp_href": {"title": "Pulp href", "type": "string", "format": "uri", "readOnly": true}, "pulp_created": {"title": "Pulp created", "description": "Timestamp of creation.", "type": "string", "format": "date-time", "readOnly": true}, "plan": {"title": "Plan", "description": "Migration Plan in JSON format", "type": "object"}}}, "MigrationPlanRun": {"type": "object", "properties": {"validate": {"title": "Validate", "description": "If ``True``, migration cannot happen without successful validation of the Migration Plan", "type": "boolean", "default": false}, "dry_run": {"title": "Dry run", "description": "If ``True``, performs validation of a Migration Plan only, no migration is run.", "type": "boolean", "default": false}}}, "AsyncOperationResponse": {"required": ["task"], "type": "object", "properties": {"task": {"title": "Task", "description": "The href of the task.", "type": "string", "format": "uri"}}}, "pulp_2to3_migration.Pulp2Content": {"required": ["pulp2_id", "pulp2_content_type_id", "pulp2_last_updated", "pulp2_storage_path"], "type": "object", "properties": {"pulp_href": {"title": "Pulp href", "type": "string", "format": "uri", "readOnly": true}, "pulp_created": {"title": "Pulp created", "description": "Timestamp of creation.", "type": "string", "format": "date-time", "readOnly": true}, "pulp2_id": {"title": "Pulp2 id", "type": "string", "maxLength": 255, "minLength": 1}, "pulp2_content_type_id": {"title": "Pulp2 content type id", "type": "string", "maxLength": 255, "minLength": 1}, "pulp2_last_updated": {"title": "Pulp2 last updated", "type": "integer"}, "pulp2_storage_path": {"title": "Pulp2 storage path", "type": "string", "minLength": 1}, "downloaded": {"title": "Downloaded", "type": "boolean", "default": false}, "pulp3_content": {"title": "Pulp3 content", "type": "string", "format": "uri", "x-nullable": true}}}, "pulp_2to3_migration.Pulp2Repository": {"required": ["pulp2_object_id", "pulp2_repo_id", "pulp2_repo_type"], "type": "object", "properties": {"pulp_href": {"title": "Pulp href", "type": "string", "format": "uri", "readOnly": true}, "pulp_created": {"title": "Pulp created", "description": "Timestamp of creation.", "type": "string", "format": "date-time", "readOnly": true}, "pulp2_object_id": {"title": "Pulp2 object id", "type": "string", "maxLength": 255, "minLength": 1}, "pulp2_repo_id": {"title": "Pulp2 repo id", "type": "string", "minLength": 1}, "pulp2_repo_type": {"title": "Pulp2 repo type", "type": "string", "minLength": 1}, "is_migrated": {"title": "Is migrated", "type": "boolean", "default": false}, "not_in_plan": {"title": "Not in plan", "type": "boolean", "default": false}, "pulp3_repository_version": {"title": "Pulp3 repository version", "description": "RepositoryVersion to be served", "type": "string", "format": "uri", "x-nullable": true}, "pulp3_remote_href": {"title": "Pulp3 remote href", "description": "\nGet pulp3_remote_href from pulp2repo\n", "type": "string", "readOnly": true, "minLength": 1}, "pulp3_publication_href": {"title": "Pulp3 publication href", "description": "\nGet pulp3_publication_href from pulp3_repository_version\n", "type": "string", "readOnly": true, "minLength": 1}, "pulp3_distribution_hrefs": {"description": "\nGet pulp3_distribution_hrefs from pulp3_repository_version\n", "type": "array", "items": {"type": "string", "minLength": 1}, "readOnly": true}, "pulp3_repository_href": {"title": "Pulp3 repository href", "description": "\nGet pulp3_repository_href from pulp2repo\n", "type": "string", "readOnly": true, "minLength": 1}}}}, "tags": [{"name": "migration-plans", "x-displayName": "Migration-Plans"}, {"name": "pulp2content", "x-displayName": "Pulp2Content"}, {"name": "pulp2repositories", "x-displayName": "Pulp2Repositories"}]} \ No newline at end of file diff --git a/docs/_templates/restapi.html b/docs/_templates/restapi.html index 58289356..6d5f9360 100644 --- a/docs/_templates/restapi.html +++ b/docs/_templates/restapi.html @@ -18,7 +18,7 @@ - + diff --git a/docs/conf.py b/docs/conf.py index ec235771..35e2b40e 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -18,6 +18,11 @@ import pulp_2to3_migration +try: + import sphinx_rtd_theme +except ImportError: + sphinx_rtd_theme = False + # -- Project information ----------------------------------------------------- diff --git a/docs/configuration.rst b/docs/configuration.rst new file mode 100644 index 00000000..3e47903b --- /dev/null +++ b/docs/configuration.rst @@ -0,0 +1,46 @@ +Configuration +============= + +Requirements +------------ + +* /var/lib/pulp is shared from Pulp 2 machine +* access to Pulp 2 database + + +Configuration +------------- + +On Pulp 2 machine +***************** + +1. Make sure MongoDB listens on the IP address accesible outside, it should be configured as +one of the ``bindIP``s in ``/etc/mongod.conf``. + +2. Make sure ``/var/lib/pulp`` is on a shared filesystem. + + +On Pulp 3 machine +***************** +1. Mount ``/var/lib/pulp`` to your Pulp 3 storage location. By default, it's ``/var/lib/pulp``. + +2. Configure your connection to MongoDB in your settings, ``/etc/pulp/settings.py``. You can use +the same configuration as you have in Pulp 2 (only seeds might need to be different, it depends +on your setup). By default it's configured to connect to ``localhost:27017``. + +E.g. + +.. code:: python + + PULP2_MONGODB = { + 'name': 'pulp_database', + 'seeds': ':27017', + 'username': '', + 'password': '', + 'replica_set': '', + 'ssl': False, + 'ssl_keyfile': '', + 'ssl_certfile': '', + 'verify_ssl': True, + 'ca_path': '/etc/pki/tls/certs/ca-bundle.crt', + } diff --git a/docs/index.rst b/docs/index.rst index 8b6fa63a..9b7eb3ee 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,16 +1,29 @@ -.. pulp-2to3-migration documentation master file, created by - sphinx-quickstart on Wed Sep 25 16:20:06 2019. - You can adapt this file completely to your liking, but it should at least - contain the root `toctree` directive. -Welcome to pulp-2to3-migration's documentation! -=============================================== +Migration Plugin +================ + +The ``pulp_2to3_migration`` plugin extends `pulpcore `__ +to support migration from Pulp 2 to Pulp 3. + +Supported plugins: + - Pulp 2 ISO can be migrated into Pulp 3 File. + - Pulp 2 Docker can be migrated into Pulp 3 Container. + - RPM plugin migration is planned and currently in development. + + +Table of Contents +----------------- .. toctree:: - :maxdepth: 2 + :maxdepth: 1 + installation + configuration + migration_plan + workflows + restapi changes - + plugin_writers_guide Indices and tables diff --git a/docs/installation.rst b/docs/installation.rst new file mode 100644 index 00000000..d603a3f3 --- /dev/null +++ b/docs/installation.rst @@ -0,0 +1,59 @@ +User Setup +========== + +Ansible Installer (Recommended) +------------------------------- + +We recommend that you install `pulpcore`, all the content plugins you need and +`pulp-2to3-migration` plugin together using the `Ansible installer +`_. The remaining steps are all +performed by the installer and are not needed if you use it. + +Pip Install +----------- + +This document assumes that you have +`installed pulpcore `_ +and any content plugins you need into the virtual environment ``pulpvenv``. + +Users should install from **either** PyPI or source. + +From PyPI +********* + +.. code-block:: bash + + sudo -u pulp -i + source ~/pulpvenv/bin/activate + pip install pulp-2to3-migration + +From Source +*********** + +.. code-block:: bash + + sudo -u pulp -i + source ~/pulpvenv/bin/activate + git clone https://github.com/pulp/pulp-2to3-migration.git + cd pulp-2to3-migration + pip install -e . + +Make and Run Migrations +----------------------- + +.. code-block:: bash + + export DJANGO_SETTINGS_MODULE=pulpcore.app.settings + django-admin makemigrations pulp_2to3_migration + django-admin migrate pulp_2to3_migration + +Run Services +------------ + +.. code-block:: bash + + django-admin runserver 24817 + gunicorn pulpcore.content:server --bind 'localhost:24816' --worker-class 'aiohttp.GunicornWebWorker' -w 2 + sudo systemctl restart pulpcore-resource-manager + sudo systemctl restart pulpcore-worker@1 + sudo systemctl restart pulpcore-worker@2 diff --git a/docs/migration_plan.rst b/docs/migration_plan.rst new file mode 100644 index 00000000..2d79b49f --- /dev/null +++ b/docs/migration_plan.rst @@ -0,0 +1,58 @@ +Migration Plan +-------------- + +To configure what to migrate to Pulp 3, one needs to define a Migration Plan (MP). +A MP defines which plugins to migrate and how. +Migration plugin is declarative, fully migrated content and repositories in Pulp 3 will +correspond to the most recent MP which has been run. + +.. note:: + It is possible to have orphaned content in Pulp 3 after multiple migration re-runs. After the + migration is fully complete, run orphan cleanup task in Pulp 3 to remove content which is not a + part of any repository version. + +The type of a plugin specified in a MP is a Pulp 2 plugin. E.g. one needs to specify ``iso`` to +migrate Pulp 2 ISO content and repositories into Pulp 3 File plugin. + +Examples of the migration plan: + +* Migrate all for a specific plugin + +.. code:: json + + { + "plugins": [ + { + "type": "iso" + } + ] + } + + +* Migrate a pulp 2 repository ``file`` (into a Pulp 3 repository with one repository version), + using an importer from a pulp 2 repository ``file2``, and distributors from a repository ``file`` + and ``file2`` + +.. code:: json + + { + "plugins": [ + { + "type": "iso", + "repositories": [ + { + "name": "file", + "repository_versions": [ + { + "pulp2_repository_id": "file", + "pulp2_distributor_repository_ids": [ + "file", "file2" + ] + } + ], + "pulp2_importer_repository_id": "file2" + } + ] + } + ] + } diff --git a/docs/plugin_writers_guide.rst b/docs/plugin_writers_guide.rst new file mode 100644 index 00000000..6e11ab6a --- /dev/null +++ b/docs/plugin_writers_guide.rst @@ -0,0 +1,43 @@ +Plugin Writer's Guide +===================== + +If you are extending this migration tool to be able to migrate the plugin of your interest +from Pulp 2 to Pulp 3, here are some guidelines. + + +1. Create a migrator class (subclass the provided ``Pulp2to3PluginMigrator`` class). There should be +one migrator class per plugin. Define all the necessary attributes and methods for it (see +``Pulp2to3PluginMigrator`` for more details) + +2. Discovery of the plugins is done via entry_points. Add your migrator class defined in step 1 +to the list of the ``migrators`` entry_points in ``setup.py``. + +3. Add a Content model to communicate with Pulp 2. + +* It has to have a field ``TYPE_ID`` which will correspond to the ``_content_type_id`` of your + Content in Pulp 2. Don't forget to add it to ``pulp2_content_models`` in step 1. + +4. Add a Content model to pre-migrate Pulp2 content to (subclass the provided ``Pulp2to3Content`` +class). It has to have: + +* a field ``pulp2_type`` which will correspond to the ``_content_type_id`` of your Content in Pulp 2. +* on a Meta class a ``default_related_name`` set to ``_detail_model`` +* a classmethod ``pre_migrate_content_detail`` (see ``Pulp2to3Content`` for more details) +* a coroutine ``create_pulp3_content`` (see ``Pulp2to3Content`` for more details) + +Don't forget to add this Content model to your migrator class. + +If your content has one artifact and if you are willing to use the default implementation of the +first stage of DeclarativeContentMigration, on your Content model you also need: + +* an ``expected_digests`` property to provide expected digests for artifact creation/validation +* an ``expected_size`` property to provide the expected size for artifact creation/validation +* a ``relative_path_for_content_artifact`` property to provide the relative path for content + artifact creation. + +5. Subclass the provided ``Pulp2to3Importer`` class and define ``migrate_to_pulp3`` method which +creates a plugin Remote instance based on the provided pre-migrated ``Pulp2Importer``. + +6. Subclass the provided ``Pulp2to3Distributor`` class and define ``migrate_to_pulp3`` method which +creates a plugin Publication and/or Distribution instance (depends on the plugin) based on the +provided pre-migrated ``Pulp2Distributor``. \ No newline at end of file diff --git a/docs/restapi.rst b/docs/restapi.rst new file mode 100644 index 00000000..4a6eb9b4 --- /dev/null +++ b/docs/restapi.rst @@ -0,0 +1,9 @@ +REST API +======== + +Pulpcore Reference: `pulpcore API documentation `_. + +Pulp 2to3 Migration API +----------------------- + +See the `pulp_2to3_migration API documentation <../restapi.html>`_ \ No newline at end of file diff --git a/docs/workflows.rst b/docs/workflows.rst new file mode 100644 index 00000000..6e444802 --- /dev/null +++ b/docs/workflows.rst @@ -0,0 +1,59 @@ +Workflow +======== + +All the commands should be run on Pulp 3 machine. + +1. Create a :doc:`Migration Plan <../migration_plan>` + +.. code:: bash + + $ # migrate content for Pulp 2 ISO plugin + $ http POST :24817/pulp/api/v3/migration-plans/ plan='{"plugins": [{"type": "iso"}]}' + + HTTP/1.1 201 Created + { + "pulp_created": "2019-07-23T08:18:12.927007Z", + "pulp_href": "/pulp/api/v3/migration-plans/59f8a786-c7d7-4e2b-ad07-701479d403c5/", + "plan": "{ "plugins": [{"type": "iso"}]}" + } + + +2. Use the ``pulp_href`` of the created Migration Plan to run the migration + + +.. code:: bash + + $ http POST :24817/pulp/api/v3/migration-plans/59f8a786-c7d7-4e2b-ad07-701479d403c5/run/ + + HTTP/1.1 202 Accepted + { + "task": "/pulp/api/v3/tasks/55db2086-cf2e-438f-b5b7-cd0dbb7c8cf4/" + } + + +3. For listing the mapping for Pulp 2 to Pulp 3 + +.. code:: bash + + $ http :24817/pulp/api/v3/pulp2repositories/ + + HTTP/1.1 200 OK + { + "count": 1, + "next": null, + "previous": null, + "results": [ + { + "is_migrated": true, + "not_in_plan": false, + "pulp2_object_id": "5dbc478c472f68283ad8e6bd", + "pulp2_repo_id": "file-large", + "pulp3_distribution_hrefs": [], + "pulp3_publication_href": [], + "pulp3_remote_href": "/pulp/api/v3/remotes/file/file/ca0e505e-51c2-46e1-be40-3762d372f9b2/", + "pulp3_repository_version": null, + "pulp_created": "2019-11-01T14:59:04.648920Z", + "pulp_href": "/pulp/api/v3/pulp2repositories/92c6d1c8-718b-4ea9-8a23-b2386849c2c5/" + } + ] + }