Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PR #2497/962d28ad backport][3.17] Loosened import version-restriction to x.y instead of x.y.z. #2524

Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGES/2269.bugfix
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Loosened the version-restrictions on PulpImport to only require X.Y matching.
65 changes: 49 additions & 16 deletions docs/workflows/import-export.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,22 +37,6 @@ In order to minimize space utilization, import/export operates on sets of
:term:`Artifacts<Artifact>` only once per-export, rather than once for each
:term:`Repository` being exported.

.. note::

Export will not operate on :term:`RepositoryVersions<RepositoryVersion>` that have
been synchronized using ``policy=on_demand``. :term:`Artifacts<Artifact>` must actually
exist in order to be exported - this is, after all the only way for the Downstream Pulp
instance to gain access to them!

.. note::

Import and Export strictly control which directories may be read from/written to via
the settings options ``ALLOWED_IMPORT_PATHS`` and ``ALLOWED_EXPORT_PATHS``.
These default to empty, if not explicitly set attempts to import or export will fail
with a validation error like

``"Path '/tmp/exports/' is not an allowed export path"``

Definitions
^^^^^^^^^^^
Upstream
Expand Down Expand Up @@ -87,6 +71,55 @@ Import order
ModelResources be imported in order. Plugins are responsible for specifying the
import-order of the ModelResources they own

Assumptions
^^^^^^^^^^^

The import/export workflow operates on a set of assumptions. Violating them will result
in error-messages as described below.

On-Demand content not supported
-------------------------------

Export will not operate on :term:`RepositoryVersions<RepositoryVersion>` that have
been synchronized using ``policy=on_demand`` or ``policy=streamed``. :term:`Artifacts<Artifact>`
must actually exist in order to be exported - this is, after
all the only way for the Downstream Pulp instance to gain access to them!

If a repository is specified for export that utilized on-demand/streamed syncing, the
export will fail with a RuntimeError:

``Remote artifacts cannot be exported.``

Export/Import Directories must be explicitly allowed
----------------------------------------------------

Import and Export strictly control which directories may be read from/written to via
the settings options ``ALLOWED_IMPORT_PATHS`` and ``ALLOWED_EXPORT_PATHS``.
These default to empty - if they not explicitly set, attempts to import or export will fail
with a validation error like

``"Path '/tmp/exports/' is not an allowed export path"``

Installed plugins must match
----------------------------

A Downstream must support the complete set of plugins present in a given export. If the
export includes plugins that are not installed in the Downstream, an import attempt will
fail with a validation error like

``Export uses pulp_rpm which is not installed.``

Version-compatibility required
------------------------------

The export-to-import workflow is built on the assumption that the Upstream and
Downstream instances are running "compatible" versions of pulpcore and plugins. In this
context, "compatible" is defined as **"share the same X.Y version"**. If this is not the
case, an import attempt will fail with a validation error like

``Export version 3.14.15 of pulpcore incompatible with installed version 3.16.3.``


Exporting
^^^^^^^^^

Expand Down
18 changes: 13 additions & 5 deletions pulpcore/app/tasks/importer.py
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,12 @@ def _import_file(fpath, resource_class, retry=False):


def _check_versions(version_json):
"""Compare the export version_json to the installed components."""
"""
Compare the export version_json to the installed components.

An upstream whose db-metadata doesn't match the downstream won't import successfully; check
for compatibility and raise a ValidationError if incompatible versions are found.
"""
error_messages = []
for component in version_json:
try:
Expand All @@ -101,10 +106,13 @@ def _check_versions(version_json):
_("Export uses {} which is not installed.").format(component["component"])
)
else:
if version != component["version"]:
# Check that versions are compatible. Currently, "compatible" is defined as "same X.Y".
# Versions are strings that generally look like "X.Y.Z" or "X.Y.Z.dev"; we check that
# first two places are the same.
if version.split(".")[:2] != component["version"].split(".")[:2]:
error_messages.append(
_(
"Export version {export_ver} of {component} does not match "
"Export version {export_ver} of {component} incompatible with "
"installed version {ver}."
).format(
export_ver=component["version"],
Expand All @@ -113,8 +121,8 @@ def _check_versions(version_json):
)
)

if error_messages:
raise ValidationError((" ".join(error_messages)))
if error_messages:
raise ValidationError((" ".join(error_messages)))


def import_repository_version(importer_pk, destination_repo_pk, source_repo_name, tar_path):
Expand Down
28 changes: 28 additions & 0 deletions pulpcore/tests/unit/test_import_checks.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
from unittest.mock import patch

from django.test import TestCase
from rest_framework.serializers import ValidationError

from pulpcore.app.tasks.importer import _check_versions


class TestObject:
version = "1.2.3" # Every component is vers 1.2.3


class TestCheckVersions(TestCase):
@patch("pulpcore.app.tasks.importer.get_distribution", return_value=TestObject())
def test_vers_check(self, mock_get_distribution):
export_json = [{"component": "xyz", "version": "1.2.3"}]
_check_versions(export_json)

export_json = [{"component": "xy", "version": "1.2"}]
_check_versions(export_json)

export_json = [{"component": "x_noty_z", "version": "1.4.3"}]
with self.assertRaises(ValidationError):
_check_versions(export_json)

export_json = [{"component": "notx_y_z", "version": "2.2.3"}]
with self.assertRaises(ValidationError):
_check_versions(export_json)