Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip zchunk metadata at sync time #1337

Merged
merged 1 commit into from May 13, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 11 additions & 1 deletion docs/tech-reference/rpm.rst
Expand Up @@ -122,7 +122,17 @@ sadly a dead link) contains ``data`` elements with one attribute, ``type``. The
``data`` element refers to. Common values are ``group``, ``filelists``, ``group_gz``,
``primary``, ``other``, ``filelists_db``, ``primary_db``, and ``other_db``. ``<thing>_db``
refer to SQLite versions of the metadata, while those sans ``_db`` refer to XML
versions. Each ``data`` element contains several other elements describing the
versions. Each metadata can be compressed, often the format of compression will be present in the
``type`` attribute: ``<thing>_gz`` or ``<thing>_zck`` mean that metadata was compressed using
gzip or zchunk format respectively.

.. note::
Metadata compressed into `Zchunk format <https://fedoraproject
.org/wiki/Changes/Zchunk_Metadata>`_ is not synced and ignored by Pulp to avoid creating a
broken repository at publish time. Zchunk metadata is optional, metadata in other formats is
present as well, so all repositories are synced properly.

Each ``data`` element contains several other elements describing the
metadata and where it is located: ``checksum``, ``location``, ``timestamp``, and
``size`` seem to be always present, with ``open-size``, ``open-checksum``, and
``database_version`` potentially appearing as well.
Expand Down
17 changes: 16 additions & 1 deletion plugins/pulp_rpm/plugins/importers/yum/sync.py
Expand Up @@ -547,11 +547,26 @@ def import_unknown_metadata_files(self, metadata_files):
Import metadata files whose type is not known to us. These are any files
that we are not already parsing.

Skip those which we don't support yet and which can break repo if not supported fully.

:param metadata_files: object containing access to all metadata files
:type metadata_files: pulp_rpm.plugins.importers.yum.repomd.metadata.MetadataFiles
"""
def is_needed(metadata_type):
"""
Decide if metadata should be processed or should be skipped.

Skip zchunk metadata.

:param metadata_type: type of metadata
:type metadata_type: str
:return: True, if metadata should be processed, otherwise it is skipped
:rtype: bool
"""
return not metadata_type.endswith('_zck')

for metadata_type, file_info in metadata_files.metadata.iteritems():
if metadata_type not in metadata_files.KNOWN_TYPES:
if metadata_type not in metadata_files.KNOWN_TYPES and is_needed(metadata_type):
file_path = file_info['local_path']
checksum_type = file_info['checksum']['algorithm']
checksum_type = util.sanitize_checksum_type(checksum_type)
Expand Down