Issue when processing RHEL10.1 EPEL mirrors (and some others?)

[glenmortt]

My setup:
OS: Ubuntu 26.04
Patchman release: 4.0.19
DB Backend: Postgres

The report it is failing to process epel mirrors. Also got the same issue with docker repo for rhel 10.1.

2 Mirrors already exist (max=2), not adding more
Fetching rpm Repo data             : 100%|=======================================================================================================================================================| 4064/4064 [00:00<00:00, 32222403.51it/s]
Found yum rpm Repo - http://centos.anexia.at/epel/10.1/Everything/x86_64/repodata/repomd.xml
Fetching Package data              : 100%|==================================================================================================================================================| 3763432/3763432 [00:01<00:00, 3142376.61it/s]
Mirror Packages checksum has not changed, skipping Package refresh
No Module metadata found in http://centos.anexia.at/epel/10.1/Everything/x86_64/repodata/repomd.xml
Fetching Errata data               : 100%|==================================================================================================================================================| 2147139/2147139 [00:00<00:00, 2268359.82it/s]
Mirror Errata checksum has not changed, skipping Erratum refresh
Fetching rpm Repo data             : 100%|=======================================================================================================================================================| 4064/4064 [00:00<00:00, 30547762.47it/s]
Found yum rpm Repo - https://centos.anexia.at/epel/10.1/Everything/x86_64/repodata/repomd.xml
Fetching Package data              : 100%|==================================================================================================================================================| 3763432/3763432 [00:01<00:00, 2220139.07it/s]
Mirror Packages checksum has not changed, skipping Package refresh
No Module metadata found in https://centos.anexia.at/epel/10.1/Everything/x86_64/repodata/repomd.xml
Fetching Errata data               : 100%|==================================================================================================================================================| 2147139/2147139 [00:00<00:00, 3588248.46it/s]
Traceback (most recent call last):
  File "/usr/bin/patchman", line 521, in <module>
    main()
    ~~~~^^
  File "/usr/bin/patchman", line 515, in main
    showhelp = process_args(args)
  File "/usr/bin/patchman", line 471, in process_args
    refresh_repos(args.repo, args.force)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/patchman", line 126, in refresh_repos
    repo.refresh(force)
    ~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3/dist-packages/repos/models.py", line 95, in refresh
    refresh_rpm_repo(self)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/rpm.py", line 40, in refresh_rpm_repo
    refresh_rpm_repo_mirrors(repo)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/rpm.py", line 98, in refresh_rpm_repo_mirrors
    refresh_yum_repo(mirror, repo_data, mirror_url, errata_only)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/yum.py", line 257, in refresh_yum_repo
    refresh_repomd_updateinfo(mirror, data, mirror_url)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/yum.py", line 189, in refresh_repomd_updateinfo
    extract_updateinfo(data, url)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/errata/sources/repos/yum.py", line 35, in extract_updateinfo
    extracted = extract(data, url)
  File "/usr/lib/python3/dist-packages/util/__init__.py", line 216, in extract
    return unzstd(data)
  File "/usr/lib/python3/dist-packages/util/__init__.py", line 197, in unzstd
    zstddata = zstd.ZstdDecompressor().stream_reader(contents).read()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'compression.zstd.ZstdDecompressor' object has no attribute 'stream_reader'

Found yum rpm Repo - https://download.docker.com/linux/centos/10/x86_64/stable/repodata/repomd.xml
Fetching Package data              : 100%|=====================================================================================================================================================| 20857/20857 [00:00<00:00, 30748892.28it/s]
Mirror Packages checksum has not changed, skipping Package refresh
No Module metadata found in https://download.docker.com/linux/centos/10/x86_64/stable/repodata/repomd.xml
Fetching Errata data               : 100%|=============================================================================================================================================================| 51/51 [00:00<00:00, 386816.46it/s]
Traceback (most recent call last):
  File "/usr/bin/patchman", line 521, in <module>
    main()
    ~~~~^^
  File "/usr/bin/patchman", line 515, in main
    showhelp = process_args(args)
  File "/usr/bin/patchman", line 471, in process_args
    refresh_repos(args.repo, args.force)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/patchman", line 126, in refresh_repos
    repo.refresh(force)
    ~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3/dist-packages/repos/models.py", line 95, in refresh
    refresh_rpm_repo(self)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/rpm.py", line 40, in refresh_rpm_repo
    refresh_rpm_repo_mirrors(repo)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/rpm.py", line 98, in refresh_rpm_repo_mirrors
    refresh_yum_repo(mirror, repo_data, mirror_url, errata_only)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/yum.py", line 257, in refresh_yum_repo
    refresh_repomd_updateinfo(mirror, data, mirror_url)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/yum.py", line 189, in refresh_repomd_updateinfo
    extract_updateinfo(data, url)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/errata/sources/repos/yum.py", line 35, in extract_updateinfo
    extracted = extract(data, url)
  File "/usr/lib/python3/dist-packages/util/__init__.py", line 216, in extract
    return unzstd(data)
  File "/usr/lib/python3/dist-packages/util/__init__.py", line 197, in unzstd
    zstddata = zstd.ZstdDecompressor().stream_reader(contents).read()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'compression.zstd.ZstdDecompressor' object has no attribute 'stream_reader'

[furlongm]

Thanks for the bug report!

Can you check if these packages fix it for you: https://github.com/furlongm/patchman/releases/tag/v4.0.20-dev1

[glenmortt]

Hey, thanks! It has fixed the original issue. However, got another one :) Not sure whether to open another issue. At least it triggers only one repo:

Repository 132 : Ubuntu 22.04 x86_64 repo at https://packages.openproject.com/srv/deb/opf/openproject/stable/17/ubuntu/dists/22.04/main/binary-amd64
Found deb Repo - https://packages.openproject.com/srv/deb/opf/openproject/stable/17/ubuntu/dists/22.04/main/binary-amd64/Packages.gz
Fetching Debian Repo data
gunzip: Error -3 while decompressing data: incorrect header check
Traceback (most recent call last):
  File "/usr/bin/patchman", line 521, in <module>
    main()
    ~~~~^^
  File "/usr/bin/patchman", line 515, in main
    showhelp = process_args(args)
  File "/usr/bin/patchman", line 471, in process_args
    refresh_repos(args.repo, args.force)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/patchman", line 126, in refresh_repos
    repo.refresh(force)
    ~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3/dist-packages/repos/models.py", line 93, in refresh
    refresh_deb_repo(self)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/deb.py", line 110, in refresh_deb_repo
    packages = extract_deb_packages(package_data, mirror_url)
  File "/usr/lib/python3/dist-packages/repos/repo_types/deb.py", line 35, in extract_deb_packages
    extracted = extract(data, url).decode('utf-8')
                ^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'decode'

Btw, here's the upgrade output:

# dpkg -i python3-patchman_4.0.20-dev1-1_all.deb 
(Reading database ... 156702 files and directories currently installed.)
Preparing to unpack python3-patchman_4.0.20-dev1-1_all.deb ...
Unpacking python3-patchman (4.0.20-dev1-1) over (4.0.19-1) ...
Setting up python3-patchman (4.0.20-dev1-1) ...
apache2_invoke patchman.conf: no action - configuration was disabled by local admin
System check identified some issues:

WARNINGS:
?: (staticfiles.W004) The directory '/usr/lib/python3/dist-packages/static' in the STATICFILES_DIRS setting does not exist.

230 static files copied to '/usr/lib/python3/dist-packages/run/static'.
System check identified some issues:

WARNINGS:
?: (staticfiles.W004) The directory '/usr/lib/python3/dist-packages/static' in the STATICFILES_DIRS setting does not exist.
Operations to perform:
  Synchronize unmigrated apps: admindocs, bootstrap3, celery, django_extensions, django_filters, django_select2, django_tables2, humanize, messages, rest_framework, staticfiles, util
  Apply all migrations: admin, arch, auth, contenttypes, django_celery_beat, domains, errata, hosts, modules, operatingsystems, packages, reports, repos, rest_framework_api_key, security, sessions, sites, taggit
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
  No migrations to apply.
  Your models in app(s): 'reports' have changes that are not yet reflected in a migration, and so won't be applied.
  Run 'manage.py makemigrations' to make new migrations, and then re-run 'manage.py migrate' to apply them.
wal
warn: The user `www-data' is already a member of `patchman'.
/usr/bin/deb-systemd-helper: error: unable to read patchman-celery-worker@1.service

Remember to run 'patchman-manage createsuperuser' to create a user.

So I've done patchman-manage makemigrations and patchman-manage migrate just in case.

[glenmortt]

Uhh, I've just did patchmon -a once again and got:

2 Mirrors already exist (max=2), not adding more
Fetching rpm Repo data             : 100%|=======================================================================================================================================================| 4064/4064 [00:00<00:00, 41372940.43it/s]
Found yum rpm Repo - http://epel.mirror.omnilance.com/10.1/Everything/x86_64/repodata/repomd.xml
Fetching Package data              : 100%|=================================================================================================================================================| 3765450/3765450 [00:00<00:00, 42502218.83it/s]
Extracting 20777 Packages          : 100%|=========================================================================================================================================================| 20777/20777 [00:02<00:00, 8200.09it/s]
Adding 20777 new Packages          : 100%|==========================================================================================================================================================| 20777/20777 [01:26<00:00, 240.26it/s]
No Module metadata found in http://epel.mirror.omnilance.com/10.1/Everything/x86_64/repodata/repomd.xml
Fetching Errata data               : 100%|=================================================================================================================================================| 2148633/2148633 [00:00<00:00, 37443058.53it/s]
Extracting 4142 updateinfo Errata  :   0%|                                                                                                                                                              | 1/4142 [00:02<2:27:05,  2.13s/it]concurrent.futures.process._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/errata/utils.py", line 28, in get_or_create_erratum
    e = Erratum.objects.get(name=name)
  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 635, in get
    raise self.model.DoesNotExist(
        "%s matching query does not exist." % self.model._meta.object_name
    )
errata.models.Erratum.DoesNotExist: Erratum matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 948, in get_or_create
    return self.get(**kwargs), False
           ~~~~~~~~^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 635, in get
    raise self.model.DoesNotExist(
        "%s matching query does not exist." % self.model._meta.object_name
    )
errata.models.Erratum.DoesNotExist: Erratum matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
psycopg2.errors.StringDataRightTruncation: value too long for type character varying(255)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.14/concurrent/futures/process.py", line 254, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/lib/python3/dist-packages/errata/sources/repos/yum.py", line 68, in process_updateinfo_erratum_wrapper
    return process_updateinfo_erratum(update)
  File "/usr/lib/python3/dist-packages/errata/sources/repos/yum.py", line 80, in process_updateinfo_erratum
    e, created = get_or_create_erratum(name, e_type, issue_date, synopsis)
                 ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/errata/utils.py", line 52, in get_or_create_erratum
    e, created = Erratum.objects.get_or_create(
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        name=name,
        ^^^^^^^^^^
    ...<2 lines>...
        synopsis=synopsis,
        ^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 955, in get_or_create
    return self.create(**params), True
           ~~~~~~~~~~~^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 665, in create
    obj.save(force_insert=True, using=self.db)
    ~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 902, in save
    self.save_base(
    ~~~~~~~~~~~~~~^
        using=using,
        ^^^^^^^^^^^^
    ...<2 lines>...
        update_fields=update_fields,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1008, in save_base
    updated = self._save_table(
        raw,
    ...<4 lines>...
        update_fields,
    )
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1169, in _save_table
    results = self._do_insert(
        cls._base_manager, using, fields, returning_fields, raw
    )
  File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1210, in _do_insert
    return manager._insert(
           ~~~~~~~~~~~~~~~^
        [self],
        ^^^^^^^
    ...<3 lines>...
        raw=raw,
        ^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/query.py", line 1873, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/models/sql/compiler.py", line 1882, in execute_sql
    cursor.execute(sql, params)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        sql, params, many=False, executor=self._execute
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/lib/python3/dist-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
django.db.utils.DataError: value too long for type character varying(255)

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/patchman", line 521, in <module>
    main()
    ~~~~^^
  File "/usr/bin/patchman", line 515, in main
    showhelp = process_args(args)
  File "/usr/bin/patchman", line 471, in process_args
    refresh_repos(args.repo, args.force)
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/bin/patchman", line 126, in refresh_repos
    repo.refresh(force)
    ~~~~~~~~~~~~^^^^^^^
  File "/usr/lib/python3/dist-packages/repos/models.py", line 95, in refresh
    refresh_rpm_repo(self)
    ~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/rpm.py", line 40, in refresh_rpm_repo
    refresh_rpm_repo_mirrors(repo)
    ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/rpm.py", line 98, in refresh_rpm_repo_mirrors
    refresh_yum_repo(mirror, repo_data, mirror_url, errata_only)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/yum.py", line 257, in refresh_yum_repo
    refresh_repomd_updateinfo(mirror, data, mirror_url)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/repos/repo_types/yum.py", line 189, in refresh_repomd_updateinfo
    extract_updateinfo(data, url)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/errata/sources/repos/yum.py", line 44, in extract_updateinfo
    extract_updateinfo_concurrently(updates, elen, max_workers)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/errata/sources/repos/yum.py", line 62, in extract_updateinfo_concurrently
    for i, _ in enumerate(run_concurrently(process_updateinfo_erratum_wrapper, updates, max_workers)):
                ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/util/__init__.py", line 341, in run_concurrently
    yield future.result()
          ~~~~~~~~~~~~~^^
  File "/usr/lib/python3.14/concurrent/futures/_base.py", line 443, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3.14/concurrent/futures/_base.py", line 395, in __get_result
    raise self._exception
django.db.utils.DataError: value too long for type character varying(255)

[furlongm]

Try https://github.com/furlongm/patchman/releases/tag/v4.0.20-dev3 and let me know if it fixes those issues. You may need a migration first.

For the other warning you have, curious if you manually set STATICFILES_DIRS?