assets: check for what leapp version are assets meant for #1003

MichalHe · 2022-12-23T21:36:20Z

Introduction of versions to leapp data assets

This patch introduces a mechanism allowing specifying that certain versions of leapp data assets are meant for a particular version of leapp. The motivation is being able to pair assets with a corresponding leapp version that passed QA verification.
To provide this functionality, every asset file was extended with a new JSON field "provided_data_assets": ["X.Y", "XX.YY"]. Additional logic checking assets was also added accordingly (load_asset). Assets with invalid versions are reported together.

Ref: OAMG-7989

github-actions · 2022-12-23T21:36:35Z

Thank you for contributing to the Leapp project!

Please note that every PR needs to comply with the Leapp Guidelines and must pass all tests in order to be mergeable.
If you want to request a review or rebuild a package in copr, you can use following commands as a comment:

review please to notify leapp developers of review request
/packit copr-build to submit a public copr build using packit

To launch regression testing public members of oamg organization can leave the following comment:

/rerun to schedule basic regression tests using this pr build and leapp*master* as artifacts
/rerun 42 to schedule basic regression tests using this pr build and leapp*PR42* as artifacts
/rerun-sst to schedule sst tests using this pr build and leapp*master* as artifacts
/rerun-sst 42 to schedule sst tests using this pr build and leapp*PR42* as artifacts

Please open ticket in case you experience technical problem with the CI. (RH internal only)

Note: In case there are problems with tests not being triggered automatically on new PR/commit or pending for a long time, please consider rerunning the CI by commenting leapp-ci build (might require several comments). If the problem persists, contact leapp-infra.

MichalHe · 2022-12-31T20:55:48Z

Conducted testing (rhel8 dev vagrant box)

All assets have a version field, that matches the current leapp version.
A single asset does not have a version field
- Inhibitor: Repositories mapping data is outdated
A single asset is outdated ("leapp_version": "0.16.0", the installed leapp is 0.17.0)
- Inhibitor: Repositories mapping is outdated
A single asset has a newer version than the current leapp ("leapp_version": "0.18.0", the installed leapp is 0.17.0)
- Inhibitor: PES events file is intended for a newer leapp
All assets are missing, and thus, they need to be fetched from the online service (service serves assets with correct versions). A mock server running on the testing machine on port 8888 was set up. The server offered only one endpoint template @app.route('/api/pes/<leapp_version>/<data_file>'). Leapp was executed as follows: LEAPP_SERVICE_HOST=http://0.0.0.0:8888 leapp preupgrade --debug. The following endpoints wwere were probed by leapp:

        GET /api/pes/0.17.0/repomap.json HTTP/1.1
        GET /api/pes/0.17.0/pes-events.json HTTP/1.1
	GET /api/pes/0.17.0/device_driver_deprecation_data.json HTTP/1.1

repos/system_upgrade/common/actors/scanleappinfo/actor.py

repos/system_upgrade/common/libraries/fetch.py

repos/system_upgrade/common/actors/scanleappinfo/actor.py

repos/system_upgrade/common/actors/checkconsumedassets/actor.py

repos/system_upgrade/common/actors/checkconsumedassets/libraries/check_consumed_assets.py

Rezney · 2023-01-03T15:24:17Z

@MichalHe we will want to discuss this solution with @pirat89. Let's have a quick sync next week.

MichalHe · 2023-02-06T18:30:25Z

/packit build

MichalHe · 2023-02-07T09:58:56Z

/packit build

repos/system_upgrade/common/actors/checkconsumedassets/libraries/check_consumed_assets.py

fernflower · 2023-02-13T10:43:49Z

@pirat89 @MichalHe I am a bit confused - this PR ties leapp asset version to leapp version, but in the Friday discussion with @pirat89 it was stated that we are going to introduce a separate asset versioning in X.Y. format (starting with '1.0'). Is it me getting something wrong or this PR hasn't been updated yet?

github-actions · 2023-02-13T16:42:28Z

Testing Farm request for RHEL-7.9-ZStream/5521869 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

fernflower · 2023-02-14T14:35:32Z

Okay, having tried this with updated leapp-data I managed to crash the actor

        Check whether Leapp is using correct data assets.
Process Process-290:
Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/site-packages/leapp/repository/actor_definition.py", line 74, in _do_run
    actor_instance.run(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/leapp/actors/__init__.py", line 289, in run
    self.process(*args)
  File "/etc/leapp/repos.d/system_upgrade/common/actors/checkconsumedassets/actor.py", line 18, in process
    check_consumed_assets.inhibit_if_assets_with_incorrect_version()
  File "/etc/leapp/repos.d/system_upgrade/common/actors/checkconsumedassets/libraries/check_consumed_assets.py", line 162, in inhibit_if_assets_with_incorrect_version
    report_malformed_assets(incompatible_assets.malformed)
  File "/etc/leapp/repos.d/system_upgrade/common/actors/checkconsumedassets/libraries/check_consumed_assets.py", line 113, in report_malformed_assets
    report_parts += make_report_entries_with_unique_urls(docs_url_to_title_map)
  File "/etc/leapp/repos.d/system_upgrade/common/actors/checkconsumedassets/libraries/check_consumed_assets.py", line 44, in make_report_entries_with_unique_urls
    api.current_logger().warning(msg.format(', '.join(urls_with_multiple_titles)))
KeyError: 'urls'


======================================================================================================
Actor check_consumed_assets unexpectedly terminated with exit code: 1 - Please check the above details
======================================================================================================

Looks like L44 is missing named urls formatting parameter. But as it crashed I'd also cover it with a unit test first.
api.current_logger().warning(msg.format(urls=', '.join(urls_with_multiple_titles))

fernflower

A unit test needs to be added to cover the crash.
(I might add it myself, just taking a note)

repos/system_upgrade/common/actors/checkconsumedassets/libraries/check_consumed_assets.py

fernflower · 2023-02-14T15:18:01Z

repos/system_upgrade/common/actors/checkconsumedassets/libraries/check_consumed_assets.py

+            details = details.format(what_asset=asset.filename)
+        else:
+            article, multiple_suffix = ('the ', '') if len(asset.provided_data_streams) == 1 else ('', 's')
+            details = ' - The asset {what_asset} provides data {article}stream{mult_suffix} {provided_streams}'


Doesn't provide a nice message in all cases, for example for a single from-the-future asset 42.42:

"summary": "The currently installed Leapp consumes data stream 1.0, but the following assets provide different streams:\nThe following assets are intended for a newer leapp\n - The asset pes-events.json provides data the stream 42.42"

Note the "The asset pes-events.json provides data the stream 42.42"

fernflower · 2023-02-15T11:36:35Z

/rerun

repos/system_upgrade/common/libraries/fetch.py

github-actions · 2023-02-15T11:41:52Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5528461

github-actions · 2023-02-15T11:48:24Z

Testing Farm request for RHEL-7.9-rhui/5528461 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-15T11:49:01Z

Testing Farm request for RHEL-8.6-rhui/5528461 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

This patch introduces support for a provided_data_streams: [v1, v2] field in leapp data (pes-events, device-drivers and repomapping). The new check_consumed_assets actor will check that installed version of leapp is compatible with leapp data provided, and inform if any problems are discovered. Unit tests added as well. OAMG-7989

fernflower · 2023-02-15T12:29:18Z

/rerun

github-actions · 2023-02-15T12:34:19Z

Copr build succeeded: https://copr.fedorainfracloud.org/coprs/build/5528570

github-actions · 2023-02-15T12:39:46Z

Testing Farm request for RHEL-7.9-rhui/5528570 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-15T12:41:15Z

Testing Farm request for RHEL-8.6-rhui/5528570 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

stale

github-actions · 2023-02-15T14:02:40Z

Testing Farm request for RHEL-8.6.0-Nightly/5528461 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-15T14:08:40Z

Testing Farm request for RHEL-8.7.0-Nightly/5528461 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-15T14:26:24Z

Testing Farm request for RHEL-7.9-ZStream/5528461 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-15T14:27:13Z

Testing Farm request for RHEL-7.9-ZStream/5528461 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-15T14:56:54Z

Testing Farm request for RHEL-8.6.0-Nightly/5528570 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-15T14:58:36Z

Testing Farm request for RHEL-8.7.0-Nightly/5528570 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-15T15:08:10Z

Testing Farm request for RHEL-7.9-ZStream/5528570 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

github-actions · 2023-02-15T15:13:30Z

Testing Farm request for RHEL-7.9-ZStream/5528570 regression testing has been created.
Once finished, results should be available here.
Full pipeline log.

## Packaging - Requires cpio (oamg#979) - Requires python3-gobject-base, NetworkManager-libnm (oamg#969) - Bump leapp-repository-dependencies to 9 (oamg#969, oamg#979) ## Upgrade handling ### Fixes - Add leapp RHUI packages to an allowlist to drop confusing reports (oamg#995) - Check only mounted XFS partitions (oamg#1027) - Detect the kernel-core RPM instead of kernel to prevent an error during post-upgrade phases (oamg#1024) - Disable the amazon-id DNF plugin on AWS during the upgrade stage to omit error messages during the upgrade process caused by missing network connection (oamg#990) - Do not create new *pyc files when running leapp after the DNF upgrade transaction (oamg#1017) - Enable upgrades on s390x when /boot is part of rootfs (oamg#991) - Extend the allow list of RHUI clients by azure-sap-apps to omit confusing report (oamg#974) - Filter out PES events unrelated for the used upgrade path and handle overlapping events (oamg#1008) - Fix scan of ceph volumes on systems without ceph-osd (oamg#1011) - Fix scan of ceph volumes when ceph-osd container is not found (oamg#986) - Fix systemd symlinks that become incorrect during the IPU (oamg#972) - Fix the check of memory (RAM) limits (oamg#984) - Fix the upgrade of IBM Z machines configured with ZFCP (oamg#983) - Ignore external accounts in /etc/passwd (oamg#958) - Inhibit the upgrade when entries in /etc/fstab cause overshadowing during the upgrade (oamg#1009) - Prevent leapp failures caused by re-run of leapp in the upgrade initramfs after previous failure, which causes additional confusing error message hiding original bugs (oamg#996) - Prevent the upgrade with RHSM when a baseos and an appstream target repositories are not discovered (oamg#1001) - RHUI(Azure) handle correctly various SAP images (oamg#1037) - Rework the network configuration handling and parse the configuration data properly (oamg#969) - Set RHSM release for non-ga and non-beta channels (oamg#1033) - Use the "grub" library to find the GRUB device (oamg#989) - [IPU 7 -> 8] Detect corrupted grubenv file (oamg#1012) - [IPU 7 -> 8] Ensure that rsyncd stays enabled if it is enabled prior the upgrade(oamg#1043) - [IPU 7 -> 8] Ensure that satellite metapackages are installed after the upgrade (oamg#994) - [IPU 7 -> 8] Ensure the device_cio_free service stays enabled on s390x after the upgrade (oamg#977) - [IPU 7 -> 8] Fixed checks for RHEL SAP IPU 7.9 -> 8.6 (oamg#978) - [IPU 7 -> 8] Fixed migration of ntp to chrony when the ntp service is masked (oamg#966) - [IPU 7 -> 8] Prevent the traceback during migration of sendmail configuration files when the package is not installed (oamg#1041) - [IPU 7 -> 8] Satellite: reindex all related databases to fix issues due to new locales in RHEL 8 (oamg#1007, oamg#1018) - [IPU 7 -> 8] Use the correct domain name in SSSD reports (oamg#1040) - [IPU 8 -> 9] Added checks for RHEL SAP IPU 8.6 -> 9.0 (oamg#978) - [IPU 8 -> 9] CheckVDO: Ask user for the confirmation only on failures and undetermined devices (oamg#961) - [IPU 8 -> 9] Fix the kernel detection during initramfs creation for new kernel on RHEL 9.2+ (oamg#1048) - [IPU 8 -> 9] Fix the upgrade on Azure using RHUI for SAP Apps images (oamg#975) - [IPU 8 -> 9] Handle correctly firewalld version 0.8 (oamg#963) ### Enhancements - Set new upgrade paths (oamg#988): -- RHEL 7.9 -> 8.8, 8.6 (default: 8.8) -- RHEL 8.6 -> 9.0 -- RHEL 8.8 -> 9.2 - Check that used leapp data are valid and compatible with the installed leapp-repository (oamg#1003) - Detect a proxy configuration in YUM/DNF and adjust an error msg on issues caused by the configuration (oamg#914) - Detect and report systemd symlinks that are broken before the upgrade (oamg#972) - Drop obsoleted upgrade paths (oamg#1047) - Improve remediation instructions for packages in unknown repositories (oamg#1010) - Improve the error message to guide users when discovered more space is needed (oamg#956) - Introduce --nogpgcheck option to skip checking of RPM signatures (oamg#910) - Introduced an option to use an ISO file as a target RHEL version content source (oamg#979) - Introduced possibility to specify what systemd services should be enabled/disabled on the upgraded system (oamg#964) - Map the target repositories also based on the installed content (oamg#967) - Provide common information about systemd services (oamg#959) - Register subscribed systems automatically to Red Hat Insights unless --no-insights-register is used (oamg#1000) - Remove obsoleted GPG keys provided by RH after the upgrade to prevent errors (oamg#1022) - Run upgrade process with checking RPM signatures by default (oamg#910, oamg#993, oamg#1025) - Save breadcrumbs results as RHSM facts (oamg#1002) - Small improvements in various reports (oamg#1038, oamg#1039, oamg#1032) - [IPU 8 -> 9] Detect CIFS also when upgrading from RHEL8 to RHEL9 (PR1035) - [IPU 8 -> 9] Detect RoCE on IBM Z machines and check the configuration is safe for the upgrade (oamg#1030) - [IPU 8 -> 9] Enable upgrades of RHEL 8 for SAP HANA to RHEL 9 on ppc64le (oamg#1042) - [IPU 8 -> 9] Improve the handling of blocklisted certificates (oamg#992) ## Additional changes interesting for devels - Started work on bringing up networking inside the upgrade initramfs - currently available for testing and development purposes when LEAPP_DEVEL_INITRAM_NETWORK is set (oamg#960) - Add leapp debug tools to the upgrade initramfs - dracut upgrade module (oamg#997) - Enable disabling of DNF plugins in the dnfplugin library (oamg#990)

## Packaging - Requires cpio (#979) - Requires python3-gobject-base, NetworkManager-libnm (#969) - Bump leapp-repository-dependencies to 9 (#969, #979) ## Upgrade handling ### Fixes - Add leapp RHUI packages to an allowlist to drop confusing reports (#995) - Check only mounted XFS partitions (#1027) - Detect the kernel-core RPM instead of kernel to prevent an error during post-upgrade phases (#1024) - Disable the amazon-id DNF plugin on AWS during the upgrade stage to omit error messages during the upgrade process caused by missing network connection (#990) - Do not create new *pyc files when running leapp after the DNF upgrade transaction (#1017) - Enable upgrades on s390x when /boot is part of rootfs (#991) - Extend the allow list of RHUI clients by azure-sap-apps to omit confusing report (#974) - Filter out PES events unrelated for the used upgrade path and handle overlapping events (#1008) - Fix scan of ceph volumes on systems without ceph-osd (#1011) - Fix scan of ceph volumes when ceph-osd container is not found (#986) - Fix systemd symlinks that become incorrect during the IPU (#972) - Fix the check of memory (RAM) limits (#984) - Fix the upgrade of IBM Z machines configured with ZFCP (#983) - Ignore external accounts in /etc/passwd (#958) - Inhibit the upgrade when entries in /etc/fstab cause overshadowing during the upgrade (#1009) - Prevent leapp failures caused by re-run of leapp in the upgrade initramfs after previous failure, which causes additional confusing error message hiding original bugs (#996) - Prevent the upgrade with RHSM when a baseos and an appstream target repositories are not discovered (#1001) - RHUI(Azure) handle correctly various SAP images (#1037) - Rework the network configuration handling and parse the configuration data properly (#969) - Set RHSM release for non-ga and non-beta channels (#1033) - Use the "grub" library to find the GRUB device (#989) - [IPU 7 -> 8] Detect corrupted grubenv file (#1012) - [IPU 7 -> 8] Ensure that rsyncd stays enabled if it is enabled prior the upgrade(#1043) - [IPU 7 -> 8] Ensure that satellite metapackages are installed after the upgrade (#994) - [IPU 7 -> 8] Ensure the device_cio_free service stays enabled on s390x after the upgrade (#977) - [IPU 7 -> 8] Fixed checks for RHEL SAP IPU 7.9 -> 8.6 (#978) - [IPU 7 -> 8] Fixed migration of ntp to chrony when the ntp service is masked (#966) - [IPU 7 -> 8] Prevent the traceback during migration of sendmail configuration files when the package is not installed (#1041) - [IPU 7 -> 8] Satellite: reindex all related databases to fix issues due to new locales in RHEL 8 (#1007, #1018) - [IPU 7 -> 8] Use the correct domain name in SSSD reports (#1040) - [IPU 8 -> 9] Added checks for RHEL SAP IPU 8.6 -> 9.0 (#978) - [IPU 8 -> 9] CheckVDO: Ask user for the confirmation only on failures and undetermined devices (#961) - [IPU 8 -> 9] Fix the kernel detection during initramfs creation for new kernel on RHEL 9.2+ (#1048) - [IPU 8 -> 9] Fix the upgrade on Azure using RHUI for SAP Apps images (#975) - [IPU 8 -> 9] Handle correctly firewalld version 0.8 (#963) ### Enhancements - Set new upgrade paths (#988): -- RHEL 7.9 -> 8.8, 8.6 (default: 8.8) -- RHEL 8.6 -> 9.0 -- RHEL 8.8 -> 9.2 - Check that used leapp data are valid and compatible with the installed leapp-repository (#1003) - Detect a proxy configuration in YUM/DNF and adjust an error msg on issues caused by the configuration (#914) - Detect and report systemd symlinks that are broken before the upgrade (#972) - Drop obsoleted upgrade paths (#1047) - Improve remediation instructions for packages in unknown repositories (#1010) - Improve the error message to guide users when discovered more space is needed (#956) - Introduce --nogpgcheck option to skip checking of RPM signatures (#910) - Introduced an option to use an ISO file as a target RHEL version content source (#979) - Introduced possibility to specify what systemd services should be enabled/disabled on the upgraded system (#964) - Map the target repositories also based on the installed content (#967) - Provide common information about systemd services (#959) - Register subscribed systems automatically to Red Hat Insights unless --no-insights-register is used (#1000) - Remove obsoleted GPG keys provided by RH after the upgrade to prevent errors (#1022) - Run upgrade process with checking RPM signatures by default (#910, #993, #1025) - Save breadcrumbs results as RHSM facts (#1002) - Small improvements in various reports (#1038, #1039, #1032) - [IPU 8 -> 9] Detect CIFS also when upgrading from RHEL8 to RHEL9 (PR1035) - [IPU 8 -> 9] Detect RoCE on IBM Z machines and check the configuration is safe for the upgrade (#1030) - [IPU 8 -> 9] Enable upgrades of RHEL 8 for SAP HANA to RHEL 9 on ppc64le (#1042) - [IPU 8 -> 9] Improve the handling of blocklisted certificates (#992) ## Additional changes interesting for devels - Started work on bringing up networking inside the upgrade initramfs - currently available for testing and development purposes when LEAPP_DEVEL_INITRAM_NETWORK is set (#960) - Add leapp debug tools to the upgrade initramfs - dracut upgrade module (#997) - Enable disabling of DNF plugins in the dnfplugin library (#990)

MichalHe force-pushed the versioned_data_files branch 9 times, most recently from 8a91f7b to 45e2e97 Compare December 30, 2022 19:52

MichalHe force-pushed the versioned_data_files branch 4 times, most recently from f952dca to dbd9644 Compare December 31, 2022 20:35

MichalHe marked this pull request as ready for review December 31, 2022 20:55

Rezney previously requested changes Jan 3, 2023

View reviewed changes

MichalHe force-pushed the versioned_data_files branch 2 times, most recently from 35fe0c0 to 3144c48 Compare January 3, 2023 17:13

MichalHe force-pushed the versioned_data_files branch 2 times, most recently from a95b320 to 490edb0 Compare January 16, 2023 13:52

pirat89 added this to the 8.8/9.2 milestone Jan 18, 2023

pirat89 self-assigned this Feb 6, 2023

MichalHe force-pushed the versioned_data_files branch from bc542e8 to ff81358 Compare February 6, 2023 17:42

pirat89 reviewed Feb 10, 2023

View reviewed changes

repos/system_upgrade/common/actors/checkconsumedassets/libraries/check_consumed_assets.py Outdated Show resolved Hide resolved

MichalHe force-pushed the versioned_data_files branch from 3e6a6d6 to fd9e03a Compare February 12, 2023 17:49

fernflower previously requested changes Feb 14, 2023

View reviewed changes

pirat89 reviewed Feb 15, 2023

View reviewed changes

repos/system_upgrade/common/libraries/fetch.py Outdated Show resolved Hide resolved

fernflower force-pushed the versioned_data_files branch from 904b022 to e0106c4 Compare February 15, 2023 12:28

fernflower approved these changes Feb 15, 2023

View reviewed changes

fernflower merged commit becb7e5 into oamg:master Feb 15, 2023

pirat89 added the changelog-checked The merger/reviewer checked the changelog draft document and updated it when relevant label Feb 21, 2023

pirat89 mentioned this pull request Feb 21, 2023

# Release 0.18.0 #1049

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

assets: check for what leapp version are assets meant for #1003

assets: check for what leapp version are assets meant for #1003

MichalHe commented Dec 23, 2022 •

edited

github-actions bot commented Dec 23, 2022

MichalHe commented Dec 31, 2022

Rezney commented Jan 3, 2023

MichalHe commented Feb 6, 2023

MichalHe commented Feb 7, 2023

fernflower commented Feb 13, 2023

github-actions bot commented Feb 13, 2023

fernflower commented Feb 14, 2023 •

edited

fernflower left a comment

fernflower Feb 14, 2023

fernflower commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

fernflower commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

assets: check for what leapp version are assets meant for #1003

assets: check for what leapp version are assets meant for #1003

Conversation

MichalHe commented Dec 23, 2022 • edited

Introduction of versions to leapp data assets

github-actions bot commented Dec 23, 2022

Thank you for contributing to the Leapp project!

MichalHe commented Dec 31, 2022

Conducted testing (rhel8 dev vagrant box)

Rezney commented Jan 3, 2023

MichalHe commented Feb 6, 2023

MichalHe commented Feb 7, 2023

fernflower commented Feb 13, 2023

github-actions bot commented Feb 13, 2023

fernflower commented Feb 14, 2023 • edited

fernflower left a comment

Choose a reason for hiding this comment

fernflower Feb 14, 2023

Choose a reason for hiding this comment

fernflower commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

fernflower commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

github-actions bot commented Feb 15, 2023

MichalHe commented Dec 23, 2022 •

edited

fernflower commented Feb 14, 2023 •

edited