Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

repo_checker: provide --post-comments option on project_only subcommand (and cleanups). #1042

Conversation

@jberry-suse
Copy link
Contributor

jberry-suse commented Aug 3, 2017

  • f42cc4b:
    repo_checker: truncate long messages to avoid crashing OBS.

  • 722764a:
    repo_checker: provide --post-comments option on project_only subcommand.

  • ce35a68:
    repo_checker: provide optional parsing of install check output and mapping to package.

  • d87ee4c:
    repo_checker: package_whitelist() does not need to support group splitting to find product.

  • 924db60:
    osclib/core: provide package_binary_list() to efficiently obtain binary to package map.

  • 0452fb2:
    osclib/core: add package to BinaryParsed tuple.

  • 16c5c92:
    osclib/core: provide BINARY_REGEX named group pattern and utilize.

  • cfbd479:
    ReviewBot: comment_write(): support package.

Deployment as service should be blocked by #1029 even though old repo_checker dumped to file regardless this is noisy and makes false positives more annoying.

Invocation is simple:

./repo_checker.py --verbose project_only --post-comments openSUSE:Factory

I went ahead and did a manual run in order to get the cleanup started especially with SLE/Leap 15 branching from Factory. The script produced 111 comments on devel packages containing the aggregated problems.

To be clear this approach:

  • parses the installcheck & file conflict output
  • maps the binary names back to packages
  • groups all messages related to the same package
  • sorts the output and posts as comment on package in devel project in order to trigger notifications

I debated posting bugs to bugzilla, but that would be a fair bit more work especially considering the clean comments api we have for avoiding useless updates, but ensuring a single comment. I did envision shared bugs between two packages that have file conflicts which would be neat, but this just duplicates those messages to both packages.

A hash representing the binaries and number of issues is generated to avoid updating comment for version updates or release increments while still properly picking up new or changes to existing issues.

Depending on how this works out and opinions this approach can be broadened for the rest of repo_checker or at minimum include for duplicate binary checks.

After the recent fixes that were hiding all -32bit imported binaries in the x86_64 space added to the already exposed issues there are quite a few even after all my fixes.

See dashboard/repo_checker for an updated report.

The end result are package comments like the following on devel:languages:ruby:extensions/rubygem-celluloid.

image

One caveat being that there is no simple way of removing these comments since the OBS comment API does not provide a way of querying all comments by a user so each devel project would have to be scanned. Alternatively, on accept comments for all effected packages could be removed. I would expect these comments to be relatively rare once the new repo_checker is enforcing cleanliness on new requests so this is more of a stop-gap for getting maintainers to act for the initial cleanup. It seems simple enough for maintainers to remove them once fixes are accepted.

Fixes #1030.

@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Aug 3, 2017

Could also use the new package_binary_list() for duplicate check and the ignore list of deletes to avoid individual package queries. The check command being relevant to #993.

@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Aug 3, 2017

hehe, this already worked...I got an email notifying me on one of my packages about an issue of which I was not aware.

@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Aug 4, 2017

As seen in this example the comments are resulting in fixes!

https://build.opensuse.org/package/show/devel:languages:python/pymol

@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Aug 4, 2017

Re-ran today for stats (package, len(sections)) [112 packages]:

('rubygem-rhc', 6)
('rubygem-celluloid', 6)
('perl-Selenium-Remote-Driver', 1)
('nss_ldap', 1)
('dice', 2)
('deepin-music-player', 2)
('hdjmod', 1)
('zerobranestudio', 2)
('pcfclock', 1)
('rubygem-r10k', 6)
('openmpi', 1)
('epour', 2)
('ghc-dice', 2)
('ghc-postgresql-simple-migration', 2)
('espeak-gui', 4)
('installation-images', 10)
('vegastrike-speech', 1)
('kernel-syms', 1)
('yast2-docker', 1)
('sysdig', 1)
('ghc-protobuf-simple', 2)
('libfabric', 1)
('virtaal', 4)
('python3-geoip2', 2)
('enchant', 1)
('php5-ZendFramework', 2)
('gns3-server', 2)
('tkman', 2)
('vhba-kmp', 1)
('rubygem-puppet_forge', 6)
('opentoonz', 2)
('octave-forge-ocs', 2)
('gns3', 2)
('ghc-convert-annotation', 2)
('prosody', 2)
('dkms', 1)
('openssl-1_1_0', 1)
('xapps', 1)
('rubygem-bundler', 8)
('libqt5-qt3d', 4)
('boost', 2)
('ginac', 2)
('xtables-addons', 1)
('rubygem-gpgme', 6)
('rubygem-sdoc', 6)
('hdf5', 3)
('rubygem-logging', 6)
('libqt5-qtwebengine', 2)
('vegastrike-music', 1)
('rubygem-ra10ke', 4)
('python-web.py', 1)
('buzztrax', 1)
('docker-bench-security', 1)
('bbswitch', 1)
('deepin-tool-kit', 14)
('rubygem-winrm-2_1', 4)
('nagvis', 4)
('ghc-inline-r', 2)
('icingaweb2-module-director', 2)
('virtualbox', 2)
('jhbuild', 2)
('openssl-1_0_0', 1)
('ruby2.4', 2)
('ha-cluster-bootstrap', 2)
('Mesa', 1)
('geis', 1)
('kaccounts-integration', 1)
('kiwi', 2)
('python-carrot', 2)
('paperwork', 2)
('rubygem-winrm-fs', 6)
('python-kiwi', 1)
('yast2-drbd', 1)
('perl-CPAN-Meta-YAML', 2)
('patterns-caasp', 1)
('rubygem-winrm-elevated', 6)
('libstatgrab', 1)
('kopano', 58)
('hgview', 4)
('python-launchpadlib', 2)
('hawk2', 2)
('rubygem-rake', 2)
('kopano-webapp', 2)
('rubygem-faraday_middleware-0_9', 4)
('rubygem-pry_debug', 6)
('rubygem-jekyll', 4)
('python-plaso', 2)
('rubygem-nokogiri-1_6', 6)
('libpinyin', 1)
('arpack-ng', 2)
('openmpi2', 1)
('docker-compose', 1)
('rubygem-studio_api', 6)
('scim-bridge', 2)
('libvoikko', 1)
('scalapack', 2)
('octave-forge-secs1d', 2)
('kdebase3-SuSE', 4)
('fwupdate', 3)
('protobuf', 2)
('nss-pam-ldapd', 1)
('gobby', 4)
('fusecompress', 2)
('purpose', 2)
('python-sqlalchemy-migrate', 2)
('vegastrike-extra', 1)
('ktexteditor', 2)
('bind', 1)
('pymol', 1)
('tulip', 16)
('dcmtk', 6)
('octave-forge-secs3d', 2)
@jberry-suse jberry-suse force-pushed the jberry-suse:repo_checker-project_only-post-comments branch from f42cc4b to 9443c3d Aug 4, 2017
@openSUSE openSUSE deleted a comment from coveralls Aug 4, 2017
@coveralls

This comment has been minimized.

Copy link

coveralls commented Aug 4, 2017

Coverage Status

Coverage decreased (-0.2%) to 48.347% when pulling 9443c3d on jberry-suse:repo_checker-project_only-post-comments into cdde2bf on openSUSE:master.

@jberry-suse jberry-suse merged commit 072d1eb into openSUSE:master Aug 4, 2017
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@jberry-suse jberry-suse deleted the jberry-suse:repo_checker-project_only-post-comments branch Aug 4, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

2 participants
You can’t perform that action at this time.