Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkglistgen: integrate drop list creation into update_and_solve and project family methods #1386

Merged
merged 11 commits into from Feb 7, 2018

Conversation

@jberry-suse
Copy link
Contributor

jberry-suse commented Feb 6, 2018

  • dc3b9e1:
    osclib/conf: include Leap for Factory during pkglistgen.

  • c56c23f:
    pkglistgen: include some hints about the difference between solv files.

  • b58cb89:
    pkglistgen: integrate drop list creation into update_and_solve.

    In order to generate a complete drop list the nonfree repository must also
    be cached for both current and published builds. The solv files are then
    merged and used to generate drop list.

  • 5e35baa:
    pkglistgen: do_dump_solv(): handle old-style product repo format.

    This allows for generating solv cache for Leap versions prior to 15.0
    which avoids the need to have them checked into git like previous
    package-lists repository.

  • 0590162:
    pkglistgen: do_create_droplist(): print to file when output_dir available.

  • 4d98ff8:
    pkglistgen: provide do_update_merge() to combine free and nonfree solv.

  • 9e77f2c:
    osclib/util: provide project_list_family* and project_version().

  • 378470e:
    osclib/stagingapi: get_staging_projects(): utilize project_list_prefix().

  • 42e698e:
    osclib/core: provide project_list_prefix().

  • 5aa70e1:
    osclib/conf: add download-baseurl for Leap and Factory.

  • 3f8cf29:
    osclib/conf: support version in project pattern as value replacement.

The product family functions should be usable in leaper.py to avoid all the hard-coding and priority orders and projects that has to be updated each cycle.

Rather than checking solv files into git like package-lists this code creates solv files for the relevant products in the family in a local cache. Unless I am missing something there seems to be no reason to keep them in git since they can be re-created. As discussed in #1375 they can be sync'd somewhere for backup if desired (as perhaps Tumbleweed may desire to keep all snapshots just in case). Then again, we can re-create them for at least the last three months from my Tumbleweed Snapshots ;).

Locally, the following exist after running for Leap 15.0.

~/.cache/opensuse-packagelists/solv/
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.2
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.2/openSUSE-Addon-NonOss-FTP-x86_64-Build0073.solv
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.2/openSUSE-42.2-x86_64-Build0283.merged.solv
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.2/openSUSE-42.2-x86_64-Build0283.solv
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.3
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.3/openSUSE-Addon-NonOss-FTP-x86_64-Build0082.solv
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.3/openSUSE-42.3-x86_64-Build0331.solv
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.3/openSUSE-42.3-x86_64-Build0331.merged.solv
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:15.0
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:15.0/openSUSE-15.0-x86_64-Build115.1.solv
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:15.0/openSUSE-Leap-15.0-Addon-NonOss-FTP-x86_64-Build115.1.solv
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:15.0/openSUSE-15.0-x86_64-Build115.1.merged.solv
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.1
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.1/openSUSE-42.1-x86_64-Build0265.solv
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.1/openSUSE-Addon-NonOss-FTP-i586-x86_64-Build0039.solv
~/.cache/opensuse-packagelists/solv/openSUSE:Leap:42.1/openSUSE-42.1-x86_64-Build0265.merged.solv

mergesolv seems to create a solv file that is smaller than the size of the two source solv files. I haven't verified if that makes sense, but performing the same operation as package-lists version. I noticed that steamcmd is show as removed when it exists in Leap 15.0, but did not investigate further.

Once deployed the obsoletepackages.inc in 000package-groups should be dropped to avoid confusion as this will generate one in 000product each run.

In general the command line classes should be thin glue code that maps command line arguments to function calls instead of contain the bulk of code. With this and my previous PR to migrate wrapper scripts into pkglistgen.py it is ever more clear the misuse of the ToolBase.CommandLineInterface class. Since a variety of methods within that class need to be called by higher level wrapper methods they should be separated to allow for all arguments to be passed directly. Having everything in CLI class results in the awquardness of having to rerun setup_tool() multiple times to change global arguments (like repos) and copy/pass around opts, not to mention subcmd (ie. do_thing('thing', ...)). For example see the extremely small repo_checker CLI class.

https://github.com/openSUSE/osc-plugin-factory/blob/cd1fa5db15f01317f9a1c9893380e17d42122fc0/repo_checker.py#L500-L529

Fixes #1375.

@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Feb 6, 2018

This should handle Factory worklfow to include Leap family for obsoletes drop list.

'pkglistgen-product-family-include': 'openSUSE:Leap:N',
@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Feb 6, 2018

The brittleness of tests that re-create OBS was shown again as it did not handle a properly encoded query string that my refactoring generated. Fixed that and a few minor flake8 notes.

@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Feb 6, 2018

Created #1387 as a reminder to utilize the project_list_family*() methods.

@jberry-suse jberry-suse force-pushed the jberry-suse:pkglistgen-droplist branch from ff6b273 to 99c36e4 Feb 6, 2018
@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Feb 6, 2018

Corrected edge-case with staging scope.

jberry-suse added 3 commits Feb 6, 2018
In order to generate a complete drop list the nonfree repository must also
be cached for both current and published builds. The solv files are then
merged and used to generate drop list.
@jberry-suse jberry-suse force-pushed the jberry-suse:pkglistgen-droplist branch from 99c36e4 to 3760840 Feb 6, 2018
@openSUSE openSUSE deleted a comment from coveralls Feb 6, 2018
@jberry-suse jberry-suse merged commit 56df1cf into openSUSE:master Feb 7, 2018
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@jberry-suse jberry-suse deleted the jberry-suse:pkglistgen-droplist branch Feb 7, 2018
@nilxam

This comment has been minimized.

Copy link
Contributor

nilxam commented Feb 8, 2018

Hmm... AFAICS the update in 000product/000release-packages what removed old obsoletepackage against the package were obsoleted in the version older than 42.1 ie. 13.2,etc., I'll ignore that dup failure on openqa for now, but we have to decide should we keep support those version upgrade case in Leap 15.0, and just in case we still have 13.2 dup test on openqa for TW and Leap 15.

ref. https://openqa.opensuse.org/tests/604949#step/zdup/144

@lnussel

@DimStar77

This comment has been minimized.

Copy link
Contributor

DimStar77 commented Feb 8, 2018

TW tests back to 13.1; 12.x was dropped as maintenance was getting too tedious (not from a product pov, but the disks were getting too small for the ever growing updates, and it was not worthy to produce new base disks)

For TW, we still test upgrades from 13.1, 13.2, 42.[123]

@nilxam

This comment has been minimized.

Copy link
Contributor

nilxam commented Feb 8, 2018

just noticed the discussion on #1375 (comment) , seems we can remove 13.x test from leap 15 openqa group.

@jberry-suse

This comment has been minimized.

Copy link
Contributor Author

jberry-suse commented Feb 9, 2018

Yeah, the current code only includes 42.{1,2,3} and 15.0 pre-releases. If that needs to be different let me know.

@lnussel

This comment has been minimized.

Copy link
Member

lnussel commented Feb 9, 2018

If we only knew how much 13.2 users are left ... I wanted to drop support for upgrading from 13.2 already with 42.3 but the access statistics of download.o.o made me reconsider that.
In general I'm fine with limiting the upgrade calculations and tests to 42.* for Leap 15.0.

jberry-suse added a commit to jberry-suse/openSUSE-release-tools that referenced this pull request Feb 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

4 participants
You can’t perform that action at this time.