Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accept -Q / -K in show-build-deps, sort-dependencies, xbps-cycles.py #47888

Merged
merged 4 commits into from
Apr 8, 2024

Conversation

tornaria
Copy link
Contributor

@tornaria tornaria commented Dec 23, 2023

As it is now, if pkgA checkdepends on pkgB, sort-dependencies will still print pkgA before pkgB. This causes CI to build pkgB twice: first build pkgA which forces implicit build of pkgB; then build of pkgB (explicit, so it will ignore the package is already built).

A concrete example:

$ ./xbps-src sort-dependencies python3-process-tests python3-pytest-cov
python3-pytest-cov
python3-process-tests

The example above causes python-process-tests to be built twice, as shown in:
https://github.com/void-linux/void-packages/actions/runs/7107346278/job/19348602412?pr=47610

After this commit one can do

$ ./xbps-src -Q sort-dependencies python3-process-tests python3-pytest-cov
python3-process-tests
python3-pytest-cov

The CI issue is fixed by pasing -Q to sort-dependencies when we are doing a test build.

Jump to #47888 (comment) to skip comments about earlier versions of the PR.

Testing the changes

  • I tested the changes in this PR: YES

@classabbyamp classabbyamp added new-package This PR adds a new package xbps-src xbps-src related and removed new-package This PR adds a new package labels Dec 24, 2023
@sgn
Copy link
Member

sgn commented Dec 25, 2023

We need to ignore checkdepends during non-testing-build

@tornaria
Copy link
Contributor Author

We need to ignore checkdepends during non-testing-build

This only affects order, in the sense that sort-dependencies will not add anything, just sort.

For example:

$ ./xbps-src sort-dependencies python3-pytest-cov
python3-pytest-cov

even if python3-pytest-cov checkdepends on python3-process-tests.

OTOH,in

$ ./xbps-src sort-dependencies python3-process-tests python3-pytest-cov
python3-pytest-cov
python3-process-tests

it is irrelevant for non-testing-build in which order these two packages are built, so it should not be a problem.

The only case where this would make a difference is if there is a build-check cycle. Then this will print a warning, and IMO it should be considered a bug.

Example:

$ XBPS_CHECK_PKGS=full ./xbps-src sort-dependencies python3-jupyter_server python3-jupyter_server_terminals
tsort: /tmp/tmp.owWM5auH2j: input contains a loop:
tsort: python3-jupyter_server
tsort: python3-jupyter_server_terminals
python3-jupyter_server_terminals
python3-jupyter_server

Note this is only a cycle because of XBPS_CHECK_PKGS=full which is one way build-check cycles can be broken. Indeed:

$ XBPS_CHECK_PKGS=yes ./xbps-src sort-dependencies python3-jupyter_server python3-jupyter_server_terminals
python3-jupyter_server_terminals
python3-jupyter_server

@tornaria
Copy link
Contributor Author

Also, note that there are several situations of non-testing-build where checkdepends will still be required (see #46207).

@tornaria
Copy link
Contributor Author

Pushed a new version. Now show-build-deps will include checkdepends when using -Q or -K.

This means sort-dependencies will do the same when using -Q or -K.

As is this won't fix CI, since common/travis/changed_templates.sh doesn't pass -Q to sort-dependencies.

It's also not 100% correct since there are a number of situations where XBPS_CHECK_PKGS can be set but check will not run. See here: https://github.com/void-linux/void-packages/pull/46207/files#diff-2553330b694a471fd9a77f9645b03f16c9c92e8e656a5b44634d816cbdaa2a5dR140-R144 for a set of conditions which imply no testing for a particular package. Maybe this can be refactored somewhere and used for this PR and also to fix #46207.

The idea is: read the pkg, determine if checks would be run for this pkg (based on make_check, value of XBPS_CHECK_PKGS, ci-skip, etc) and only include checkdepends together with build depends if the checks will be run.

This also means it's easy to use common/scripts/xbps-cycles.py to catch build-check cycles by doing something like XBPS_CHECK_PKGS=yes common/scripts/xbps-cycles.py.

Proposal (again): consider build-check cycles with XBPS_CHECK_PKGS=yes to be a bug; only allow build-check cycles when XBPS_CHECK_PKGS=full.

@tornaria
Copy link
Contributor Author

New version:

  • make sure ./xbps-src show-build-deps PKG will not exit 1: changed [ -n "$XBPS_CHECK_PKGS" ] && show_pkgs_check_deps to [ -z "$XBPS_CHECK_PKGS" ] || show_pkg_check_deps
  • Fix show-build-deps for dependencies *-32bit (try ./xbps-src show-build-deps gcc-multilib)
  • Fix dependencies in four packages so show-build-deps won`t fail.

Now running common/scripts/xbps-cycles.py finishes without any error (and no cycles), and it's also possible to run XBPS_CHECK_PKGS=yes common/scripts/xbps-cycles.py (no errors, 6 cycles) and XBPS_CHECK_PKGS=full common/scripts/xbps-cycles.py (no errors, 8 cycles)

…tep()

This function contains the logic that determines whether the check step
will be skipped for the current pkg, taking into account `make_check`.

Use this new function in `install_pkg_deps()` so that it uses a more
accurate condition to skip installing check dependencies.

For instance, check dependencies for a pkg with `make_check=extended`
will no longer be installed when using `-Q`. Similar for `make_check=ci-skip`.

Replaces: void-linux#46207
Due to this change, `./xbps-src sort-dependencies` will take
checkdepends into account when using -Q or -K.

Before this commit, if `pkgA` checkdepends on `pkgB`, sort-dependencies
could still print `pkgA` before `pkgB`. This causes CI to build `pkgB`
twice: first when building `pkgA`, which forces implicit build of pkgB;
second when building `pkgB` (explicit, so it will ignore the package is
already built).

The implementation uses `skip_check_step()` from previous commit, for
consistency, so checkdepends are only taken into account if the check
step would be enabled.

In particular, nothing is changed unless -Q or -K flag is passed.

EXAMPLE:

Before:
```
$ ./xbps-src -Q sort-dependencies python3-process-tests python3-pytest-cov
python3-pytest-cov
python3-process-tests
```

After:
```
$ ./xbps-src -Q sort-dependencies python3-process-tests python3-pytest-cov
python3-process-tests
python3-pytest-cov
```
This ensures that checkdepends will be taken into account in the build
order whenever test is enabled.
@tornaria tornaria changed the title fix sort-dependencies to use checkdepends Include checkdepends in show-build-deps / fix sort-dependencies for CI Dec 26, 2023
@tornaria
Copy link
Contributor Author

Please review. In this version I moved fixing other pkgs to #47910 to make this PR simpler.

Summary of changes:

  1. implement function skip_check_step() with the logic that determines whether the check step will be skipped for the current pkg, taking into account make_check. As a side effect install_pkg_deps() now uses a more accurate condition to skip installing checkdepends.
  2. Fix implementation of ./xbps-src show-build-deps so it includes checkdepends when using -Q or -K (using skip_check_step() to match install_pkg_deps() so it will be consistent). As a side effect, ./xbps-src sort-dependencies will take checkdepends into account.
  3. Change common/travis/build.sh so that -Q is passed to sort-dependencies when we are doing a test build.
  4. Add -Q and -K flags to common/scripts/xbps-cycles.py.

@sgn does this address your concerns?

@tornaria tornaria changed the title Include checkdepends in show-build-deps / fix sort-dependencies for CI Accept -Q / -K in show-build-deps, sort-dependencies, xbps-cycles.py Feb 15, 2024
@tornaria
Copy link
Contributor Author

ping

Copy link
Member

@Chocimier Chocimier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@tornaria
Copy link
Contributor Author

This also fixes xbulk. A quick example, currently on master:

The following is ok, since neither package depends or builddepends on the other:

$ xbulk -n python3-coverage python3-flaky
pkg=pkg-python3-coverage; ./xbps-src  pkg ${pkg#pkg-}
pkg=pkg-python3-flaky; ./xbps-src  pkg ${pkg#pkg-}

The following is NOT ok, since coverage checkdepends on flaky

$ xbulk -n -K python3-coverage python3-flaky
pkg=pkg-python3-coverage; ./xbps-src  -K pkg ${pkg#pkg-}
pkg=pkg-python3-flaky; ./xbps-src  -K pkg ${pkg#pkg-}

This is NOT ok, since

With this branch, the no-check version does the same as before, and the check version does the proper thing.

$ xbulk -n python3-coverage python3-flaky
pkg=pkg-python3-coverage; ./xbps-src  pkg ${pkg#pkg-}
pkg=pkg-python3-flaky; ./xbps-src  pkg ${pkg#pkg-}
$ xbulk -n -K python3-coverage python3-flaky
pkg=pkg-python3-flaky; ./xbps-src  -K pkg ${pkg#pkg-}
pkg=pkg-python3-coverage; ./xbps-src  -K pkg ${pkg#pkg-}

@tornaria
Copy link
Contributor Author

tornaria commented Apr 7, 2024

@sgn: this scratches an itch and I think it's in good shape. I use it sometimes by merging into my current branch (useful on jupyter branch which often has several updates with a number of check dependencies).

By fixing sort-dependencies when used with -Q or -K, this fixes CI build and xbulk with check.

@sgn sgn merged commit d8c079c into void-linux:master Apr 8, 2024
@tornaria tornaria deleted the sort-deps branch April 8, 2024 22:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
xbps-src xbps-src related
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants