Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interleave download actions with build/install actions #3777

Merged
merged 3 commits into from Mar 21, 2019

Conversation

Projects
None yet
4 participants
@Armael
Copy link
Member

commented Mar 11, 2019

Caveats:

  • Change of behavior: in the current implementation, if a "remove" needs a "fetch" and the fetch fails, opam will print a warning but do the remove nonetheless (by just removing the files). In this PR, if the fetch fails, the remove will automatically get cancelled. In practice, this can happen only for a few packages that do not use light-uninstall, but it would be annoying nonetheless. It would probably be nice to give the user a way around this, using --force or something.
  • Currently, the jobs pool is shared between build and fetch actions. This can be improved, by implementing two separate pools in the scheduler, which should improve parallelism further.
    This can be done in a separate PR.
@Armael

This comment has been minimized.

Copy link
Member Author

commented Mar 11, 2019

I just added a commit that restores the existing behavior for "remove" after a failing "fetch": a warning gets printed, and the remove still occurs.

@avsm

This comment has been minimized.

Copy link
Member

commented Mar 11, 2019

Currently, the jobs pool is shared between build and fetch actions

Weirdly enough, this speeds up some builds for me. The reason is that jobs in opam is propagated to parallel dune builds with the same value, which in turn overcommit the CPU since they all spread to the full load of parallel jobs. With this, some of the opam jobs are fetching, while the dune jobs spread to all CPUs :-)

@Armael

This comment has been minimized.

Copy link
Member Author

commented Mar 11, 2019

Well, the point of the PR is to speedup the build! But it's interesting to know that it has a positive effect even without a separate queue for downloads (which would definitely make sure that downloads cannot starve builds).

(* Print a warning, but still do the remove *)
OpamConsole.warning
"The sources of the following couldn't be obtained, they may be \
uncleanly installed:\n%s"

This comment has been minimized.

Copy link
@AltGr

AltGr Mar 12, 2019

Member

removed

This comment has been minimized.

Copy link
@Armael

Armael Mar 12, 2019

Author Member

I don't understand what you mean here.

@AltGr

AltGr approved these changes Mar 12, 2019

Copy link
Member

left a comment

Just one small typo :)
Thanks a lot!

Show resolved Hide resolved src/solver/opamActionGraph.ml Outdated
@AltGr

This comment has been minimized.

Copy link
Member

commented Mar 12, 2019

After testing, one request could be to unify the output of finished downloads with that of actions:

[atdgen.2.0.0] found in cache
[atdgen-runtime.2.0.0] found in cache
[datakit-ci.0.12.3] found in cache
[atdgen-runtime.2.0.0] found in cache
[atdgen-runtime.2.0.0] found in cache
[github.4.0.0] found in cache
[github-unix.4.0.0] found in cache
[merlin.3.2.2] found in cache
[mirage-ci.~dev] no changes from git+https://github.com/avsm/mirage-ci
[yojson.1.7.0] found in cache
⊘ removed   merlin.3.2.2
⊘ removed   mirage-ci.~dev
⊘ removed   datakit-ci.0.12.3
⊘ removed   github-unix.4.0.0
⊘ removed   github.4.0.0
⊘ removed   atdgen.2.0.0
⊘ removed   atdgen-runtime.2.0.0
⊘ removed   yojson.1.7.0
∗ installed yojson.1.7.0
∗ installed atdgen-runtime.2.0.0
∗ installed atdgen.2.0.0
∗ installed merlin.3.2.2
∗ installed github.4.0.0
∗ installed github-unix.4.0.0
∗ installed datakit-ci.0.12.3
∗ installed mirage-ci.~dev

here they don't interleave, but it would be nice for the downloads to be displayed like removals or installs, e.g. :

⇓ fetched github.4.0.0 (cached)

@Armael Armael force-pushed the Armael:download_during_build branch from 2b164ee to 4b40a23 Mar 12, 2019

@AltGr AltGr merged commit 0e40a4e into ocaml:master Mar 21, 2019

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@rjbou rjbou added this to the 2.1.0 milestone Mar 21, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.