Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Defer computation of local_bin #6764

Merged
merged 1 commit into from
Feb 1, 2023
Merged

Defer computation of local_bin #6764

merged 1 commit into from
Feb 1, 2023

Conversation

gridbugs
Copy link
Collaborator

@gridbugs gridbugs commented Dec 23, 2022 •
edited by emillon
Loading

This fixes a memo dependency cycle between evaluating globs in install stanzas and populating the artifacts database. Populating the artifacts database involves enumerating all files installed in the "bin" section which involves expanding globs as these files can be specified as globs rather than literal files. Expanding globs in the install stanza requires loading the rules for the directory containing the glob, and doing so depends on the artifacts database.

Fixes #6708

@gridbugs gridbugs force-pushed the fix-bin-install-glob-dependency-cycle branch from ce26f60 to 5cae830 Compare December 23, 2022 00:50
@gridbugs
Copy link
Collaborator Author

Fixes #6708

@gridbugs gridbugs force-pushed the fix-bin-install-glob-dependency-cycle branch from 5cae830 to 8db8875 Compare December 23, 2022 02:47
@gridbugs gridbugs changed the title Lazily compute local_bins Defer computation of local_bin Dec 23, 2022
@gridbugs gridbugs force-pushed the fix-bin-install-glob-dependency-cycle branch from 8db8875 to 40e3da6 Compare December 23, 2022 06:51
@gridbugs
Copy link
Collaborator Author

Some tests such as

dune/test/blackbox-tests/test-cases/install-with-var.t/run.t

Line 67 in 352ec1f

$ dune build @install
are failing after this change. @rgrinberg those tests mention that one of the reasons for not allowing %env in the dst of install paths in the Bin section is to prevent circular dependencies, and I think this change will prevent those circular dependencies. Should we start allowing %env in that position and remove these tests, or are there other reasons for forbidding it?

Interestingly I could reproduce the failure locally earlier today but I can no longer reproduce it locally.

@gridbugs gridbugs force-pushed the fix-bin-install-glob-dependency-cycle branch from 40e3da6 to 1bd6f66 Compare December 23, 2022 07:19
@rgrinberg
Copy link
Member

are failing after this change. @rgrinberg those tests mention that one of the reasons for not allowing %env in the dst of install paths in the Bin section is to prevent circular dependencies, and I think this change will prevent those circular dependencies. Should we start allowing %env in that position and remove these tests, or are there other reasons for forbidding it?

It appears like a good idea to lift the limitation indeed. At this point, we should probably lift all such limitations and let the user write cycles and take advantage of our general cycle detection algorithm guide them.

But we should still try to lift this limitation conditionally, so that users need to enable a newer version of the dune language to enable this.

@gridbugs
Copy link
Collaborator Author

are failing after this change. @rgrinberg those tests mention that one of the reasons for not allowing %env in the dst of install paths in the Bin section is to prevent circular dependencies, and I think this change will prevent those circular dependencies. Should we start allowing %env in that position and remove these tests, or are there other reasons for forbidding it?

It appears like a good idea to lift the limitation indeed. At this point, we should probably lift all such limitations and let the user write cycles and take advantage of our general cycle detection algorithm guide them.

But we should still try to lift this limitation conditionally, so that users need to enable a newer version of the dune language to enable this.

Sounds good. I think for simplicity then is I'll make the current behaviour produce an explicit error if the user puts globs in the bin section of the install stanza, and in future versions globs will be allowed in bin, as well as expanding environment variables.

@gridbugs gridbugs force-pushed the fix-bin-install-glob-dependency-cycle branch 2 times, most recently from cbcceee to cff5052 Compare December 27, 2022 09:28
@gridbugs
Copy link
Collaborator Author

I've ended up forcing the deferred computation if local binaries immediately after Super_context.all is computed to break the cycle but keep the current behaviour where some checks are performed as a side effect of computing the list of local binaries (checks that environment variables don't appear in certain parts of paths in the bin section of install).

Originally I was planning to look into dropping those checks altogether in future versions of dune but I think for now I'll leave it as is as. I'm not sure that it will be a trivial change and there's more pressing things I should be working on instead.

@rgrinberg
Copy link
Member

Originally I was planning to look into dropping those checks altogether in future versions of dune but I think for now I'll leave it as is as. I'm not sure that it will be a trivial change and there's more pressing things I should be working on instead.

Okay, your work around seems reasonable. How about we leave a comment or create an issue about this limitation so that we can get rid of it for 4.0?

@rgrinberg rgrinberg added this to the 3.7.0 milestone Dec 27, 2022
@gridbugs gridbugs mentioned this pull request Jan 2, 2023
Open
@gridbugs gridbugs force-pushed the fix-bin-install-glob-dependency-cycle branch from cff5052 to 4f3cb0c Compare January 2, 2023 13:13
@gridbugs
Copy link
Collaborator Author

gridbugs commented Jan 2, 2023

I've made this issue to look into removing the limitation on variable expansion in dst of files in the bin section: #6818

@emillon
Copy link
Collaborator

emillon commented Jan 30, 2023

@gridbugs ping - do you want to push this past the finish line for 3.7 which is due soon or do you want to postpone to 3.8?

@gridbugs
Defer computation of local_bins
352f207 
This fixes a memo dependency cycle between evaluating globs in install
stanzas and populating the artifacts database. Populating the artifacts
database involves enumerating all files installed in the "bin" section
which involves expanding globs as these files can be specified as globs
rather than literal files. Expanding globs in the install stanza
requires loading the rules for the directory containing the glob, and
doing so depends on the artifacts database.

Signed-off-by: Stephen Sherratt <stephen@sherra.tt>
@gridbugs gridbugs force-pushed the fix-bin-install-glob-dependency-cycle branch from 4f3cb0c to 352f207 Compare February 1, 2023 07:05
@gridbugs
Copy link
Collaborator Author

gridbugs commented Feb 1, 2023

@gridbugs ping - do you want to push this past the finish line for 3.7 which is due soon or do you want to postpone to 3.8?

Updated. Thanks for the reminder!

@emillon
Copy link
Collaborator

emillon commented Feb 1, 2023 •
edited
Loading

Thanks. I edited the original message to xref #6708 - it wasn't being linked.

@rgrinberg rgrinberg merged commit d90ec87 into ocaml:main Feb 1, 2023
@emillon emillon mentioned this pull request Feb 6, 2023
Closed
emillon added a commit to emillon/opam-repository that referenced this pull request Feb 6, 2023
@emillon
[new release] xdg, stdune, ordering, ocamlc-loc, dyn, dune, dune-site…
4eab166 
…, dune-rpc, dune-rpc-lwt, dune-private-libs, dune-glob, dune-configurator, dune-build-info, dune-action-plugin and chrome-trace (3.7.0~alpha1)

CHANGES:

- Fix parsing of OCaml errors that contain code excerpts with `...` in them.
  (ocaml/dune#7008, @rgrinberg)

- Pre-emptively clear screen in watch mode (ocaml/dune#6987, fixes ocaml/dune#6884, @rgrinberg)

- Fix cross compilation configuration when a context with targets is itself a
  host of another context (ocaml/dune#6958, fixes ocaml/dune#6843, @rgrinberg)

- Fix parsing of the `<=` operator in *blang* expressions of `dune` files.
  Previously, the operator would be interpreted as `,`. (ocaml/dune#6928, @tatchi)

- Fix `--trace-file` output. Dune now emits a single *complete* event for every
  executed process. Unterminated *async* events are no longer written. (ocaml/dune#6892,
  @rgrinberg)

- Fix preprocessing with `staged_pps` (ocaml/dune#6748, fixes ocaml/dune#6644, @rgrinberg)

- Use colored output with MDX when Dune colors are enabled.
  (ocaml/dune#6462, @MisterDA)

- Make `dune describe workspace` return consistent dependencies for
  executables and for libraries. By default, compile-time dependencies
  towards PPX-rewriters are from now not taken into account (but
  runtime dependencies always are). Compile-time dependencies towards
  PPX-rewriters can be taken into account by providing the
  `--with-pps` flag. (ocaml/dune#6727, fixes ocaml/dune#6486, @esope)

- Print missing newline after `$ dune exec`. (ocaml/dune#6821, fixes ocaml/dune#6700, @rgrinberg,
  @Alizter)

- Fix binary corruption when installing or promoting in parallel (ocaml/dune#6669, fixes
  ocaml/dune#6668, @edwintorok)

- Use colored output with GCC and Clang when compiling C stubs. The
  flag `-fdiagnostics-color=always` is added to the `:standard` set of
  flags. (ocaml/dune#4083, @MisterDA)

- Fix the parsing of decimal and hexadecimal escape literals in `dune`,
  `dune-package`, and other dune s-expression based files (ocaml/dune#6710, @shym)

- Report an error if `dune init ...` would create a "dune" file in a location
  which already contains a "dune" directory (ocaml/dune#6705, @gridbugs)

- Fix the parsing of alerts. They will now show up in diagnostics correctly.
  (ocaml/dune#6678, @rginberg)

- Fix the compilation of modules generated at link time when
  `implicit_transitive_deps` is enabled (ocaml/dune#6642, @rgrinberg)

- Allow `$ dune utop` to load libraries defined in data only directories
  defined using `(subdir ..)` (ocaml/dune#6631, @rgrinberg)

- Format dune files when they are named `dune-file`. This occurs when we enable
  the alternative file names project option. (ocaml/dune#6566, @rgrinberg)

- Move `$ dune ocaml-merlin -dump-config=$dir` to `$ dune ocaml merlin
  dump-config $dir`. (ocaml/dune#6547, @rgrinberg)

- Allow compilation rules to be impacted by `(env ..)` stanzas that modify the
  environment or set binaries. (ocaml/dune#6527, @rgrinberg)

- Coq native mode is now automatically detected by Dune starting with Coq lang
  0.7. `(mode native)` has been deprecated in favour of detection from the
  configuration of Coq. (ocaml/dune#6409, @Alizter)

- Print "Leaving Directory" whenever "Entering Directory" is printed. (ocaml/dune#6149,
  fixes ocaml/dune#138, @cpitclaudel, @rgrinberg)

- Allow `$ dune ocaml dump-dot-merlin` to run in watch mode. Also this command
  shouldn't print "Entering Directory" mesages. (ocaml/dune#6497, @rgrinberg)

- `dune clean` should no longer fail under Windows due to the inability to
  remove the `.lock` file. Also, bring the implementation of the global lock
  under Windows closer to that of Unix. (ocaml/dune#6523, @nojb)

- Remove "Entering Directory" messages for `$ dune install`. (ocaml/dune#6513,
  @rgrinberg)

- Stop passing `-q` flag in `dune coq top`, which allows for `.coqrc` to be
  loaded. (ocaml/dune#6848, fixes ocaml/dune#6847, @Alizter)

- Fix missing dependencies when detecting the kind of C compiler we're using
  (ocaml/dune#6610, fixes ocaml/dune#6415, @emillon)

- Allow `(include_subdirs qualified)` for OCaml projects. (ocaml/dune#6594, fixes ocaml/dune#1084,
  @rgrinberg)

- Accurately determine merlin configuration for all sources selected with
  `copy#` and `copy_files#`. The old heuristic of looking for a module in
  parent directories is removed (ocaml/dune#6594, @rgrinberg)

- Fix inline tests with *js_of_ocaml* and whole program compilation mode
  enabled (ocaml/dune#6645, @hhugo)

- Fix *js_of_ocaml* separate compilation rules when `--enable=effects`
  ,`--enable=use-js-string` or `--toplevel` is used. (ocaml/dune#6714, ocaml/dune#6828, ocaml/dune#6920, @hhugo)

- Fix *js_of_ocaml* separate compilation in presence of linkall (ocaml/dune#6832, ocaml/dune#6916, @hhugo)

- Remove spurious build dir created when running `dune init proj ...` (ocaml/dune#6707,
  fixes ocaml/dune#5429, @gridbugs)

- Allow `--sandbox` to affect `ocamldep` invocations. Previously, they were
  wrongly marked as incompatible (ocaml/dune#6749, @rgrinberg)

- Validate the command line arguments for `$ dune ocaml top-module`. This
  command requires one positional argument (ocaml/dune#6796, fixes ocaml/dune#6793, @rgrinberg)

- Add a `dune cache size` command for displaying the size of the cache (ocaml/dune#6638,
  @Alizter)

- Add 4.14.0 MSVC to CI (ocaml/dune#6917, @jonahbeckford)

- Fix dependency cycle when installing files to the bin section with
  `glob_files` (ocaml/dune#6764, fixes ocaml/dune#6708, @gridbugs)

- Handle "Too many links" errors when using Dune cache on Windows (ocaml/dune#6993, @nojb)
@emillon emillon mentioned this pull request Feb 17, 2023
Merged
emillon added a commit to emillon/opam-repository that referenced this pull request Feb 17, 2023
@emillon
[new release] xdg, stdune, ordering, ocamlc-loc, dyn, dune, dune-site…
c5fc79e 
…, dune-rpc, dune-rpc-lwt, dune-private-libs, dune-glob, dune-configurator, dune-build-info, dune-action-plugin and chrome-trace (3.7.0)

CHANGES:

- Allow running `$ dune exec` in watch mode (with the `-w` flag). In watch mode,
  `$ dune exec` the executed binary whenever it is recompiled. (ocaml/dune#6966,
  @gridbugs)

- `coqdep` is now called once per theory, instead of one time per Coq
  file. This should significantly speed up some builds, as `coqdep`
  startup time is often heavy (ocaml/dune#7048, @Alizter, @ejgallego)

- Add `map_workspace_root` dune-project stanza to allow disabling of
  mapping of workspace root to `/workspace_root`. (ocaml/dune#6988, fixes ocaml/dune#6929,
  @richardlford)

- Fix handling of support files generated by odoc. (ocaml/dune#6913, @jonludlam)

- Fix parsing of OCaml errors that contain code excerpts with `...` in them.
  (ocaml/dune#7008, @rgrinberg)

- Pre-emptively clear screen in watch mode (ocaml/dune#6987, fixes ocaml/dune#6884, @rgrinberg)

- Fix cross compilation configuration when a context with targets is itself a
  host of another context (ocaml/dune#6958, fixes ocaml/dune#6843, @rgrinberg)

- Fix parsing of the `<=` operator in *blang* expressions of `dune` files.
  Previously, the operator would be interpreted as `<`. (ocaml/dune#6928, @tatchi)

- Fix `--trace-file` output. Dune now emits a single *complete* event for every
  executed process. Unterminated *async* events are no longer written. (ocaml/dune#6892,
  @rgrinberg)

- Fix preprocessing with `staged_pps` (ocaml/dune#6748, fixes ocaml/dune#6644, @rgrinberg)

- Use colored output with MDX when Dune colors are enabled.
  (ocaml/dune#6462, @MisterDA)

- Make `dune describe workspace` return consistent dependencies for
  executables and for libraries. By default, compile-time dependencies
  towards PPX-rewriters are from now not taken into account (but
  runtime dependencies always are). Compile-time dependencies towards
  PPX-rewriters can be taken into account by providing the
  `--with-pps` flag. (ocaml/dune#6727, fixes ocaml/dune#6486, @esope)

- Print missing newline after `$ dune exec`. (ocaml/dune#6821, fixes ocaml/dune#6700, @rgrinberg,
  @Alizter)

- Fix binary corruption when installing or promoting in parallel (ocaml/dune#6669, fixes
  ocaml/dune#6668, @edwintorok)

- Use colored output with GCC and Clang when compiling C stubs. The
  flag `-fdiagnostics-color=always` is added to the `:standard` set of
  flags. (ocaml/dune#4083, @MisterDA)

- Fix the parsing of decimal and hexadecimal escape literals in `dune`,
  `dune-package`, and other dune s-expression based files (ocaml/dune#6710, @shym)

- Report an error if `dune init ...` would create a "dune" file in a location
  which already contains a "dune" directory (ocaml/dune#6705, @gridbugs)

- Fix the parsing of alerts. They will now show up in diagnostics correctly.
  (ocaml/dune#6678, @rginberg)

- Fix the compilation of modules generated at link time when
  `implicit_transitive_deps` is enabled (ocaml/dune#6642, @rgrinberg)

- Allow `$ dune utop` to load libraries defined in data only directories
  defined using `(subdir ..)` (ocaml/dune#6631, @rgrinberg)

- Format dune files when they are named `dune-file`. This occurs when we enable
  the alternative file names project option. (ocaml/dune#6566, @rgrinberg)

- Move `$ dune ocaml-merlin -dump-config=$dir` to `$ dune ocaml merlin
  dump-config $dir`. (ocaml/dune#6547, @rgrinberg)

- Allow compilation rules to be impacted by `(env ..)` stanzas that modify the
  environment or set binaries. (ocaml/dune#6527, @rgrinberg)

- Coq native mode is now automatically detected by Dune starting with Coq lang
  0.7. `(mode native)` has been deprecated in favour of detection from the
  configuration of Coq. (ocaml/dune#6409, @Alizter)

- Print "Leaving Directory" whenever "Entering Directory" is printed. (ocaml/dune#6419,
  fixes ocaml/dune#138, @cpitclaudel, @rgrinberg)

- Allow `$ dune ocaml dump-dot-merlin` to run in watch mode. Also this command
  shouldn't print "Entering Directory" mesages. (ocaml/dune#6497, @rgrinberg)

- `dune clean` should no longer fail under Windows due to the inability to
  remove the `.lock` file. Also, bring the implementation of the global lock
  under Windows closer to that of Unix. (ocaml/dune#6523, @nojb)

- Remove "Entering Directory" messages for `$ dune install`. (ocaml/dune#6513,
  @rgrinberg)

- Stop passing `-q` flag in `dune coq top`, which allows for `.coqrc` to be
  loaded. (ocaml/dune#6848, fixes ocaml/dune#6847, @Alizter)

- Fix missing dependencies when detecting the kind of C compiler we're using
  (ocaml/dune#6610, fixes ocaml/dune#6415, @emillon)

- Allow `(include_subdirs qualified)` for OCaml projects. (ocaml/dune#6594, fixes ocaml/dune#1084,
  @rgrinberg)

- Accurately determine merlin configuration for all sources selected with
  `copy#` and `copy_files#`. The old heuristic of looking for a module in
  parent directories is removed (ocaml/dune#6594, @rgrinberg)

- Fix inline tests with *js_of_ocaml* and whole program compilation mode
  enabled (ocaml/dune#6645, @hhugo)

- Fix *js_of_ocaml* separate compilation rules when `--enable=effects`
  ,`--enable=use-js-string` or `--toplevel` is used. (ocaml/dune#6714, ocaml/dune#6828, ocaml/dune#6920, @hhugo)

- Fix *js_of_ocaml* separate compilation in presence of linkall (ocaml/dune#6832, ocaml/dune#6916, @hhugo)

- Remove spurious build dir created when running `dune init proj ...` (ocaml/dune#6707,
  fixes ocaml/dune#5429, @gridbugs)

- Allow `--sandbox` to affect `ocamldep` invocations. Previously, they were
  wrongly marked as incompatible (ocaml/dune#6749, @rgrinberg)

- Validate the command line arguments for `$ dune ocaml top-module`. This
  command requires one positional argument (ocaml/dune#6796, fixes ocaml/dune#6793, @rgrinberg)

- Add a `dune cache size` command for displaying the size of the cache (ocaml/dune#6638,
  @Alizter)

- Fix dependency cycle when installing files to the bin section with
  `glob_files` (ocaml/dune#6764, fixes ocaml/dune#6708, @gridbugs)

- Handle "Too many links" errors when using Dune cache on Windows (ocaml/dune#6993, @nojb)

- Allow the `cinaps` stanza to set a custom alias. By default, if the alias is
  not set then the cinaps actions will be attached to both `@cinaps` and
  `@runtest` (ocaml/dune#6991, @rgrinberg)

- Add `(using ctypes 0.3)`. When used, paths in `(ctypes)` are interpreted
  relative to where the stanza is defined. (ocaml/dune#6883, fixes ocaml/dune#5325, @emillon)

- Auto-detect `dune-workspace` files as `dune` files in Emacs (ocaml/dune#7061,
  @ilankri)

- Add native support for polling mode on Windows (ocaml/dune#7010, @yams-yams, @nojb)
@gridbugs gridbugs deleted the fix-bin-install-glob-dependency-cycle branch October 11, 2023 03:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rgrinberg rgrinberg rgrinberg left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
3.7.0
Development

Successfully merging this pull request may close these issues.

Dependency cycle when using glob_files to install files to bin section
3 participants
@gridbugs @rgrinberg @emillon