Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relocation issues [ocamlfind] #10863

Open
AltGr opened this issue Nov 27, 2017 · 19 comments
Open

Relocation issues [ocamlfind] #10863

AltGr opened this issue Nov 27, 2017 · 19 comments

Comments

@AltGr
Copy link
Member

@AltGr AltGr commented Nov 27, 2017

I started testing with binary caching locally. To see how, use latest opam and see https://github.com/ocaml/opam/blob/master/shell/opam-bin-cache.sh (doc at the end).

Of course, many packages aren't relocatable yet, and the above setup allows to check the blockers. My goal here is to improve the situation step by step.

My first experiments didn't hit issues with the bytecode shebangs¹, but with ocamlfind. Hopefully they shouldn't be too difficult to solve:

  • lib/findlib.conf refers to absolute directories. Allowing to make those relative to the conf file would be a solution
  • the settings (destdir and path) can be overriden using OCAMLPATH and OCAMLFIND_DESTDIR, which could be another solution (they can be set from the opam file). However, OCAMLPATH can prepend but not override.
  • setting OCAMLFIND_CONF=/dev/null may work to force an override, though, but is a bit blunt.
  • the directory is also hardcoded in lib/ocaml/topfind and lib/toplevel/topfind. This one can't be overriden, and it turns out it's more problematic (breaks packages using ocaml setup.ml)
  • even after working around all the above, libraries shipped with OCaml are still wrong: it seems setting OCAMLLIB is required even if ocamlc -where has it right.
  • there is also lib/findlib/Makefile.config. Are there packages around that use that ?

¹ using the wrong bytecode interpreter is much less likely to happen and cause problems than linking the wrong libs or trying to install to the wrong place (this setup is combined with my namespacing hooks to ensure the latter doesn't happen)

@AltGr
Copy link
Member Author

@AltGr AltGr commented Nov 27, 2017

Here is what I added to ocamlfind's opam file to have it work (until now :))

setenv: [
  [OCAMLFIND_CONF = "/dev/null"]
  [OCAMLPATH = "%{lib}%"]
  [OCAMLFIND_DESTDIR = "%{lib}%"]
  [OCAMLLIB = "%{ocaml:lib}%"]
]

Of course, this is opam 2.0 only.
Note that the variables defined this way will also be exported by opam env.

@dbuenzli
Copy link
Collaborator

@dbuenzli dbuenzli commented Nov 27, 2017

More generally what do you think the strategy should be with programs that have configuration files in etc ? E.g for odig should I simply add:

setenv: [
   ODIG_CONF="%{etc}%/odig.conf"
]

to the opam file ?

@AltGr
Copy link
Member Author

@AltGr AltGr commented Nov 29, 2017

This is a good fallback, but it would be better if we could avoid cluttering the user environment. Best would be to infer it at runtime: would it be possible for odig to call out to opam var etc ? Failing that, maybe you could rely on $OPAM_SWITCH_PREFIX/etc (opam 2 will always define that variable as part of opam env).

EDIT: The difference between the two is that you get the switch as set in the environment (the two may not be in sync if eval $(opam env) wasn't called; for example if you just cd to a dir where there is a local switch)

@dbuenzli
Copy link
Collaborator

@dbuenzli dbuenzli commented Nov 29, 2017

Best would be to infer it at runtime: would it be possible for odig to call out to opam var etc ?

That doesn't feel a very good solution. While it would be ok for odig to do so given its scope, there are many programs out there for which it would be absurd to call opam at runtime.

@AltGr
Copy link
Member Author

@AltGr AltGr commented Dec 4, 2017

Note: setting OCAMLLIB = "%{ocaml:lib}%" is a bad idea at the moment. This is due to a limitation of opam when it reverts changes to environment variables: when you switch away from the switch where this was set, assuming that the new switch doesn't set it, rather than undefine the variable, opam will set it to the empty string. And ocamlc doesn't work properly with OCAMLLIB="".

@dbuenzli
Copy link
Collaborator

@dbuenzli dbuenzli commented Dec 4, 2017

Re my own "problems" I'm wondering if there might be a standard unix env var that distribution have agreed upon that allows to specify a path to configuration (a little bit of search on the web didn't reveal anything so far).

More generally I wonder if @gasche's work on reproducible builds and that BUILD_PATH_PREFIX_MAP specification might be useful in general for the system (so that the build products do not depend in the switch you compiled them, but I don't know exactly which strategy you adopted).

@gasche
Copy link
Member

@gasche gasche commented Dec 4, 2017

It's a nice coincidence that I spent most of my day today so far, at the Mirage retreat, implementing an OCaml library to deal with BUILD_PATH_PREFIX_MAP (encoding, decoding, rewriting), that I hope to release tonight. (My goal is to propose the code for inclusion in the compiler codebase to make absolute paths in the .cmo debug information resilient to build-path changes, but I'm releasing an external library with the hope to let users use it in their projects.)

@dbuenzli
Copy link
Collaborator

@dbuenzli dbuenzli commented Dec 4, 2017

@gasche Very cool. Please don't forget about the .cmis and the new --keep-locs default.

@gasche
Copy link
Member

@gasche gasche commented Dec 4, 2017

A pre-release of the library is available at

https://gitlab.com/gasche/build_path_prefix_map/

I'd like to test its usage in the OCaml compiler in practice (even only in a PR) before making a release.

type path = string
type path_prefix = string
type error_message = string

val encode_prefix : path_prefix -> string
val decode_prefix : string -> (path_prefix, error_message) result

type pair = { target: path_prefix; source : path_prefix }

val encode_pair : pair -> string
val decode_pair : string -> (pair, error_message) result

type map = pair option list

val encode_map : map -> string
val decode_map : string -> (map, error_message) result

val rewrite_opt : map -> path -> path option
(** [rewrite_opt map path] tries to find a source in [map]
    that is a prefix of the input [path]. If it succeeds,
    it replaces this prefix with the corresponding target.
    If it fails, it just returns [None]. *)

val rewrite : map -> path -> path
@AltGr
Copy link
Member Author

@AltGr AltGr commented Feb 13, 2018

Just a quick note: even with OCAMLPATH, OCAMLLIB, OCAMLFIND_CONF properly defined, ocamlbuild -where returns the wrong result. There seems to be no override in this case. The code refers to ocaml/ocamlbuild#69

@gasche
Copy link
Member

@gasche gasche commented Feb 14, 2018

The "new" heuristic that is implemented in ocamlbuild is described in https://github.com/ocaml/ocamlbuild/pull/240/files ; it looks at the variable OCAMLLIB dynamically, and at the configure-time values OCAML_LIBDIR and OCAMLBUILD_LIBDIR -- in particular, OCAMLLIB is only used if OCAMLBUILD_LIBDIR is a sub-path of OCAML_LIBDIR, and in that case the returned path is OCAMLLIB + (OCAMLBUILD_LIBDIR - OCAML_LIBDIR).

You should be able to reproduce the fact that, if those variables were properly set up at ocamlbuild-configuration time, OCAMLLIB can be used to influence the setting. For example, on my machine, with a completely standard opam-installation of ocamlbuild (0.11.0 on 4.06.0), I observe:

$ OCAMLLIB='foo/' ocamlbuild -where
foo/ocamlbuild

I am not sure what your setup is, and what is the result you expect, but it looks like moving an ocamlbuild binary from one opam-user to another should work as long that the filesystem organization within the OPAM root is the same.

I guess that maybe the problem is that you want to setup OCAMLLIB to something else than the opam root directory. In that case, This is a hack, but is there a configure-time choice of OCAML_LIBDIR and OCAMLBUILD_LIBDIR such that the difference correctly goes from ocamlc -where to ocamlbuild -where after relocation?

(Of course I would be happy to accept changes to the heuristic if they preserve the current use-cases and make your life easier.)

@AltGr
Copy link
Member Author

@AltGr AltGr commented Feb 14, 2018

Thanks for the details. I am testing a binary cache of opam packages, i.e. assuming that I can get the installed artefacts of one switch, and put them into another.

In this specific case, I had installed the packages first in a temporary switch /tmp/foo/_opam, then the caching system recovered their installed files and put them into ~/.opam/4.05.0. The above patch to ocamlfind's opam file had been applied, but maybe resulted in a confusing setup that broke ocamlbuild's configuration ? I believe it was the case that ocamlbuild was not taken from cache, but rebuilt on the new switch, against the relocated ocamlfind, and with the variables defined as such:

OCAMLFIND_CONF = "/dev/null"
OCAMLPATH = "~/.opam/4.05.0/lib"
OCAMLFIND_DESTDIR = "~/.opam/4.05.0/lib"
OCAMLLIB = "~/.opam/4.05.0/lib/ocaml"
@gasche
Copy link
Member

@gasche gasche commented Feb 14, 2018

Could you compile ocamlbuild with --keep-build-dir and get the content of Makefile.config in the build directory? This file records the configure-time value of OCAML_LIBDIR, and a variable LIBDIR that corresponds to what I called OCAMLBUILD_LIBDIR in my message above.

@AltGr
Copy link
Member Author

@AltGr AltGr commented Feb 16, 2018

I have been trying to, but hitting other issues along the way. I'll get back to it asap :)

@Pitometsu
Copy link

@Pitometsu Pitometsu commented Jan 28, 2019

why just do not use the Nix environment for that?

@Lupus
Copy link

@Lupus Lupus commented Aug 8, 2019

BUILD_PATH_PREFIX_MAP MR is merged, isn't that feature sufficient to solve relocation? opam-bin-cache.sh just needs to maintain this prefix map and export it down to the individual package building environment? I've tried the bin cache script myself as a way to speedup creation of multiple local switches, and things got broken with ocamlfind after I wiped one of the switches and created it again. When it works, speedup looks really great! Would be awesome to see relocation solved.

@github-actions
Copy link

@github-actions github-actions bot commented Apr 21, 2020

This issue is stale because it has been open 90 days with no activity. Remove the stale label, or comment, or this will be closed in 15 days.

@github-actions github-actions bot added the Stale label Apr 21, 2020
@Khady
Copy link
Contributor

@Khady Khady commented Apr 29, 2020

Please don't close this one dear bot.

@mseri mseri removed the Stale label Apr 29, 2020
@github-actions github-actions bot added the Stale label Jul 28, 2020
@ocaml ocaml deleted a comment from github-actions bot Jul 28, 2020
@mseri mseri removed the Stale label Jul 28, 2020
@github-actions github-actions bot added the Stale label Oct 26, 2020
@ocaml ocaml deleted a comment from github-actions bot Oct 27, 2020
@mseri mseri removed the Stale label Oct 27, 2020
@github-actions
Copy link

@github-actions github-actions bot commented Jan 25, 2021

This issue has been open 90 days with no activity. Consequently, it is being marked with the "stale" label. What this means is that the issue will be automatically closed in 30 days unless more comments are added or the "stale" label is removed. If you come across this issue in the future, you may also find it helpful to visit our forum at https://discuss.ocaml.org where queries related to OCaml package management are very welcome.

@github-actions github-actions bot added the Stale label Jan 25, 2021
@github-actions github-actions bot closed this Feb 24, 2021
@kit-ty-kate kit-ty-kate reopened this Feb 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
8 participants