New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Relocation issues [ocamlfind] #10863

Open
AltGr opened this Issue Nov 27, 2017 · 14 comments

Comments

Projects
None yet
3 participants
@AltGr
Member

AltGr commented Nov 27, 2017

I started testing with binary caching locally. To see how, use latest opam and see https://github.com/ocaml/opam/blob/master/shell/opam-bin-cache.sh (doc at the end).

Of course, many packages aren't relocatable yet, and the above setup allows to check the blockers. My goal here is to improve the situation step by step.

My first experiments didn't hit issues with the bytecode shebangs¹, but with ocamlfind. Hopefully they shouldn't be too difficult to solve:

  • lib/findlib.conf refers to absolute directories. Allowing to make those relative to the conf file would be a solution
  • the settings (destdir and path) can be overriden using OCAMLPATH and OCAMLFIND_DESTDIR, which could be another solution (they can be set from the opam file). However, OCAMLPATH can prepend but not override.
  • setting OCAMLFIND_CONF=/dev/null may work to force an override, though, but is a bit blunt.
  • the directory is also hardcoded in lib/ocaml/topfind and lib/toplevel/topfind. This one can't be overriden, and it turns out it's more problematic (breaks packages using ocaml setup.ml)
  • even after working around all the above, libraries shipped with OCaml are still wrong: it seems setting OCAMLLIB is required even if ocamlc -where has it right.
  • there is also lib/findlib/Makefile.config. Are there packages around that use that ?

¹ using the wrong bytecode interpreter is much less likely to happen and cause problems than linking the wrong libs or trying to install to the wrong place (this setup is combined with my namespacing hooks to ensure the latter doesn't happen)

@AltGr

This comment has been minimized.

Member

AltGr commented Nov 27, 2017

Here is what I added to ocamlfind's opam file to have it work (until now :))

setenv: [
  [OCAMLFIND_CONF = "/dev/null"]
  [OCAMLPATH = "%{lib}%"]
  [OCAMLFIND_DESTDIR = "%{lib}%"]
  [OCAMLLIB = "%{ocaml:lib}%"]
]

Of course, this is opam 2.0 only.
Note that the variables defined this way will also be exported by opam env.

@dbuenzli

This comment has been minimized.

Collaborator

dbuenzli commented Nov 27, 2017

More generally what do you think the strategy should be with programs that have configuration files in etc ? E.g for odig should I simply add:

setenv: [
   ODIG_CONF="%{etc}%/odig.conf"
]

to the opam file ?

@AltGr

This comment has been minimized.

Member

AltGr commented Nov 29, 2017

This is a good fallback, but it would be better if we could avoid cluttering the user environment. Best would be to infer it at runtime: would it be possible for odig to call out to opam var etc ? Failing that, maybe you could rely on $OPAM_SWITCH_PREFIX/etc (opam 2 will always define that variable as part of opam env).

EDIT: The difference between the two is that you get the switch as set in the environment (the two may not be in sync if eval $(opam env) wasn't called; for example if you just cd to a dir where there is a local switch)

@dbuenzli

This comment has been minimized.

Collaborator

dbuenzli commented Nov 29, 2017

Best would be to infer it at runtime: would it be possible for odig to call out to opam var etc ?

That doesn't feel a very good solution. While it would be ok for odig to do so given its scope, there are many programs out there for which it would be absurd to call opam at runtime.

@AltGr

This comment has been minimized.

Member

AltGr commented Dec 4, 2017

Note: setting OCAMLLIB = "%{ocaml:lib}%" is a bad idea at the moment. This is due to a limitation of opam when it reverts changes to environment variables: when you switch away from the switch where this was set, assuming that the new switch doesn't set it, rather than undefine the variable, opam will set it to the empty string. And ocamlc doesn't work properly with OCAMLLIB="".

@dbuenzli

This comment has been minimized.

Collaborator

dbuenzli commented Dec 4, 2017

Re my own "problems" I'm wondering if there might be a standard unix env var that distribution have agreed upon that allows to specify a path to configuration (a little bit of search on the web didn't reveal anything so far).

More generally I wonder if @gasche's work on reproducible builds and that BUILD_PATH_PREFIX_MAP specification might be useful in general for the system (so that the build products do not depend in the switch you compiled them, but I don't know exactly which strategy you adopted).

@gasche

This comment has been minimized.

Member

gasche commented Dec 4, 2017

It's a nice coincidence that I spent most of my day today so far, at the Mirage retreat, implementing an OCaml library to deal with BUILD_PATH_PREFIX_MAP (encoding, decoding, rewriting), that I hope to release tonight. (My goal is to propose the code for inclusion in the compiler codebase to make absolute paths in the .cmo debug information resilient to build-path changes, but I'm releasing an external library with the hope to let users use it in their projects.)

@dbuenzli

This comment has been minimized.

Collaborator

dbuenzli commented Dec 4, 2017

@gasche Very cool. Please don't forget about the .cmis and the new --keep-locs default.

@gasche

This comment has been minimized.

Member

gasche commented Dec 4, 2017

A pre-release of the library is available at

https://gitlab.com/gasche/build_path_prefix_map/

I'd like to test its usage in the OCaml compiler in practice (even only in a PR) before making a release.

type path = string
type path_prefix = string
type error_message = string

val encode_prefix : path_prefix -> string
val decode_prefix : string -> (path_prefix, error_message) result

type pair = { target: path_prefix; source : path_prefix }

val encode_pair : pair -> string
val decode_pair : string -> (pair, error_message) result

type map = pair option list

val encode_map : map -> string
val decode_map : string -> (map, error_message) result

val rewrite_opt : map -> path -> path option
(** [rewrite_opt map path] tries to find a source in [map]
    that is a prefix of the input [path]. If it succeeds,
    it replaces this prefix with the corresponding target.
    If it fails, it just returns [None]. *)

val rewrite : map -> path -> path
@AltGr

This comment has been minimized.

Member

AltGr commented Feb 13, 2018

Just a quick note: even with OCAMLPATH, OCAMLLIB, OCAMLFIND_CONF properly defined, ocamlbuild -where returns the wrong result. There seems to be no override in this case. The code refers to ocaml/ocamlbuild#69

@gasche

This comment has been minimized.

Member

gasche commented Feb 14, 2018

The "new" heuristic that is implemented in ocamlbuild is described in https://github.com/ocaml/ocamlbuild/pull/240/files ; it looks at the variable OCAMLLIB dynamically, and at the configure-time values OCAML_LIBDIR and OCAMLBUILD_LIBDIR -- in particular, OCAMLLIB is only used if OCAMLBUILD_LIBDIR is a sub-path of OCAML_LIBDIR, and in that case the returned path is OCAMLLIB + (OCAMLBUILD_LIBDIR - OCAML_LIBDIR).

You should be able to reproduce the fact that, if those variables were properly set up at ocamlbuild-configuration time, OCAMLLIB can be used to influence the setting. For example, on my machine, with a completely standard opam-installation of ocamlbuild (0.11.0 on 4.06.0), I observe:

$ OCAMLLIB='foo/' ocamlbuild -where
foo/ocamlbuild

I am not sure what your setup is, and what is the result you expect, but it looks like moving an ocamlbuild binary from one opam-user to another should work as long that the filesystem organization within the OPAM root is the same.

I guess that maybe the problem is that you want to setup OCAMLLIB to something else than the opam root directory. In that case, This is a hack, but is there a configure-time choice of OCAML_LIBDIR and OCAMLBUILD_LIBDIR such that the difference correctly goes from ocamlc -where to ocamlbuild -where after relocation?

(Of course I would be happy to accept changes to the heuristic if they preserve the current use-cases and make your life easier.)

@AltGr

This comment has been minimized.

Member

AltGr commented Feb 14, 2018

Thanks for the details. I am testing a binary cache of opam packages, i.e. assuming that I can get the installed artefacts of one switch, and put them into another.

In this specific case, I had installed the packages first in a temporary switch /tmp/foo/_opam, then the caching system recovered their installed files and put them into ~/.opam/4.05.0. The above patch to ocamlfind's opam file had been applied, but maybe resulted in a confusing setup that broke ocamlbuild's configuration ? I believe it was the case that ocamlbuild was not taken from cache, but rebuilt on the new switch, against the relocated ocamlfind, and with the variables defined as such:

OCAMLFIND_CONF = "/dev/null"
OCAMLPATH = "~/.opam/4.05.0/lib"
OCAMLFIND_DESTDIR = "~/.opam/4.05.0/lib"
OCAMLLIB = "~/.opam/4.05.0/lib/ocaml"
@gasche

This comment has been minimized.

Member

gasche commented Feb 14, 2018

Could you compile ocamlbuild with --keep-build-dir and get the content of Makefile.config in the build directory? This file records the configure-time value of OCAML_LIBDIR, and a variable LIBDIR that corresponds to what I called OCAMLBUILD_LIBDIR in my message above.

@AltGr

This comment has been minimized.

Member

AltGr commented Feb 16, 2018

I have been trying to, but hitting other issues along the way. I'll get back to it asap :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment