Fails to lookup opam installed libs

[dbuenzli]

According to @samoht ocp-index should look into opam installed libs. This doesn't seem to be the case here:

> ocp-index complete Tsdl.Sdl.init
> 

But

> ocp-index complete -I `opam config var tsdl:lib` Tsdl.Sdl.init
Tsdl.Sdl.init val Init.t -> unit result
    {{:http://wiki.libsdl.org/SDL_Init}SDL_Init}
Tsdl.Sdl.init_sub_system val Init.t -> unit result
    {{:http://wiki.libsdl.org/SDL_InitSubSystem}SDL_InitSubSystem}
> 

with

> ocp-index --version
1.0.0

[dbuenzli]

Btw. a command to print the lookup paths would be nice. (e.g. `diagnose``)

[AltGr]

Thanks a lot for the feedback !
I confirm this is a regression: it actually works as soon as you're within a project but fails when the project root is not found ; fixing that, it shouldn't take long.
You can get (some) information by setting OCP_DEBUG to 1, but true, the search paths would be useful.

[dbuenzli]

I'm not sure the problem is only here. For example in one directory of my projects it works:

> ocp-index complete Tsdl.Sdl.init
Tsdl.Sdl.init val Init.t -> unit result
    {{:http://wiki.libsdl.org/SDL_Init}SDL_Init}
Tsdl.Sdl.init_sub_system val Init.t -> unit result
    {{:http://wiki.libsdl.org/SDL_InitSubSystem}SDL_InitSubSystem}

But in emacs when I edit a file in this directory in doesn't complete. Do you have any ints on how to debug in emacs ?

[AltGr]

Should be fixed by 34618b7. Can you confirm ?

[dbuenzli]

Yes it works now when I'm outside a project on the command line. However it doesn't work in my emacs.

It seems I get only auto-completions available from the current project (taken from _build I guess) and not from other packages (stdlib included).

When I try to complete a non-project identifier (e.g. Hashtbl), the mode line says "auto-complete-mode is not enabled" (but it is when I auto-complete identifiers from the project). Here's what I have in my .emacs:

; Auto complete mode
(add-to-list 'load-path "~/.emacs.d/auto-complete-mode")
(require 'auto-complete-config)
(add-to-list 'ac-dictionary-directories 
             "~/.emacs.d/auto-complete-mode/ac-dict")
(ac-config-default)
(setq ac-auto-start nil)

...

;; caml-mode
(add-to-list 'load-path "~/.emacs.d/caml-mode")
(add-to-list 'auto-mode-alist '("\\.ml[iylp]?$" . caml-mode))
(autoload 'caml-mode "caml" "Major mode for editing OCaml code." t)
(autoload 'run-caml "inf-caml" "Run an inferior OCaml process." t)
(autoload 'camldebug "camldebug" "Run ocamldebug on program." t)
(add-to-list 'interpreter-mode-alist '("ocamlrun" . caml-mode))
(add-to-list 'interpreter-mode-alist '("ocaml" . caml-mode))
(if window-system (require 'caml-font))

; Access to opam's emacs's site-lisp
(add-to-list 'load-path 
             (concat
              (replace-regexp-in-string "\n$" ""
               (shell-command-to-string "opam config var share"))
              "/emacs/site-lisp"))

; ocp-indent 
(require 'ocp-indent)
(setq ocp-indent-path 
      (concat 
       (replace-regexp-in-string "\n$" ""
          (shell-command-to-string "opam config var bin")) "/ocp-indent"))
(setq ocp-indent-config "strict_with=always,match_clause=4,strict_else=never")

; ocp-index
(require 'ocp-index)
(setq ocp-index-path
      (concat 
       (replace-regexp-in-string "\n$" ""
        (shell-command-to-string "opam config var bin")) "/ocp-index"))

[dbuenzli]

Ok now I would really like to fix that. However it seems that ocp-index is not pinable at the moment:

> opam install ocp-index
The following actions will be performed:
 - install ocp-index.pinned
1 to install | 0 to reinstall | 0 to upgrade | 0 to downgrade | 0 to remove

  Installing ocp-index.pinned 
ocp-index.pinned Fetching https://github.com/OCamlPro/ocp-index.git
Building ocp-index.pinned:
  ./configure --prefix /Users/dbuenzli/.opam/4.00.1
  make
  make install
[ERROR] The compilation of ocp-index.pinned failed.
Removing ocp-index.pinned.
  ./configure --prefix /Users/dbuenzli/.opam/4.00.1
  make uninstall
[WARNING] failure in package uninstall script, some files may remain:
# opam-version 1.1.0
# os           darwin
# command      make uninstall
# path         /Users/dbuenzli/.opam/4.00.1/build/ocp-index.pinned
# compiler     4.00.1
# exit-code    2
# env-file     /Users/dbuenzli/.opam/4.00.1/build/ocp-index.pinned/ocp-index-60406-d8a4ae.env
# stdout-file  /Users/dbuenzli/.opam/4.00.1/build/ocp-index.pinned/ocp-index-60406-d8a4ae.out
# stderr-file  /Users/dbuenzli/.opam/4.00.1/build/ocp-index.pinned/ocp-index-60406-d8a4ae.err
### stdout ###
# ...[truncated]
# checking for ocamldoc... ocamldoc
# checking for ocamlbuild... ocamlbuild
# checking for ocamlfind... ocamlfind
# checking for OCaml findlib package cmdliner... found
# checking for OCaml findlib package curses... not found
# configure: creating ./config.status
# config.status: creating Makefile.config
# 
# Executables will be installed in /Users/dbuenzli/.opam/4.00.1/bin
# opam-installer remove ocp-index.install
### stderr ###
# make: opam-installer: No such file or directory
# make: *** [uninstall] Error 1



===== ERROR while installing ocp-index.pinned =====
# opam-version 1.1.0
# os           darwin
# command      make
# path         /Users/dbuenzli/.opam/4.00.1/build/ocp-index.pinned
# compiler     4.00.1
# exit-code    2
# env-file     /Users/dbuenzli/.opam/4.00.1/build/ocp-index.pinned/ocp-index-60406-c61e7b.env
# stdout-file  /Users/dbuenzli/.opam/4.00.1/build/ocp-index.pinned/ocp-index-60406-c61e7b.out
# stderr-file  /Users/dbuenzli/.opam/4.00.1/build/ocp-index.pinned/ocp-index-60406-c61e7b.err
### stdout ###
# ...[truncated]
# checking for ocamldoc... ocamldoc
# checking for ocamlbuild... ocamlbuild
# checking for ocamlfind... ocamlfind
# checking for OCaml findlib package cmdliner... found
# checking for OCaml findlib package curses... not found
# configure: creating ./config.status
# config.status: creating Makefile.config
# 
# Executables will be installed in /Users/dbuenzli/.opam/4.00.1/bin
# ocp-build -init -install-lib /Users/dbuenzli/.opam/4.00.1/lib/ocp-index
### stderr ###
# ...[truncated]
#    Incomplete package "ocp-index-lib" missed by 3 packages
#    ocp-grep (program,ocp)
#      in /Users/dbuenzli/.opam/4.00.1/build/ocp-index.pinned/src
#    ocp-browser (program,ocp)
#      in /Users/dbuenzli/.opam/4.00.1/build/ocp-index.pinned/src
#    ocp-index (program,ocp)
#      in /Users/dbuenzli/.opam/4.00.1/build/ocp-index.pinned/src
# Error: project contains no targets
#   Are your .ocp files empty ?
# make: *** [ocp-build.root] Error 2

'opam install ocp-index' failed.

[AltGr]

Ah sorry, I've been trying to reproduce but without success, this is quite weird :/
I've just been updating the build system, the packages now need new OPAM definitions that I just put online at https://github.com/AltGr/opam-repo-dev

[dbuenzli]

Wouldn't it be easier to add an opam file at the root of your projects to make them reliably pinnable ?

[dbuenzli]

I think that asking your users to add an opam repository to test things is less convenient (e.g. it will try to update all the other tools).

[AltGr]

err, yeah, I was focused on having the repo work properly, but that's the best solution in the meantime, right.
This should all be released quite soon to avoid the hassle
(I put the tmp repo online just so that you have access to the opam files and don't get stuck, half-expecting you to copy the opam files to your ~/.opam/<switch>/overlay/ocp-inde* ; the newer ocp-index will need the newer ocp-indent as well, since the libs are now defined as findlib packages: ocp-indent.lib instead of ocp-indent-lib)

Sorry for the trouble+noise, bad timing :). Adding the opam files to the repos now

[dbuenzli]

There's still a problem:

> opam pin
ocp-indent           git      https://github.com/OCamlPro/ocp-indent.git
ocp-index            git      https://github.com/OCamlPro/ocp-index.git
> opam install ocp-index
[NOTE] /Users/dbuenzli/.opam/4.00.1/packages.dev/ocp-indent.pinned exists although ocp-indent.pinned is not installed. Removing it.
The following actions will be performed:
 - install ocp-indent.pinned [required by ocp-index]
 - install ocp-index.pinned
2 to install | 0 to reinstall | 0 to upgrade | 0 to downgrade | 0 to remove
Do you want to continue ? [Y/n] Y

  Installing ocp-indent.pinned 
ocp-indent.pinned Fetching https://github.com/OCamlPro/ocp-indent.git
Building ocp-indent.pinned:
  ./configure --prefix /Users/dbuenzli/.opam/4.00.1
  make
Installing ocp-indent.pinned.
[ERROR] The compilation of ocp-indent.pinned failed.
Removing ocp-indent.pinned.
  Nothing to do.

[ERROR] Due to some errors while processing ocp-indent.pinned, the following actions will NOT proceed:
 - install ocp-index.pinned

===== ERROR while installing ocp-indent.pinned =====
Internal error:
  While installing the following files:
 - man/man1/ocp-indent.1 in /Users/dbuenzli/.opam/4.00.1/man
'opam install ocp-index' failed.

[AltGr]

Ok, that should be fixed now ! Thanks for your patience and for the reports :)

[dbuenzli]

Ok. So I still have the same problem, the only completions I get are the ones per project (from the _build dir), all the other fail (e.g. List.<TAB> the cursor becomes red, nothing is shown/logged). Is there some kind of log I can look at ?

[dbuenzli]

It does work in the terminal though, it's really in emacs that it doesn't work.

[AltGr]

M-x ocp-index-debug-mode should print out the calls to ocp-index in Messages

[AltGr]

Then you should feed it your file as stdin
(eg ocp-index complete --show=types --full-open IndexBuild --context : --sexp List. <current_file.ml)

[dbuenzli]

This is really weird. The calls for List are the same for the modules that are in my _build. The calls work in the terminal but not in emacs:

> /Users/dbuenzli/.opam/4.00.1/bin/ocp-index complete --show=types --full-open Test --context : --sexp Dic < test.ml 
(
  ("Dicomm" (:name . "Dicomm") (:type . "") (:kind . "module"))
  ("Dicomtrip" (:name . "Dicomtrip") (:type . "") (:kind . "module"))
)
> /Users/dbuenzli/.opam/4.00.1/bin/ocp-index complete --show=types --full-open Test --context : --sexp Lis < test.ml 
(
  ("List" (:name . "List") (:type . "") (:kind . "module"))
  ("ListLabels" (:name . "ListLabels") (:type . "") (:kind . "module"))
  ("Listbox" (:name . "Listbox") (:type . "") (:kind . "module"))
)

[dbuenzli]

To be clear, the first call works, the second one doesn't.

[AltGr]

It must be something in your emacs environment variables then. Depending on how you run emacs the proper OPAM initialization might not be done there ?

[AltGr]

Could you try M-! opam config var lib & ? It's what ocp-index uses

[dbuenzli]

returns /Users/dbuenzli/.opam/4.00.1/lib. Besides if I try:

M-! /Users/dbuenzli/.opam/4.00.1/bin/ocp-index complete --show=types --full-open Test --context : --sexp Lis < test.ml

It returns

(
  ("List" (:name . "List") (:type . "") (:kind . "module"))
  ("ListLabels" (:name . "ListLabels") (:type . "") (:kind . "module"))
  ("Listbox" (:name . "Listbox") (:type . "") (:kind . "module"))
)

[dbuenzli]

Mmmh I see you use call-process-region, that may be the issue because on osx it's a little bit hard to have a good environment, see this.

[dbuenzli]

Ok so here I have:

M-x eval-expression (call-process "opam" nil t) =>
Debugger entered--Lisp error: (file-error "Searching for program" "No such file or directory" "opam")
call-process("opam" nil t)

But:

M-x eval-expresssion (call-process-shell-command "opam" nil t) =>
usage: opam [--version]
            [--help]
            <command> [<args>]

The most commonly used opam commands are:
    init         Initialize OPAM state.
    list         Display the list of available packages.
    info         Display information about specific packages.

So that must be the issue. We need to find a way so that we can work correctly on osx with these damned environment variables.

[dbuenzli]

What I like about the approach mentioned is that you don't have to kill your emacs whenever you switch. Would it be an absolute horror to wrap the ocp-{index,indent} calls with emacs' shell commands ?

This would also kill these lines from my .emacs (I thought that by having them would solve the environment problem, not the dynamic switching though, but I didn't think ocp-index would itself call opam):

(setq ocp-indent-path 
      (concat 
       (replace-regexp-in-string "\n$" ""
          (shell-command-to-string "opam config var bin")) "/ocp-indent"))

(setq ocp-index-path
      (concat 
       "sh -lc "
       (replace-regexp-in-string "\n$" ""
        (shell-command-to-string "opam config var bin")) "/ocp-index"))

[dbuenzli]

Ok so here's my proposal. By having

(setq shell-command-switch "-lc")

in your .emacs it ensure that opam is correctly initialized with your current switch when you use shell commands. If we invoke ocp-index as follows in ocp-index.el we are sure we always have the curren't switch's ocp-indent (this is modeled after what is done in call-process-shell-command)

> diff -u ocp-index.el.old ocp-index.el
--- ocp-index.el.old    2014-01-09 10:11:08.000000000 +0100
+++ ocp-index.el    2014-01-09 10:38:02.000000000 +0100
@@ -83,11 +83,12 @@
     cmd))

 (defun ocp-index-run (cmd &rest args)
-  (let ((args (apply 'ocp-index-args cmd args)))
-    (with-output-to-string
-      (apply 'call-process-region (point-min) (point) ocp-index-path
-             nil (list standard-output nil) nil
-             args))))
+  (let* ((args (apply 'ocp-index-args cmd args))
+         (shell-cmd (mapconcat 'identity (list* ocp-index-path args) " ")))
+         (with-output-to-string
+           (apply 'call-process-region (point-min) (point) shell-file-name
+                  nil (list standard-output nil) nil
+                  (list shell-command-switch shell-cmd)))))

 (defun ac-ocp-index-candidates ()
   (let* ((output (ocp-index-run "complete" "--sexp" ac-prefix))

This also allows me to get rid of that setq ocp-index-path in my .emacs.

[AltGr]

Sounds perfect ! Thanks a lot for helping investigate and solve this :)
I'll integrate this shortly

[dbuenzli]

Your welcome. Btw. could we have the same for ocp-indent here ?

[AltGr]

Yup, I intend to do that too