New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OPAM in a box #1035

Closed
avsm opened this Issue Dec 13, 2013 · 28 comments

Comments

Projects
None yet
6 participants
@avsm
Member

avsm commented Dec 13, 2013

This is a tracker thread to pull together the various scripts and tools required to generate an "OPAM in a box" suitable for teaching purposes. It should result in:

  • A base OCaml installation of 4.01.0 (available via Ubuntu PPA in avsm/ppa)
  • an OPAM installation with a user-configured set of packages all pre-installed, including any external library dependencies (e.g. libpcre). (see https://github.com/AltGr/docker-opam)
  • An .ocamlinit file preconfigured to require all the libraries.
  • The utop toplevel.
  • The merlin tool configured to work with Emacs and Vim.
  • An opam-doc-generated set of cross-referenced HTML docs for all the installed packages, served via a localhost webserver.
  • A local HTTP remote of all the archives, so that it can be used without requiring external net access. (OPAM 1.1.1dev has opam-admin make --resolve <pkgs> to generate this)
  • an iPython/OCaml console running (ocaml/opam-repository#1690)

For a first milestone, we're going to pull this together using Ubuntu and shell scripting, and wrap it for release using Docker containers (see http://anil.recoil.org/2013/11/15/docker-and-opam.html for an overview of this). When this works satisfactorily, we can distribute the Ubuntu+Docker binaries in a VM for those folk using Windows as a second step.

This thread arose from a discussion with various profs and @yminsky, and a followup meeting with @AltGr and @samoht. We'll use this issue to track progress of the various steps above.

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Dec 13, 2013

Member

For opam-doc, see https://github.com/ocamllabs/opam-doc -- that repository takes a package set and generates documentation suitable for serving from a localhost webserver. Cohttp-0.9.14 installs a cohttp-server which simply serves local files over HTTP, so that can be used to serve the documentation too.

Member

avsm commented Dec 13, 2013

For opam-doc, see https://github.com/ocamllabs/opam-doc -- that repository takes a package set and generates documentation suitable for serving from a localhost webserver. Cohttp-0.9.14 installs a cohttp-server which simply serves local files over HTTP, so that can be used to serve the documentation too.

@samoht

This comment has been minimized.

Show comment
Hide comment
@samoht

samoht Dec 13, 2013

Member

A minor remark: using 4.01.0 as system compiler will not work with opam-doc (see ocamllabs/opam-doc#68)

Member

samoht commented Dec 13, 2013

A minor remark: using 4.01.0 as system compiler will not work with opam-doc (see ocamllabs/opam-doc#68)

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Dec 13, 2013

Member

Good point; since we're going to be redistributing binaries, we might as well just compile up a custom switch for 4.01.0 anyway. This could also include the frame-pointer to have better debugging, since this particular one wouldn't be for performance-critical code.

Member

avsm commented Dec 13, 2013

Good point; since we're going to be redistributing binaries, we might as well just compile up a custom switch for 4.01.0 anyway. This could also include the frame-pointer to have better debugging, since this particular one wouldn't be for performance-critical code.

@AltGr

This comment has been minimized.

Show comment
Hide comment
@AltGr

AltGr Dec 13, 2013

Member

The latest version of https://github.com/AltGr/docker-opam provides a straight-forward make.sh script that takes a list of packages and:

  • builds a base docker image with OPAM (note, we could export that to the docker repo)
  • creates a git mirror of the opam repository containing the needed package archives (so that no further access to the individual packages URLs is needed from the docker container ; that step could be skipped though). Note that we don't mirror the depexts (yet).
  • installs the required OPAM packages (together with their depexts)
Member

AltGr commented Dec 13, 2013

The latest version of https://github.com/AltGr/docker-opam provides a straight-forward make.sh script that takes a list of packages and:

  • builds a base docker image with OPAM (note, we could export that to the docker repo)
  • creates a git mirror of the opam repository containing the needed package archives (so that no further access to the individual packages URLs is needed from the docker container ; that step could be skipped though). Note that we don't mirror the depexts (yet).
  • installs the required OPAM packages (together with their depexts)
@AltGr

This comment has been minimized.

Show comment
Hide comment
@AltGr

AltGr Dec 13, 2013

Member

The second point above uses the new opam-admin make --resolve <packages> to mirror the selected packages and all their dependencies, at the best version possible.

Member

AltGr commented Dec 13, 2013

The second point above uses the new opam-admin make --resolve <packages> to mirror the selected packages and all their dependencies, at the best version possible.

@jnfoster

This comment has been minimized.

Show comment
Hide comment
@jnfoster

jnfoster Dec 28, 2013

[Cc @bennn @mdgeorge]

So is this happening? I'm at the point where I need to finalize the VM for spring 2013 CS 3110 at Cornell very soon.

Thanks,
Nate

jnfoster commented Dec 28, 2013

[Cc @bennn @mdgeorge]

So is this happening? I'm at the point where I need to finalize the VM for spring 2013 CS 3110 at Cornell very soon.

Thanks,
Nate

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Dec 28, 2013

Member

It is -- I did some cleanups of the opam-doc code over the vacation, and one version is up here: http://mirage.github.io. There are still a number of layout issues, and the front page needs to be templated (so that you don't just get a raw dump of modules, but a better front page).

So, if you don't mind being on the bleeding edge of some of this, I can create an Ubuntu VM and create the Docker images. Your best VM choice is likely to be Ubuntu Saucy x86_64, as that'll run Docker out of the box.

Member

avsm commented Dec 28, 2013

It is -- I did some cleanups of the opam-doc code over the vacation, and one version is up here: http://mirage.github.io. There are still a number of layout issues, and the front page needs to be templated (so that you don't just get a raw dump of modules, but a better front page).

So, if you don't mind being on the bleeding edge of some of this, I can create an Ubuntu VM and create the Docker images. Your best VM choice is likely to be Ubuntu Saucy x86_64, as that'll run Docker out of the box.

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Dec 28, 2013

Member

Actually, I'm going to give Vagrant a try after seeing @reitblatt's commit in frenetic-lang/frenetic#202. It seems pretty portable and easy to script too, whereas Docker is Linux-only at present.

Member

avsm commented Dec 28, 2013

Actually, I'm going to give Vagrant a try after seeing @reitblatt's commit in frenetic-lang/frenetic#202. It seems pretty portable and easy to script too, whereas Docker is Linux-only at present.

@jnfoster

This comment has been minimized.

Show comment
Hide comment
@jnfoster

jnfoster Dec 28, 2013

Excellent! Added bonus: this would let folks easily customize -- e.g.,
clone the GitHub repo for our assignment harness.

-N

On Sat, Dec 28, 2013 at 3:21 PM, Anil Madhavapeddy <notifications@github.com

wrote:

Actually, I'm going to give Vagrant a try after seeing @reitblatthttps://github.com/reitblatt's
commit in frenetic-lang/frenetic#202frenetic-lang/frenetic#202.
It seems pretty portable and easy to script too, whereas Docker is
Linux-only at present.


Reply to this email directly or view it on GitHubhttps://github.com/ocaml/opam/issues/1035#issuecomment-31304053
.

jnfoster commented Dec 28, 2013

Excellent! Added bonus: this would let folks easily customize -- e.g.,
clone the GitHub repo for our assignment harness.

-N

On Sat, Dec 28, 2013 at 3:21 PM, Anil Madhavapeddy <notifications@github.com

wrote:

Actually, I'm going to give Vagrant a try after seeing @reitblatthttps://github.com/reitblatt's
commit in frenetic-lang/frenetic#202frenetic-lang/frenetic#202.
It seems pretty portable and easy to script too, whereas Docker is
Linux-only at present.


Reply to this email directly or view it on GitHubhttps://github.com/ocaml/opam/issues/1035#issuecomment-31304053
.

@avsm avsm referenced this issue Dec 30, 2013

Merged

script improvements #90

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Dec 30, 2013

Member

I've got a Vagrant repository at https://github.com/avsm/vagrant-opam that serves docs, installs Vim and utop/merlin.

Can any Emacs users help me out with a script for that too?

Member

avsm commented Dec 30, 2013

I've got a Vagrant repository at https://github.com/avsm/vagrant-opam that serves docs, installs Vim and utop/merlin.

Can any Emacs users help me out with a script for that too?

@samoht

This comment has been minimized.

Show comment
Hide comment
@samoht

samoht Dec 30, 2013

Member

Hadn't had much time to test (the compilation time of the whole thing is still a bit high), but I think avsm/vagrant-opam#1 is working (at least for my settings).

Feel free to send a PR on https://github.com/samoht/ocaml-emacs-settings to improve the emacs mode.

Member

samoht commented Dec 30, 2013

Hadn't had much time to test (the compilation time of the whole thing is still a bit high), but I think avsm/vagrant-opam#1 is working (at least for my settings).

Feel free to send a PR on https://github.com/samoht/ocaml-emacs-settings to improve the emacs mode.

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Dec 30, 2013

Member

We have everything except a local remote now, which requires OPAM 1.1.1 -- I'll create some PPA snapshots when I get back to Cambridge later in the week.

Member

avsm commented Dec 30, 2013

We have everything except a local remote now, which requires OPAM 1.1.1 -- I'll create some PPA snapshots when I get back to Cambridge later in the week.

@jnfoster

This comment has been minimized.

Show comment
Hide comment
@jnfoster

jnfoster Dec 31, 2013

Awesome!

What do you think about adding Eclipse support (there's a PPA)? Or should
Cornell make a patch on top of the OPAM Vagrant scripts?

-N

On Mon, Dec 30, 2013 at 5:50 PM, Anil Madhavapeddy <notifications@github.com

wrote:

We have everything except a local remote now, which requires OPAM 1.1.1 --
I'll create some PPA snapshots when I get back to Cambridge later in the
week.


Reply to this email directly or view it on GitHubhttps://github.com/ocaml/opam/issues/1035#issuecomment-31374115
.

jnfoster commented Dec 31, 2013

Awesome!

What do you think about adding Eclipse support (there's a PPA)? Or should
Cornell make a patch on top of the OPAM Vagrant scripts?

-N

On Mon, Dec 30, 2013 at 5:50 PM, Anil Madhavapeddy <notifications@github.com

wrote:

We have everything except a local remote now, which requires OPAM 1.1.1 --
I'll create some PPA snapshots when I get back to Cambridge later in the
week.


Reply to this email directly or view it on GitHubhttps://github.com/ocaml/opam/issues/1035#issuecomment-31374115
.

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Dec 31, 2013

Member

I reckon this should be the kitchen sink edition with everything in. If you could send a pull req with a setup-eclipse.sh script, I'm happy to try it out. I'd like to publish this one on the RWO site as well, and several people asked about Eclipse.

Member

avsm commented Dec 31, 2013

I reckon this should be the kitchen sink edition with everything in. If you could send a pull req with a setup-eclipse.sh script, I'm happy to try it out. I'd like to publish this one on the RWO site as well, and several people asked about Eclipse.

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Jan 1, 2014

Member

@samoht, I see lots of errors on the Vagrant build with the emacs script. It completes successfully, so just checking if they're important or not.

In toplevel form:
auto-complete-config.el:31:1:Error: Cannot open load file: popup
Compiling /home/vagrant/.emacs.d/elpa/auto-complete-20131128.233/auto-complete-pkg.el...
Wrote /home/vagrant/.emacs.d/elpa/auto-complete-20131128.233/auto-complete-pkg.elc
Compiling /home/vagrant/.emacs.d/elpa/auto-complete-20131128.233/auto-complete.el...

In toplevel form:
auto-complete.el:51:1:Error: Cannot open load file: popup
Checking /home/vagrant/.emacs.d/elpa/auto-complete-20131128.233/dict...
Done (Total of 1 file compiled, 2 failed, 1 skipped)
Member

avsm commented Jan 1, 2014

@samoht, I see lots of errors on the Vagrant build with the emacs script. It completes successfully, so just checking if they're important or not.

In toplevel form:
auto-complete-config.el:31:1:Error: Cannot open load file: popup
Compiling /home/vagrant/.emacs.d/elpa/auto-complete-20131128.233/auto-complete-pkg.el...
Wrote /home/vagrant/.emacs.d/elpa/auto-complete-20131128.233/auto-complete-pkg.elc
Compiling /home/vagrant/.emacs.d/elpa/auto-complete-20131128.233/auto-complete.el...

In toplevel form:
auto-complete.el:51:1:Error: Cannot open load file: popup
Checking /home/vagrant/.emacs.d/elpa/auto-complete-20131128.233/dict...
Done (Total of 1 file compiled, 2 failed, 1 skipped)
@samoht

This comment has been minimized.

Show comment
Hide comment
@samoht

samoht Jan 2, 2014

Member

Hum, seems that popup is not correctly installed. I'll add it to the package list.

Member

samoht commented Jan 2, 2014

Hum, seems that popup is not correctly installed. I'll add it to the package list.

@AltGr

This comment has been minimized.

Show comment
Hide comment
@AltGr

AltGr Jan 8, 2014

Member

@samoht hmm yes, if you're using apt-get, popup.el is part of auto-complete-el (http://packages.ubuntu.com/saucy/all/auto-complete-el/filelist) -- so I think this should be alright ; but using the new emacs package system that may not be the case.
I am now testing using apt-get. Any reason why we may prefer one way or the other ?

Member

AltGr commented Jan 8, 2014

@samoht hmm yes, if you're using apt-get, popup.el is part of auto-complete-el (http://packages.ubuntu.com/saucy/all/auto-complete-el/filelist) -- so I think this should be alright ; but using the new emacs package system that may not be the case.
I am now testing using apt-get. Any reason why we may prefer one way or the other ?

@yminsky

This comment has been minimized.

Show comment
Hide comment
@yminsky

yminsky Jan 9, 2014

I'm curious what the status of this is? Is there something in a good enough state for Cornell to use? Has anyone reached out to the other schools on the email thread?

yminsky commented Jan 9, 2014

I'm curious what the status of this is? Is there something in a good enough state for Cornell to use? Has anyone reached out to the other schools on the email thread?

@AltGr

This comment has been minimized.

Show comment
Hide comment
@AltGr

AltGr Jan 9, 2014

Member

The general setup is working, but the editor configuration isn't very nice yet. I am trying to have better emacs support, and ocp-indent support in vim.

I also found that you can run into trouble with the vagrant VM when you have a locale set, which is just a matter of running 'LC_ALL=C vagrant ssh'.

Member

AltGr commented Jan 9, 2014

The general setup is working, but the editor configuration isn't very nice yet. I am trying to have better emacs support, and ocp-indent support in vim.

I also found that you can run into trouble with the vagrant VM when you have a locale set, which is just a matter of running 'LC_ALL=C vagrant ssh'.

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Jan 9, 2014

Member

On 9 Jan 2014, at 11:00, Louis Gesbert notifications@github.com wrote:

I also found that you can run into trouble with the vagrant VM when you have a locale set, which is just a matter of running 'LC_ALL=C vagrant ssh'

We should set it in the VM for now until its fixed upstream; I created avsm/vagrant-opam#3 to track this.

-a

Member

avsm commented Jan 9, 2014

On 9 Jan 2014, at 11:00, Louis Gesbert notifications@github.com wrote:

I also found that you can run into trouble with the vagrant VM when you have a locale set, which is just a matter of running 'LC_ALL=C vagrant ssh'

We should set it in the VM for now until its fixed upstream; I created avsm/vagrant-opam#3 to track this.

-a

@lefessan

This comment has been minimized.

Show comment
Hide comment
@lefessan

lefessan Jan 9, 2014

Contributor

Do you have a deadline from Cornell, or other universities, to know at least when the first VM should be shipped ? It would allow us to schedule the important work first, leaving polishing for later.

Contributor

lefessan commented Jan 9, 2014

Do you have a deadline from Cornell, or other universities, to know at least when the first VM should be shipped ? It would allow us to schedule the important work first, leaving polishing for later.

@jnfoster

This comment has been minimized.

Show comment
Hide comment
@jnfoster

jnfoster Jan 9, 2014

We'd like to get this finalized in one week.

I punted the setup-eclipse.sh task over to my TAs. I'd like them to own
this... I'll nudge them to get a pull request soon.

-N

On Thu, Jan 9, 2014 at 6:04 AM, Anil Madhavapeddy
notifications@github.comwrote:

On 9 Jan 2014, at 11:00, Louis Gesbert notifications@github.com wrote:

I also found that you can run into trouble with the vagrant VM when you
have a locale set, which is just a matter of running 'LC_ALL=C vagrant ssh'

We should set it in the VM for now until its fixed upstream; I created
avsm/vagrant-opam#3 to track this.

-a


Reply to this email directly or view it on GitHubhttps://github.com/ocaml/opam/issues/1035#issuecomment-31920779
.

jnfoster commented Jan 9, 2014

We'd like to get this finalized in one week.

I punted the setup-eclipse.sh task over to my TAs. I'd like them to own
this... I'll nudge them to get a pull request soon.

-N

On Thu, Jan 9, 2014 at 6:04 AM, Anil Madhavapeddy
notifications@github.comwrote:

On 9 Jan 2014, at 11:00, Louis Gesbert notifications@github.com wrote:

I also found that you can run into trouble with the vagrant VM when you
have a locale set, which is just a matter of running 'LC_ALL=C vagrant ssh'

We should set it in the VM for now until its fixed upstream; I created
avsm/vagrant-opam#3 to track this.

-a


Reply to this email directly or view it on GitHubhttps://github.com/ocaml/opam/issues/1035#issuecomment-31920779
.

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Jan 10, 2014

Member

@jnfoster, I'll start publishing binary images before Eclipse is in; it can just be rebuilt when the pull request shows up. @AltGr, if you can branch OPAM 1.1.1, then I can add support for a local package repository, which needs the latest OPAM I believe.

Member

avsm commented Jan 10, 2014

@jnfoster, I'll start publishing binary images before Eclipse is in; it can just be rebuilt when the pull request shows up. @AltGr, if you can branch OPAM 1.1.1, then I can add support for a local package repository, which needs the latest OPAM I believe.

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Jan 15, 2014

Member

I've pushed a number of cleanups and will start publishing binary packages of the boxes tomorrow. @jnfoster, any luck with the Eclipse pull?

The most important part has also been done: the ascii art! avsm/vagrant-opam@0ba2974

Member

avsm commented Jan 15, 2014

I've pushed a number of cleanups and will start publishing binary packages of the boxes tomorrow. @jnfoster, any luck with the Eclipse pull?

The most important part has also been done: the ascii art! avsm/vagrant-opam@0ba2974

@jnfoster

This comment has been minimized.

Show comment
Hide comment
@jnfoster

jnfoster Jan 17, 2014

My TAs are still blocked on that download.camlcity.org bug. I verified
(from a different location so I don't think it's a network issue). Are
others still able to build the VM?

-N

On Wed, Jan 15, 2014 at 4:53 PM, Anil Madhavapeddy <notifications@github.com

wrote:

I've pushed a number of cleanups and will start publishing binary packages
of the boxes tomorrow. @jnfoster https://github.com/jnfoster, any luck
with the Eclipse pull?

The most important part has also been done: the ascii art!
avsm/vagrant-opam@0ba2974avsm/vagrant-opam@0ba2974


Reply to this email directly or view it on GitHubhttps://github.com/ocaml/opam/issues/1035#issuecomment-32419334
.

jnfoster commented Jan 17, 2014

My TAs are still blocked on that download.camlcity.org bug. I verified
(from a different location so I don't think it's a network issue). Are
others still able to build the VM?

-N

On Wed, Jan 15, 2014 at 4:53 PM, Anil Madhavapeddy <notifications@github.com

wrote:

I've pushed a number of cleanups and will start publishing binary packages
of the boxes tomorrow. @jnfoster https://github.com/jnfoster, any luck
with the Eclipse pull?

The most important part has also been done: the ascii art!
avsm/vagrant-opam@0ba2974avsm/vagrant-opam@0ba2974


Reply to this email directly or view it on GitHubhttps://github.com/ocaml/opam/issues/1035#issuecomment-32419334
.

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Jan 17, 2014

Member

That was fixed last week to point to a mirror. I've done a fresh build, and I only ran into a disk space issue which I'm figuring out how to fix atm by adding a build disk.

On 17 Jan 2014, at 05:40, Nate Foster notifications@github.com wrote:

My TAs are still blocked on that download.camlcity.org bug. I verified
(from a different location so I don't think it's a network issue). Are
others still able to build the VM?

-N

On Wed, Jan 15, 2014 at 4:53 PM, Anil Madhavapeddy <notifications@github.com

wrote:

I've pushed a number of cleanups and will start publishing binary packages
of the boxes tomorrow. @jnfoster https://github.com/jnfoster, any luck
with the Eclipse pull?

The most important part has also been done: the ascii art!
avsm/vagrant-opam@0ba2974avsm/vagrant-opam@0ba2974


Reply to this email directly or view it on GitHubhttps://github.com/ocaml/opam/issues/1035#issuecomment-32419334
.


Reply to this email directly or view it on GitHub.

Member

avsm commented Jan 17, 2014

That was fixed last week to point to a mirror. I've done a fresh build, and I only ran into a disk space issue which I'm figuring out how to fix atm by adding a build disk.

On 17 Jan 2014, at 05:40, Nate Foster notifications@github.com wrote:

My TAs are still blocked on that download.camlcity.org bug. I verified
(from a different location so I don't think it's a network issue). Are
others still able to build the VM?

-N

On Wed, Jan 15, 2014 at 4:53 PM, Anil Madhavapeddy <notifications@github.com

wrote:

I've pushed a number of cleanups and will start publishing binary packages
of the boxes tomorrow. @jnfoster https://github.com/jnfoster, any luck
with the Eclipse pull?

The most important part has also been done: the ascii art!
avsm/vagrant-opam@0ba2974avsm/vagrant-opam@0ba2974


Reply to this email directly or view it on GitHubhttps://github.com/ocaml/opam/issues/1035#issuecomment-32419334
.


Reply to this email directly or view it on GitHub.

@avsm

This comment has been minimized.

Show comment
Hide comment
@avsm

avsm Feb 17, 2014

Member

I've also updated the checklist to note that we have an awesome IPython/OCaml interactive console now, so we could include that in the default binary image when it's merged into OPAM trunk.

Member

avsm commented Feb 17, 2014

I've also updated the checklist to note that we have an awesome IPython/OCaml interactive console now, so we could include that in the default binary image when it's merged into OPAM trunk.

@AltGr AltGr modified the milestones: outgoing, 1.2 Jun 27, 2014

@AltGr AltGr modified the milestones: 1.3, Next Nov 19, 2014

@AltGr

This comment has been minimized.

Show comment
Hide comment
@AltGr

AltGr Aug 30, 2016

Member

Closing this issue as it is vastly out of date, and although the support for this feature is not perfect yet, most of it is there, and the remaining bits already have active issues for them.

Member

AltGr commented Aug 30, 2016

Closing this issue as it is vastly out of date, and although the support for this feature is not perfect yet, most of it is there, and the remaining bits already have active issues for them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment