Project-local package dirs with precedence

[ozra]

Having worked a couple of years with iojs for the money, I really enjoyed the modularity of projects vs the C++ world.
I've already had several problems with Nim, while hacking the compiler etc. that ~./.nimble/.. pkgs and global nim stdlibs are preferred over the ones from the dir I'm working in.

This is mostly a nim mod-lookup issue, and partly nimble.

Basically a project-local mod deps dir - like "node_modules/" in iojs/nodejs - it should have precedance. Also, those package dirs, should easily be pure github checkouts, for deving on deps, hand in hand with a project.

This way version matched packages can be stored per project - they could still be cached in home-dir/nimble. I'd prefer 'proj-local' pkgs as default. But an option would be ok, "nimble install local..."

#114 (comment)
nim-lang/Nim#2819 (comment)
nim-lang/Nim#2819 (comment)

As for the module lookup, I guess that's in nim's domain.

[josephwecker]

Copying a discussion from another thread here where it belongs:

---

josephwecker:

BTW- my comment glossed over nimble's package versioning. Aside from that though it does allow for nimble in the future (if desired) to optionally install packages local to a project and just have everything work as usual. It'soff-topic but it has been extremely useful to me in just about every language I've used- js, ruby (rails especially), c, erlang, ...

---

dom96:

@josephwecker In what way is that useful?

---

josephwecker:

@dom96 I don't want to hijack this thread- so warning: somewhat offtopic from this issue.

And it is a minority case compared to the semi-global package cache as nimble correctly implemented, but generally it is useful as a more clear & abrupt separation of a projects personal space vs shared space; an extension of the reason nimble probably uses ~/.nimble/... instead of /opt/nimble or /usr/include/nimble.

There are various deployment / replication / compiler-farm / multi-project-editing-workflows / debug scenarios where it's very useful to have all or some of the dependent packages that the package manager acquired right there in the main project's subdirectories- neither affecting nor being affected by anything else on that machine. npm, rebar (for erlang-OTP), and rails are the ones I've actively used in the past that come to mind right away that have the ability to optionally install local to the project. In C I've done essentially the same thing in order to experiment with different stdlib "overrides" for super-high-availability zero-copy servers back at twitch.tv etc.

---

josephwecker:

@dom96 similarly- Ruby, perl, and to a lesser degree python have rvm/perlbrew/virtualenv for allowing any project to have its own version of the VM/compiler as well as its own "package space"/gemset/etc. separate from the shared one...

[dom96]

I think the current way to develop (dependency) packages is good enough (TM).

Clone the dependency repo. Work on it. Then run nimble install in its directory. Switch to your project and test it.

[ozra]

Ah, yes. Is there anyway of setting a "temporary" version / dev-version or something then - so it doesn't shadow the "legit" version of it? (The sandboxing we so dearly want)

So, if working on dep "foo" of version "0.4.7" - I don't wanna do the version bumping - it's up to upstream, but I don't want it to shadow foo@0.4.7, so I want my changes at foo@0.4.8 (patch-version +1), but without affecting the source...

What do you thing, some idea?

[dom96]

If you don't want it to overwrite the older non-dev version then you will need to bump the version.

[cdunn2001]

I think the current way to develop (dependency) packages is good enough (TM).

But could be better. Let's make it better.

@ozra and @josephwecker, the new --nimbledir flag is a step in the right direction, but I don't want to have to type that every time I run nimble in a particular workspace directory. (A "workspace" may contain multiple git directories.) My idea is for nimble to search for the nimble.ini first in the current directory (and maybe in the parent) which would then serve to define a workspace. If not found, then it could look in $AppDir/nimble/.

This is pretty easy to do. The first step is this change, which was rejected. It's worth reconsidering. For now, I'll experiment with my temporary fork, but I will continue to hope that it can eventually be merged back into nimble.

I'm very tired and maybe not thinking clearly enough. Has anybody else used Go? I think its build-system is impressively flexible. We can borrow ideas from there. An environment variable like GOPATH might be the way to go, rather than a configuration file.

[dom96]

So, in addition to --global (still to be implemented, install to /usr/lib/nimble, or somewhere more appropriate) and --local (default, install to ~/.nimble), you want to also have --project (install to $PWD/.nimble, or something like that)?

That's something I would be willing to implement, as it plays nicely with and will create a beautiful trifecta of --global, --local and --project.

I still feel like I don't understand the advantage this brings. Do you want to have multiple copies of the same git repository on your system, updating each separately as-needed, developing the copy that you want to test via some other package?

[cdunn2001]

Yes, multiple copies of git-repos. These are separate workspaces. I don't want to litter my "normal" Nim installation with any WIP.

I also think a local nimble.ini file makes sense, to over-ride the global one in ~. That's how many tools work, including git itself. Or an environment variable could define the workspace, which is basically how GOPATH works. However, command-line flags are almost equivalent to environment variables since I can use an alias within a given shell:

alias nimble=\nimble --nimbledir=/my/local/workspace/path

I can use different shells for different workspaces. That's good enough for me. So I guess --nimbledir works for me after all.

I don't know anything about --local/--global... I guess that comes from #80. Well, I think those flags are confusing when --nimbledir exists. Do they over-ride it? Do they only alter a default? If so, what happens when I specify both? It's confusing. In UNIX, most installers rely on --prefix, which might be /usr by default. For local installs, people always specify the local prefix. I guess you don't want to type --nimbledir=~/.nimble all the time. So how about this:

• If --nimbledir is provided, use that always.
• If not, then look for any parent directory called .nimble. If found, use that.
• Otherwise, use the global system default.

However, I prefer this simpler logic:

• If --nimbledir is specified, use it always.
• Otherwise, use ~/.nimble.

I don't see a value in --global. Explicit is better.

[cdunn2001]

Oh, I see. In #80, some people want nimble packages to come from multiple installations, particularly including the system root, /usr/lib/nim or whatever. I'll comment there.

[dom96]

I would prefer not to have Nimble search any parent directories. I think that searching for a directory called .nimble will only create confusion.

--global will act like an alias for --nimbledir:/usr/lib/nimble, if the user specifies a --nimbledir then the global will be overriden.

[ozra]

I prefer command line flags over PATH. But I'd really prefer the local .nimble conf. That way "nimble install" wouldn't accidentally overwrite system wide version if I'm tired and do the wrong thing (which incidentally does happen from time to time). This way deps stay stable system wide, while the WPC version is sandboxed and only "installed" project locally. It makes sense. It could require it to be in cwd (aka, calling nimble when in src/ dir won't work - no parent searching..) [ed: - would this make nimble install still work, and thus cause the problem I mentioned, or does that too require to be 'in the right dir', yes?]

[dom96]

I suppose I have the same question for you as I asked here: #80 (comment)

What happens if I have the same package (of two different versions) in my GOPATH? How does Go resolve which one it is that I want?

i.e. how can Nimble disambiguate the local package vs. the system package?

[cdunn2001]

@dom96, Let's establish how to deal with multiple, unrelated workspaces first. The issue of precedence is better discussed in #80, in relation to system installations. (With Go, only standard packages are in GOROOT. If you want to edit them, you kinda need to check-out the entire Go compiler. I think that's fine.)

@ozra, A local nimble-conf file is an interesting solution for telling nimble where to install. However, if it contains absolute paths, then it cannot be in the repository. Also, if it doesn't look at parent directories, why not simply install in the CWD? I think this really gets at the fundamental problem with today's nimble. It's way too complicated (in different ways than we need). If I run nimble install in ~/workspace/foo, it goes into ~/.nimble/.... That's counter-intuitive. ~/.nimble has source-code already. It should use only the repos in its own sub-dirs.

With GOPATH, a workspace contains:

GOPATH=/home/user/gocode

/home/user/gocode/
    src/
        foo/
            bar/               (go code in package bar)
                x.go
            quux/              (go code in package main)
                y.go
    bin/
        quux                   (installed command)
    pkg/
        linux_amd64/
            foo/
                bar.a          (installed package object)

The repositories are in src/. When it builds any given package, it builds into bin/ and pkg/. If something is needed from the web, goget will pull into src/ and then build into bin/ and pkg/OS/PKG. That's simple. It's very easy to understand what came from where.

If you want to remove a repo, then remove it manually. Nimble does not need to clone into /tmp/. Repos are way smaller than binaries.

• http://golang.org/pkg/go/build/
• http://golang.org/cmd/go/