nimble needs to support system-wide installations

[philip-wernersbach]

(See http://forum.nim-lang.org/t/656 and http://github.com/Homebrew/homebrew/pull/34693)

I'd like to get nimble into some package managers alongside nim. In order to do this, nimble needs to support being installed to arbitrary system-wide locations, and needs to coexist with the nimble installation in a user's home directory. My solution to this is nimble-wrapper, but there's probably a better way to do this.

[philip-wernersbach]

Copying @mindplay-dk and @Varriount because they were discussing packaging nimble with nim on the forums.

[Varriount]

Ok, so first, let me see if I understand your requirements:

• Nimble needs to support installation to arbitrary locations
• Nimble needs to support multiple installations in a single system

So, what currently happens when one attempts these things? I develop on Windows, so the problem is slightly different for me.

[philip-wernersbach]

Currently the only way to install to arbitrary locations is to modify the nimbleDir variable in $AppDir/nimble/nimble.ini. This is a no-go because packages can't install things to the user's home directory, so packages can't modify $AppDir/nimble/nimble.ini to set the correct location. Even if we could modify $AppDir/nimble/nimble.ini, doing so would take the installation in a user's home directory out of the equation.

For comparison, ruby's solution to this is that its library loading code supports multiple paths and specifies both system-wide paths and a path in the user's home directory by default. Ruby then iterates through the paths and uses the first one that contains the library to be loaded.

[mindplay-dk]

I'm not an expert on this subject, but FWIW... I hated the way Ruby's package manager worked, where dependencies were installed and shared between projects. In contrast, I love how Composer installs dependencies to your project's own vendor folder, avoiding the clutter and potential conflicts associated with sharing dependencies between projects.

Because Nim links everything statically (and produces stand-alone executables by default) sharing dependent packages between projects probably doesn't even make any sense to begin with, so maybe this concern is irrelevant.

I also love how Node comes with NPM already installed out of the box - this is the thing I was requesting, primarily. If we have a working de-facto standard package manager for Nim, it should be available right away, encouraging people to create and consume packages and building an ecosystem :-)

[philip-wernersbach]

Now that I think about it, I think @mindplay-dk's suggestion to package nimble with nim makes a lot of sense. I originally envisioned packaging nimble separately from nim, but nimble is such an essential part of the nim tooling that I don't really have "separation of concerns" issues with packaging them together. I also think that not having to install nimble separately would be a benefit to end users. This would also make my life easier when I start making Debian and Fedora packages for nim and nimble, as I would have to maintain half as many packages.

Again for comparison, Ruby took this route as well. RubyGems is a separate project from Ruby that had to be installed separately, but starting with Ruby 1.9, RubyGems is packaged with Ruby.

[philip-wernersbach]

@mindplay-dk: Could you open a PR for the vendor folder functionality? The vendor folder solution is probably better than having to deal with side-by-side system- and user-specific library install locations.

Also, jgoodgive from the forums asked about this, so I'm pointing him here.

[mindplay-dk]

Thanks for the vote of confidence, but I'm probably not the right person for the job - I've just barely started learning Nim, and I haven't even used Nimble yet :-)

[Varriount]

@mindplay-dk Eh, what? Nim dynamically links by default. It loads things from dynamic libraries by default.

[dom96]

@philip-wernersbach I can easily extend Nimble to support multiple nimbleDir's. I can also add a --global flag which will install packages globally to /usr/lib/nimble/ or something like that. Maybe you could tell me the best path for global installs of Nimble libraries?

[philip-wernersbach]

@dom96: Here are my suggestions:

1. Still have packages install to a user's home directory by default, but add the --global flag for installing packages globally.
2. Make /usr/lib/nim/nimble the global package path.
3. When a package is installed globally and installs a binary, do the following:
   • Install the binary to /usr/lib/nim/nimble/bin like normal.
   • Add a symlink for the binary into /usr/lib/nim/bin.
   • Add a symlink for the binary into the same folder as the nim executable. For instance, if the nim executable is installed to /some/path/nim/bin and c2nim is installed, a symlink to c2nim would be installed to /some/path/nim/bin/c2nim.
4. Have the nimble installation instructions from Github install the nimble binary to the same folder as the nim executable.

After that we have the packages and the Windows installer put the nimble binary into the same folder as the nim executable, and so the user can override/upgrade the bundled nimble by installing from Github.

[jpoirier]

A bit late to the discussion...

+1 for including Nimble with the binary distributions.

My suggestions for *nix systems:

• put default local installs in ~/nimble
• put default global installs (--global) in either /usr/local/nimble or /opt/nimble
• have nimble in the same folder as the nim executable; when nimble is packaged with the compiler is there really any point, or purpose for, putting it in nimble/bin?
• only put officially packaged executables in nim/bin
• third party executables can go in nimble/bin but let the user explicitly add it to their path

[cdunn2001]

Go-lang (go-build/go-install/go-get) solves this completely. It has $GOROOT for the system installation, and $GOPATH for a colon-separated list of workspaces. Each workspace can have multiple go-repos. Ordinarily, you don't need to set $GOROOT at all.

The Go folks have nailed this. Go is very similar to Nim, so I suggest following their lead. The main difference is that Go code can import from URLs (e.g. GitHub), whereas nimble relies on nimbledata.json. I think nimble is better for that.

[dom96]

The main difference is that Go code can import from URLs (e.g. GitHub), whereas nimble relies on nimbledata.json

That's incorrect. Firstly, Nimble relies on packages.json optionally. Secondly, Nimble does support installation from URLs. You can easily execute nimble install git://github.com/dom96/jester for example.

It has $GOROOT for the system installation, and $GOPATH for a colon-separated list of workspaces.

So $GOPATH is a list of paths which the Go compiler uses to find Go "packages". This sounds like the equivalent of --nimblePath in the compiler. Difference is that it takes only one path and that it's not an environment variable. It's also set automatically for you to ~/.nimble/pkgs. I'm not sure but I think you can specify it multiple times.

I have a small amount of experience with Go, but the experience that I do have with it was enough to get me annoyed with the need to set $GOPATH.

[cdunn2001]

Nimble does support installation from URLs. You can easily execute nimble install git://github.com/dom96/jester for example.

You don't understand. Go allows import "github.com/user/project.git" in source code. That's a major difference between Go and Nim. D offers something similar, iirc. I always thought that was silly, but I guess some people like it. I don't want to get into the URL aspect because I don't think it's important.

Other than that difference, the way Go is compiled and the way Nim is compiled are fundamentally similar. If you architect nimble without knowing a great deal about how and why go-install works as it does, then you are missing a huge amount of wisdom developed within Google.

So $GOPATH ... sounds like the equivalent of --nimblePath in the compiler.

GOPATH is a colon-separated list of workspaces, and each workspace can have multiple git-repos. An element of GOPATH is like a --nimbleDir in Nimble. I haven't tried --nimblePath yet. I don't understand how the compiler can do more with Nimble than I can do from the command-line.

enough to get me annoyed with the need to set $GOPATH

You want the defaults to do the best thing, right? I understand that laudable goal. But many people juggle multiple projects simultaneously. Please do not require everyone to work the way you do. Many of us work with many people on many projects in parallel, and we are often waiting for changes here and there to get through various channels. If I could work on just one thing at a time, I would. That is a nice position to be in. But normally, I am in negotiations, or waiting for someone to return from vacation, or making changes for an ongoing code-review, etc. The versions in ~/.nimble need to be the stable versions. Other sets of packages need to be maintained for days or weeks. I need to be able to switch to an old directory of git-repos and have things just work. --nimbleDir is better than nothing, but I won't admit that it's a good solution. If I forget to type it, then I blow away my ~/.nimble packages? That's pretty terrible. I don't understand why you think that danger is better than the evils you see in environment variables are parent-directory traversal.

Beyond multiple workspaces, people in this thread are pointing out the need to operate with both local and system (shared) packages in the same build. I'm saying that those are fundamentally different because they have different directory structures. The system installation (at least for other languages) lacks git-repos and intermediate files. A system installation and a workspace will never be the same thing. And I'm further saying that the Go folks have found a solution which works well for a variety of use-cases. I remember the days of much more complexity for Go. Today, GOROOT and GOPATH solve the problem reasonably well.

But that's go. What about git? When you run git, it checks parent directories until it finds .git. I'm sure you realize that, but you don't seem to acknowledge it. git also allows environment variables to over-ride configuration files, but it does not require them. Everything can be configured via configuration files, and not only from ~/.gitconfig. It's very flexible.

I think I understand where you're coming from. Honestly, I admire your desire for simplicity. I wish my world could be that simple.

[dom96]

You are correct about my desire for simplicity. I am glad that you acknowledge it :)

What happens if I have the same package (of two different versions) in my GOPATH? How does Go resolve which one it is that I want?

By the way, I think that we've begun discussing something which is offtopic for this issue.

But anyway, another question that I have is, how does Rust's Cargo solve these issues?

[cdunn2001]

A system-wide installation is related to an ad hoc project if there is a list of directories in the project. I don't need that personally. I'm happy with just one workspace for a set of packages. But some people seem to want multiple workspaces in the same build.

When searching, gobuild takes the first package found in the search path. GOROOT would come last. However, if an installed package depends on something which exists in a local workspace version, then the installed package needs to be rebuilt. In that case, lacking permissions in the system directory, you need to "goget" that package.

When installing, goinstall would install a given package into the workspace in which it resides. A workspace has an src/ directory of repos, as well as pkg/ and bin/.

At any rate, this is not my primary interest. I am looking for multiple, unrelated workspaces that I can switch between. This thread shows some support for that idea, but yes, #131 is a better place to discuss it.

[wicast]

Never install something system-wide beyond my system package manager.That's one thing what I learned from Archlinux.
We can totally live without using any package managers,make&sudo make install can do. Why we use it is because they are handy,reliable and manageable.
I never prefer to use pip, npm or any other certain language package manager to install some into a system folder. Instead, pacman in Archlinux can handle everything,it wrapping those language package manager,the system-wide installation belongs to pacman.
This one is mainly for system-wide executable.

For nimble(and other similar tools),I use it for development dependencies management,under personal folders.
This one is mainly for source code libraries.Only in the user folder

One should't use these two type managers in a same place, they will conflicts.

So I don't wish nimble to touch my system folder,but add a system-wide folder by agreement that allows extra nim source codes to import is quite necessary for downstream packaging.

The easiest way is just add another nimble path, not only $home.

--global option？I really don't want it,if this supported,I won't use it either.
Buy the way, I hate modifing my $PATH configuration.

[dom96]

--global option？I really don't want it

Then you won't have to use it. Simple.

[wicast]

But this didn't solve my related problem…I still need my user to use dirty workaround.

[dom96]

It will solve your problem. The path to where Nimble will install packages globally will be added to Nim's config file by default (once I implement this feature).

[wicast]

Thanks a lot.
And I asked some of my friends using Rust,they say cargo didn't support global install.The executable only generated in ~/.cargo/bin like nimble now.

Go is the language I mainly used recently.The package management of golang has a huge issue,there is no version lock officially supported by Google, even their official package don't have any release tag numbers.I can't believe that how these packages are working fine for so long(actually they really didn't break anything but no release tag means no stable guarantee).
If you want version lock, you need some thrid part tools like glide. But it's not unique, not everyone is using version lock in go or using the same version control tool.Offical go get?No way(you can switch to the git repo and manually checkout to a certain version,but who will do that?).
Luckily,go is fully static link,so repackaging a go package with executable only is easy and controllable but still terrible.

[dom96]

@wicast I'm not sure what you mean by "no version lock". Also, how does this relate to this issue?

[wicast]

@dom96 I mean they don't write which version of your dependencies are in golang, and go get will using master branch as default.

I was considered about the situation when multiple version of a package was needed by two different package.I need to ensure is the naming styles still following name-version pattern in global nimble path.This will be important when downstream repackaging

When I trying to package nake, I did't follow this pattern but working.

Another question is if a newer version is installed in user folder and another older version in global.Which one will nim choose(suppose this two all fit the version value requirement).

I have no idea which way is the proper one.

And Golang is a really negative example I'd like to tell you,they don't solving version of dependencies , I hope it won't happen to nim.

[FedericoCeratto]

/usr has been managed by package managers across many distributions for a long time. Oftentimes language packages defaulting to /usr led to overwritten files and pain for the users. Please consider using /opt" as default.

[mindplay-dk]

I like the approach taken by Composer (and I think npm, not sure) where
globally installed packages are not available to your projects as
dependencies at all.

Instead, your globally installed packages are just dependencies of your
"global project", a project managed by the package manager, for the purpose
of installing command line tools etc. - not for the purpose of sharing a
copy of a library between multiple projects, which, yes, leads straight to
dependency hell.

Of course, both Composer and npm also maintain a local cache of all
versions of all used packages - so often they only have to copy a local
repository when you repeatedly install the same version of the same
dependency to different projects.

I agree that Go's approach is simplistic, though on the other hand, that's
their objective. If you make a breaking change, it's a new package. Simple,
though a bit naive.
On Dec 28, 2015 22:56, "wicast" notifications@github.com wrote:

@dom96 https://github.com/dom96 I mean they don't write which version
of your dependencies are in golang, and go get will using master branch as
default.

I was considered about the situation when multiple version of a package
was needed by two different package.I need to ensure is the naming styles
still following name-version pattern in global nimble path.This will be
important when downstream repackaging

When I trying to package nake, I did't follow this pattern but working.

And if a newer version is installed in user folder, and there is another
older version in global.Which one will nim choose(suppose this two all fit
the version value requirement).

I have no idea which way is the proper one.

And Golang is a really negative example I'd like to tell you,they don't
solving version of dependencies , I hope it won't happen to nim.

—
Reply to this email directly or view it on GitHub
#80 (comment).

[pb-cdunn]

This is a huge problem.

% ls -ld /tmp/nimblecache/
drwxr-xr-x 3 zkronenberg Domain Users 23 May 15 10:41 /tmp/nimblecache/

That's not me! So I cannot use nimble at all.

I cannot avoid this by setting $TMPDIR because nimble uses os.getTempDir(), which uses a semi-hard-coded path. (If you could accept -d:tempdir, that might be enough.) A simple fix is to use $NIMBLE_DIR/cache/ instead.

I understand that there are other problems for multi-user systems, but I can circumvent those by setting $NIMBLE_DIR (which is great, btw, thanks). This one is insurmountable.

[dom96]

Yeah, this is a massive problem. Another simple fix is to save in /tmp/$USER/nimblecache, can we get a quick PR for this as a workaround?

[cdunn2001]

I'll write up a PR, unless you're already working on it ...

[cdunn2001]

The question is what to do on Windows. We could check for $USER in the environment, and use it iff set.

The other question is how to find /tmp. It's more common to check $TMPDIR on Linx, and getTempDir() is discouraged in Nim.

I suggest we use $TMPDIR/$USER/nimblecache if both are set, and getTempDir()/nimblecache otherwise. That way current behavior can be maintained on systems which lack USER or TMPDIR, but people have ways around this when needed.

[brentp]

is this in latest nimble? I am still getting an error unless I explicitly set my $TMPDIR as another user on the system already has files in /tmp/nimblecache/

export TMPDIR=/tmp/$USER/
u6000771@winchester:~$ nimble install -y nimble
Downloading https://github.com/nim-lang/nimble using git
  Verifying dependencies for nimble@0.10.2
 Installing nimble@0.10.2
   Building nimble/nimble using c backend
    Prompt: nimble@0.10.2 already exists. Overwrite? -> [forced yes]
   Success: nimble installed successfully.
u6000771@winchester:~$ unset TMPDIR
u6000771@winchester:~$ nimble install -y nimble
Downloading https://github.com/nim-lang/nimble using git
io.nim(660)              writeFile
Error: unhandled exception: cannot open: /tmp/nimblecache/nimscriptapi.nim [IOError]
u6000771@winchester:~$ ~/.nimble/bin/nimble  install -y nimble
Downloading https://github.com/nim-lang/nimble using git
io.nim(660)              writeFile
Error: unhandled exception: cannot open: /tmp/nimblecache/nimscriptapi.nim [IOError]

[dom96]

Yes, it should be. Are you sure you're running the latest Nimble? (Try nimble install nimble@#head).

[brentp]

no. still not working for me.

(base) u6000771@winchester:~/Projects/src/slivar$ nimble install nimble@#head
Downloading https://github.com/nim-lang/nimble using git
  Verifying dependencies for nimble@#head
 Installing nimble@#head
   Building nimble/nimble using c backend
unset TMPDIR   Warning: Symlink already exists in /uufs/chpc.utah.edu/common/HIPAA/u6000771/.nimble/bin/nimble. Replacing.
   Success: nimble installed successfully.
(base) u6000771@winchester:~/Projects/src/slivar$ unset TMPDIR
(base) u6000771@winchester:~/Projects/src/slivar$ nimble install nimble@#head
Downloading https://github.com/nim-lang/nimble using git
io.nim(660)              writeFile
Error: unhandled exception: cannot open: /tmp/nimblecache/nimscriptapi.nim [IOError]
(base) u6000771@winchester:~/Projects/src/slivar$ which nimble
~/Projects/src/nim/bin/nimble
(base) u6000771@winchester:~/Projects/src/slivar$ ~/.nimble/bin/nimble install nimble@#head
Downloading https://github.com/nim-lang/nimble using git
io.nim(660)              writeFile
Error: unhandled exception: cannot open: /tmp/nimblecache/nimscriptapi.nim [IOError]
(base) u6000771@winchester:~/Projects/src/slivar$ 

[brentp]

$ ~/.nimble/bin/nimble --version           
nimble v0.10.2 compiled at 2019-09-05 20:55:52
git hash: 38cd55e71af3df66c26d02d127c454d5b0a31e58

[dom96]

(base) u6000771@winchester:~/Projects/src/slivar$ which nimble
~/Projects/src/nim/bin/nimble

This is wrong, Nimble is installed into ~/.nimble/bin, just like every other Nimble binary package.

[brentp]

yes, but in that same snippet, I'm also checking the nimble in ~/.nimble/bin

[dom96]

oh, apologies, I should have read further.

Well, @cdunn2001's code is definitely in master so I'm not sure what the problem is (https://github.com/nim-lang/nimble/blob/master/src/nimblepkg/tools.nim#L169), maybe it's not being used for some reason? Is TMPDIR and USER in your env?

[brentp]

yeah. i'm not sure what's going on, but on that cluster env, I have now set my TMPDIR to /tmp/$USER in my .bashrc to get around this problem.

[genotrance]

I'm tracking the proposal for the project local dependencies feature here. It is related to this global nimble dir feature so appreciate all feedback to help finalize the design.

[cdunn2001]

@genotrance, I'm confused by "localdeps". Not sure what that's for. But I've been using $NIMBLEDIR successfully to avoid global installation. Araq's suggestions for eventually interacting with nimbledir make sense to me.

I hope the TMPDIR nimblecache problems are resolved. I don't know what happened in the cluster env for brentp.

[genotrance]

Sure $NIMBLEDIR can be used as well to achieve local deps today, I'll document it. This RFC is to mainly define the behavior and motivation. The code changes are quite minimal but it will be new for users.

[FedericoCeratto]

After almost 5 years from opening this issue, Nim still has among the lowest penetration in OS distributions compared to similar languages.
Apparently no Nim libraries has been packaged anywhere yet. This is limiting the adoption of the language in different environments.
Can we make sure that either Nim or Nimble can find and use packages installed by the OS?

Related to nim-lang/RFCs#173
(See also nim-lang/RFCs#173 (comment))

[dom96]

Someone (maybe I) added this a while back: https://github.com/nim-lang/Nim/blob/devel/config/nim.cfg#L49

This should be expanded to include whatever other paths are needed and Nimble needs to be expanded to support these as well. That's it.

The only major step is implementing a --global flag in Nimble to install packages into whatever is the appropriate place on each distro (ideally it should be a single place... oh how I love inconsistencies between Linux distros...) and to keep track of the packages installed there. Should just be a case of changing/adding some paths.

[FedericoCeratto]

@dom96 It's up to the OS package mantainers to hardcode the correct path in the Nim/Nimble packages and build library packages accordingly. This is how it is done for Python, Rust, Go, C, Perl, Ruby and so on.
The only crucial requirement from Nimble (or equivalent) is to play nice with OS-installed libraries:

• find the packages installed globally based on the hardcoded path
• don't expect to write in it - just use what is found
• don't try to find packages elsewhere or do networking unless specified otherwise

These are very simple requirements but need to work reliably over time.

[dom96]

don't expect to write in it - just use what is found

What if I want every user on my system to have access to a Nimble package that can only be installed via Nimble? That is a use case that I have had in the past.

[FedericoCeratto]

@dom96 that's a different use case: a path under /opt for Nimble-installed packages and one under /usr for OS-installed packages - with different behaviors.

[CyberTailor]

After almost 5 years from opening this issue, Nim still has among the lowest penetration in OS distributions compared to similar languages. Apparently no Nim libraries has been packaged anywhere yet. This is limiting the adoption of the language in different environments. Can we make sure that either Nim or Nimble can find and use packages installed by the OS?

Things have changes. Nim libraries are now packaged in Nix and Gentoo (using custom build systems though, not nimble).

[FedericoCeratto]

@CyberTailor would you mind sharing how the OS-provided packages are picked up by Nim and Nimble? Nimble does not support OS-installed packages, and also cannot mix and match packages from multiple --nimbleDir locations. Perhaps you are using exclusively nim with -p?

[CyberTailor]

Yes. There are two custom build systems (Nix and mine) that find dependencies and call nim with corresponding -p options to build binaries.

[Yepoleb]

I just noticed the --global flag has been taken over by nimble develop to mean a user level development dependency as opposed to a package level one. This makes it either unavailable to indicate system level packages or confusing if it only has this special meaning for develop. What's the best move here?