Version pining doesn't work

[nerzhul]

Hello

Describe the bug
i'm trying to pin some packages versions in order to be coherent with our CI/CD environment testing toolchain, and it seems devenv builds properly the packages, devenv info return the right version, but running command return upstream version.

To reproduce
Gist: https://gist.github.com/nerzhul/0efd6f1927a5756775004524efaca358

It builds properly the versions and devenv info returns the proper output i expect

# packages
- pre-commit-3.3.3
- terraform-1.5.0
- terraform-ls-0.30.1
- tfsec-1.28.1
- cachix-1.5
- statix-0.5.6
- vulnix-1.10.1
- deadnix-1.2.0
- nil-2023-05-09
- git-2.41.0
- direnv-2.32.3
- starship-1.15.0
- kubectl-1.27.3
- bash-completion-2.11
- terragrunt-0.46.3

but terraform -v and terragrunt -v returns the upstream versions

Terraform v1.5.1
on linux_amd64
terragrunt version v0.47.0

i'm on archlinux with nix, just FYI and here is the which return, proving it uses nix and devenv

❯ which terraform
/nix/store/j67y43xmd3bd9d4ylb9g69yal081wfpc-devenv-profile/bin/terraform

❯ which terragrunt
/nix/store/j67y43xmd3bd9d4ylb9g69yal081wfpc-devenv-profile/bin/terragrunt

Version

devenv: 0.6.2

[putchar]

i got the same results while trying to use this devenv.nix on a clean devenv repository

also while trying to install both terraform and terragrunt specific versions somewhere else (inside my flake.nix NixOS config for instance)

    home.packages = [

      (pkgs.terragrunt.overrideAttrs (attrs: {
        version = "0.46.3";
      }))
      (pkgs.terraform.overrideAttrs (oldAttrs: rec {
        version = "1.5.0";
      }))

I got the same results

[domenkozar]

To properly override the version you need to also override src and name:

(pkgs.terraform.overrideAttrs (attrs: rec {
		version = "1.5.0";
                name = "terraform-${version}";
                src = pkgs.fetchurl { url = "..."; sha256 = "..."; };
}));

[nerzhul]

thanks for the try, unfortunately it's more complicated than this for a simple override as it depends on buildGoModules. I'll try to do it better later :)

[nerzhul]

unfortunately it doesn't work like this for terraform, which depends on gomodules build, instead if used fixed nixpkgs version as it's golang it's not a big deal

[thenonameguy]

The "proper" way of depending on a given package version is to find the matching the nixpkgs commit ref that has it working.

You can add that nixpkgs version to your devenv.yaml as a new input terraform-nixpkgs. And use that in your devenv.nix expression, instead of pkgs.

Here is a very handy community website to find the correct Git commit:
https://lazamar.co.uk/nix-versions/?channel=nixpkgs-unstable&package=terraform

[paroxp]

This isn't necessarily a related comment, but I'll try it anyway.

It would be good to come to some sort of a consensus around how to handle versioning for different languages and services.

For instance, I am really digging all the great work @bobvanderlinden has been doing around languages.ruby.version, languages.ruby.versionFile or to even expand on that further, and intake a single file covering multiple languages in one go.

This effort, though excites me about the entire devenv ecosystem even more, is moot when we look at how this is handled in the languages.rust.version or the lack of setting in languages.javascript.

I do appreciate how difficult it could be for different software. I can imagine typescript for instance, not only having to deal with the version of tsc but also node or npm.

Reading the comments above, I am also worried about the user experience. Nix is powerful for sure. But let's not kid ourselves - it's not the simplest thing to grasp. Us trying to make it simpler for the end user, should not hurt, right?

I am looking at the two "proper" ways of managing versions of say, terraform. And I cannot help but question, why I as a user, need to dig through nixpkgs history to find the specific version of software I may need for one project, and another version for a different project. Why as a user, I need to worry about the hash? Why, should I force everyone in my team, to do so when they are bumping the version of a language, instead of relying on .terraform-version?

And yes, I do get the core idea of nix, but let's put the nerd aside... All I'm trying to do, is get on with my work. This is where Bob comes in to save us I think. Let the hash live in the lock file.

Little rant or vent this turned out to be, and a long way around saying:

Would it be possible if we enabled discussions on this repository, and started an RFC? I'm sure the community will enjoy another conversation in the realm of tabs versus spaces.

[bobvanderlinden]

The problem with solutions like languages.ruby.version is that it requires manual effort and maintenance. The underlying repository is https://github.com/bobvanderlinden/nixpkgs-ruby. To set it up has taken quite a bit of effort and long build times, the same effort needs to be put into any other package/language as well. We already have a few of these repositories:

• ruby: https://github.com/bobvanderlinden/nixpkgs-ruby
• php: https://github.com/fossar/nix-phps
• python: https://github.com/cachix/nixpkgs-python

But we do not have one for every package. If we want different versions of terraform, then someone needs to put in the effort to package them, just like the above repositories are doing. There just isn't an automated way to do the same atm.

Why, should I force everyone in my team, to do so when they are bumping the version of a language, instead of relying on .terraform-version?

It isn't really needed for everyone of the team to do this if you already have done it 😅 The key here is that someone needs to package them. Currently it's no-one is doing that in a public way, so anyone in the team needs to do this privately. It sounds like there is an opportunity to set up something like a nixpkgs-terraform repository if the whole team is depending on it. 👍

https://lazamar.co.uk/nix-versions/ is a really good 'next best thing'. It just doesn't expose its index in a way that is accessible from Nix. That means it's hard to improve the UX in devenv.sh for this in a clean maintainable way. In addition, the downside of using nixpkgs commits as the base for specific versions of packages is that it will fetch the whole dependency tree for that version of X: it'll be huge (in size) when using multiple versions. It'll also eventually run into incompatibilities when mixing package managers and library versions (pip and npm do not incorporate support for using specifically versioned native dependencies).

I think this is however the method that is used in devbox and flox. Both support multiple versions of packages using their own 'mirror'/'adapter' repository for nixpkgs. Same downsides as using lazamar, but they have indexed nixpkgs in a way that their tool can refer to package@X.X.X instead of nixpkgs commits. If the indexed nixpkgs repositories would be public, then devenv could make use of it as well. (allowing terraform@X.X.X as a package). The downsides do still apply for this method as well.

[paroxp]

Yeah, absolutely :) I'm sure there is a lot of work involved - though, naively I believe my assumption was that the work is initial.

I'd be keen to understand why it's not all that automatable 🤔

https://lazamar.co.uk/nix-versions/ is a really good 'next best thing'. It just doesn't expose its index in a way that is accessible from Nix.
...
If the indexed nixpkgs repositories would be public, then devenv could make use of it as well.

This prompts me with some ideas 🤔 It feels doable - though I don't want to commit myself to any promises 😅 Shame lazmar isn't easily identifiable with some sourcecode. I'm sure there's something I am overlooking that I wouldn't appreciate with my current way of thinking. I shall digest that further.

Thanks for the clarification anyway!