Better package source fetching

[blaggacao]

Courtesy of research conducted by @GTrunSec

Would your feature fix an existing issue?

The current way of fetching package sources is by using nix flake builtin capabilities via a sub-flake located in ./pkgs/flake.nix.

This workflow is seen critically by the community as an overuse of flakes.

Describe the solution you'd like

https://github.com/berberman/nvfetcher
Implements a flake-native version updater which relies on purpose built tooling for upstream version detection on a variety of source hosting platforms.

It implements a simple to read toml file for specifying those source repositories (instead of sort of "twisting" flake inputs to fit).

It abstracts away the perceived complexity of ./pkgs/flake.nix entirely.

Describe alternatives you've considered

At some point, flakes might mature into a state where source fetching is natively supported and semantically useful version data (ref in addition to rev) will be accessible as an input's metadata. However the corresponding suggestion in upstream nix (link tbd) was closed by @edolstra for the time being, which indicates that this use case will not enjoy prime support any time soon.

Additional context

• We have to analyze if the underlying nvchecker has superior equal or inferior version fetching qualities than builtin nix.
• To complement the bigger picture within the ecosystem it might also be in order to compare the underlying nvchecker sumarily with @ryantm 's nixpkgs-update.

[Pacman99]

I've yet to use it, but another option is https://github.com/Mic92/nix-update.
It should work well with our current pkgs folder and its formatting.

[blaggacao]

I've yet to use it, but another option is https://github.com/Mic92/nix-update.

Mic92/nix-update#57

---

Edit: It looks like nvchecker vastly out-competes nix-update on available plugins

nvchecker plugins vs. nix-update plugins.

And by the way, probably via nvfetcher also out-competes nix built-in fetchers that could be updated via flake inputs. @berberman Can you enlighten us?

[nrdxp]

For me, the whole point was to avoid any external tooling. We could have just used niv if we were gonna do this 😆

I guess the real frustration is this: Why does the community consider source fetching an overuse of flakes? Nix is a software package manager, this is literally it's main job, why wouldn't we want to optimize and simplify that workflow?

I do agree that the current implementation is lacking and overly difficult, but I still think the best solution is to fix it upstream, rather than creating yet another Nix crutch.

[blaggacao]

/cc @tomberek since you where skeptical on using sub flakes in the recent office hours.

[tomberek]

I don’t think I have the context to understand the full question. Specifically, I dont see what the sub-flake is doing that couldn’t be done in the main devos flake.

Regarding the “overuse of flakes”: my question would be what kind of states would be representable in a sub flake design that would not be with just a main flake. Are they desirable, any illegal states? Maintenance efforts? Mismatched lock files?

Generally, flakes are good to ensure you enter the Nix world in a pure manner. Once inside, flakes CAN provide a common/default interface. In this case it is to introduce additional inputs?

A conversation perhaps?

[nrdxp]

my question would be what kind of states would be representable in a sub flake design that would not be with just a main flake

I don't think there are many (that I can think of). Biggest reason I wanted to branch off into a subflake was just for a clean separation. Since there is no programmatic way to split up flake inputs this was essentially my last resort. I wasn't 100% pleased with the end result, but I figured at least it would expose some pain points that could then (hopefully) be addressed upstream. I also wanted to showcase what could be accomplished if the community did decide to embrace them.

Here's a few issues off the top of my head:

• flake inputs are duplicated across the subflakes flake.nix and the toplevel one
• a nix flake update doesn't recursively update local subflakes, which is confusing to say the least
• subflakes do generally feel like an abuse of flakes as they are now, but this is not to say that they can't be improved to be a first class addition, after all the entire flakes effort started off as a way to provide a proper module layer to Nix, or at least its predecessor did.

I would also appreciate a more official way to specify public vs private information in flakes. What if a bunch of inputs are needed for development, but not to actually build the flake outputs? Why does the user need to pull them all in?

At the very least, if we don't want to support subflakes in Nix, we shoulld probably explicitly ban their usage, rather than allow the confusing situation that exists atm. I wouldn't even mind that much if they were removed, if some additional mechanism was introduced to split up inputs into multiple files without resorting to starting a new repo. Maybe even something as simple as an overloaded import statement that, when used in inputs, just pulled in a list of flake refs from some other place.

But, to me, flakes always felt like a very nice replacement for the manual specification of package sources, and I was sort of dismayed to see the unbiquitous usage of fetch* functions years after their introduction. What's a lock file for if not to make this process easier? Maybe I'm just confused about the original purpose of flakes to begin with, but I don't think I'm the only one 😄

[blaggacao]

What if a bunch of inputs are needed for development, but not to actually build the flake outputs? Why does the user need to pull them all in?

I recently figured out that inputs are actually lazily fetched. A very intricate way of getting downstream deps into one's flake.lock file but actually never fetching them (or only once, if flake.lock file doesn't exists). That could be improved, indeed.

[berberman]

I'm not sure what's the context here, but using flakes or niv to track and update such packages sounds kind of abuse to me. Let me share some ideas of nvfetcher. The design purpose of nvfetcher is to provide a non-intrusive way automating nix packages updates, with a key idea that "how to track the version update of a package" and "how to fetch a package" should be independent. For the former one, I believe that nvchecker is a successful tool with exhaustive version sources supported in this field, so there is no need to invent the version tracking here again - nvfetcher does nothing about this but call nvchecker as a sub-process. For the later one, once we have a version number, we need construct nix fetchers to do prefetch, obtaining the integrity hash. The decoupling of version source definition and nix fetcher definition makes it possible to deal with some complicated cases.

For example, fcitx5-pinyin-zhwiki is a dictionary released monthly. But the real contents in its git repo is a dictionary generator which we don't care about, whereas the dictionary data we intrested in is under a sinlge release tag. nvfetcher is able to handle it with the following configuration:

# nvchecker.toml
[fcitx5-pinyin-zhwiki]
src.aur = "fcitx5-pinyin-zhwiki"
fetch.url = "https://github.com/felixonmars/fcitx5-pinyin-zhwiki/releases/download/0.2.2/zhwiki-$ver.dict"

As you can see, src.aur points the version source to its aur package, which indicates that we will update this package when its aur package updates. fetch.url says that the dictionary should be downloaded from a url ($ver is the result of running nvchecker). Running nvfetcher with this, a nix file will be created:

# sources.nix
{ fetchgit, fetchurl }:
{
  fcitx5-pinyin-zhwiki = {
    pname = "fcitx5-pinyin-zhwiki";
    version = "20210520";
    src = fetchurl {
      sha256 = "17f6cmvs0xsi3sq6da98865szzkfpk2smdag06aikl8bn9s5v584";
      url = "https://github.com/felixonmars/fcitx5-pinyin-zhwiki/releases/download/0.2.2/zhwiki-20210520.dict";
    };
  };
}

which gives us all we need to create drv of this package. nvfetcher will

In summary, I think this kind of nix packages update is very different from the purpose of flake's or niv's update mechanism. As we know, there are already many tools designed to handle this, e.g. nix-update we discussed above. nvfetcher is kind of "yet another" tool for this, but it makes the entire process much clearer and flexible. Currently my flakes repo is relying on it, which I think is a successful story in my book. And I'm happy to extend nvfetcher if need be.

[blaggacao]

@berberman Is nvfetcher currently capable of handling all nvchecker's upstream sources or are there certain limitations so that X or Y upstream surce cannot be used with nvfetcher although nvchecker would have had support for them?

[berberman]

@berberman Is nvfetcher currently capable of handling all nvchecker's upstream sources or are there certain limitations so that X or Y upstream surce cannot be used with nvfetcher although nvchecker would have had support for them?

So far nvfetcher only supports few nvchecker upstream sources. Personally, I think some of them are less common so I just didn't add them. I guess it won't be hard to embrace all nvchecker 's capability, though it would be somewhat tedious.

[nrdxp]

You arguments are fairly convincing @berberman. I appreciate the detailed feedback. After thinking it over again, I'm happy to test and review a pr on nvfetcher, since the current state of affairs is definitely less than ideal.

[GTrunSec]

I'm ready to PR this feature.(Example https://github.com/GTrunSec/nixos-flk/blob/main/nvfetcher.toml). Waiting for #301

[blaggacao]

@GTrunSec #301 is merged, I think you're good to go. Curious to see what you'll come up with.

[tgunnoe]

More than one flake per repo doesn't make sense to me. the packages in /pkgs/ how they were before, just a list and matching nixpkgs layout and you could still do nix run github:tgunnoe/nixflk#lzwolf was just lovely. now its a burden and confusing updating to a new configuration, almost every time

[blaggacao]

The good thing about the nvfetcher implementation is that it is completely optional and unobstrusive.

Commenting / deleging one line in https://www.github.com/divnix/devos/tree/59f32cbff3319ca95f10f17073ce581cad5a1fd9/pkgs%2Fdefault.nix otherwise disables it. 😄

But nvfetcher is also a real utility estate.

Merging this is still a bit blocked by upstream digga, but should land soon (today?).

[blaggacao]

Done in #325