WIP: Add nix to travis matrix

[srid]

Resolves #336

Travis is expected to timeout during the Nix build. The proposed fix is to use cachix. Perhaps a kowainik cache can be created by someone (who?) who will retain its private keys (and be authorized to push to it)?

Checklist:

• Update CHANGELOG
  • summoner-cli/CHANGELOG.
  • summoner-tui/CHANGELOG.

• Keep code style used in the changed files (see style-guide for more details).
• Use the stylish-haskell file.
• Update documentation (README, haddock) if required.
• Create a new test project using summoner and check that the changes work as expected.

Hint: Add the [ci skip] text to the docs-only related commit's name, so no need to wait for CI to pass.

[chshersh]

@srid Thanks for your work on this issue! Looking forward to seeing the end result 👍
I tried to read about cachix but I don't quite understand how it works and how it's different... Isn't Nix already has binary caches, why do we need another service?

Creating Kowainik cache is not a problem at all 👌 @vrom911 and I can create and maintain Kowainik account for this cache.

[srid]

Nix does use the official cache; but since our default.nix overrides certain source dependencies - they lead to re-building a bunch of packages manually.

Travis times out building them all.

So, the proposed idea is to use a secondary cache (i.e., in addition to the official cache), and run nix-build | cachix push kowainik (whoever owns the cachix cache should do this, as only they can 'push' to the cache), and then use cachix use kowainik (anybody can use the cache, as long as it is public) in .travis.yml, just prior to running nix commands, so that the Travis build uses the cache.

It is not perfect solution though, because every time we change default.nix source deps we might have to re-build and re-push to the cache before the PR travis builds use them successfully.

[srid]

I can write up a small guide (as a section in README.md) on how to use cachix to push/use cache for this project. Let me know if you'd like that. I would use my own cache bucket. Before merging this PR, you can follow the guide but use your's (instead of mine) - so that you have control over the cache.

[chshersh]

@srid Is it enough to create cache locally on my machine and then use it on Travis CI? Because operating systems on my machine and on Travis can be different... I don't know how sensitive cachix to this kind of stuff.

If nix-build | cachix push kowainik is the only command I need to call locally from the summoner repo to populate the cache, I can do this (assuming I have nix and cachix installed). Could you update .travis.yml with the proper cachix commands using kowainik cache? So once I push the cache I can restart the CI job.

[chshersh]

Also, do we still need to pin this version?

summoner/default.nix

Line 5 in be71488

pkgs ? import (builtins.fetchTarball "https://github.com/nixos/nixpkgs/archive/ed07403382c.tar.gz") {}

Can we use just <nixpkgs>?

[srid]

Yes, it is recommended to pin nixpkgs for evaluation reproducibility.

Only the default compiler (pkgs.haskellPackages) is cached; so that's another reason to pin. If we use <nixpkgs> that might have a different default compiler, which would have us override the compiler thereby needing us to build the packages manually.

[srid]

@srid Is it enough to create cache locally on my machine and then use it on Travis CI?

Good question; it probably needs to be same arch (Linux64?). I already run NixOS, but I imagine if I build the binaries on Mac, it probably won't work on travis.

[chshersh]

@srid When trying to push kowainik cache I see the following error:

unpacking 'https://github.com/nixos/nixpkgs/archive/ed07403382c.tar.gz'...
error: Package ‘cudatoolkit-9.2.148.1’ in /nix/store/rcj6djg2vzwgikglmf45rafs5cgj6385-source/pkgs/development/compilers/cudatoolkit/default.nix:189 has an unfree license (‘unfree’), refusing to evaluate.

a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
  { allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.

(use '--show-trace' to show detailed location information)

NoInput "You need to specify store paths either as stdin or as a cli argument"

Could you, please, help to understand, what does it mean and how to resolve it?

[chshersh]

Probably the before-script part should be written under this matrix item near script. Also, I would prefer uniform style, like this:

    install:
      - ...
    before_script:
      - ...
      - ...
    script:
      - ...

[chshersh]

@srid I've pushed cachix cache to kowainik. I see that CI passed but it took a long time and doesn't look like it reused cachix cache...

One thing I noticed while building with summoner with nix:

1. It also builds documentation. In our case it's redundant and extra work so would be nice to disable it 🙂
2. If my understanding is correct, the build locally takes so much time because some projects are not cached (because we override them). There are 3 our packages: relude, tomland and shellmet. It would be nice to fix them so we don't need to override them. But we don't know how to fix them so Nix works...

[chshersh]

Maybe it doesn't use my cachix cache because by default Travi CI is using nix-env and nix-build version 2.0.4 but on my machine it's 2.3

Suggested change

	- language: nix
	- language: nix
	nix: 2.3

[srid]

Hmm. Looking at the log, I see that it is indeed pulling some packages from your cache:
, but I'm not sure why it proceeds to compile the other packages without using the binary ones from your cache ....

[srid]

At first glance, I think it might be because of the extra Haskell dependencies used in nix-shell --run 'cabal new-test all --enable-tests that are not pushed in your nix-build -A summoner{,-tui} | cachix push kowainik push ...

Let me think.

[srid]

Note to self (when I come back to this):

This might be relevant: cachix/cachix#52

[domenkozar]

I'm happy to help, but have a few questions:

• why isn't travis configured to push? nix-build | cachix push summoner
• how comes it's building with cabal, while it should be using Nix expressions?

[chshersh]

@domenkozar I appreciate any help! But I can't answer the questions, I have only basic familiarity with nix. Probably @srid can clarify some details.

[srid]

why isn't travis configured to push? nix-build | cachix push kowainik

I think it should be! Wouldn't that require adding @chshersh 's private key on the Travis' machine? How is it that normally done?

how comes it's building with cabal, while it should be using Nix expressions?

It could be that the test-suite stanza is not used by nix-shell. But why would that be problematic here?

[domenkozar]

I think it should be! Wouldn't that require adding @chshersh 's private key on the Travis' machine? How is it that normally done?

You need to encrypt secrets via travis

It could be that the test-suite stanza is not used by nix-shell. But why would that be problematic here?

Ah so nix-shell provides haskell dependencies? That would make sense.

BTW, we released cachix-action for github, I'd recommend switching to that since it's going to be much easier. Although for your use case you might want to check cachix/cachix-action#8

[chshersh]

@srid @domenkozar I propose to close this PR and configure Nix build using GitHub actions. I've already done this successfully for our repositories tomland and relude. You can see this PR in relude for example of how I did this:

• https://github.com/kowainik/relude/pull/208/files

@domenkozar Thank you for your work on cachix, it's indeed very easy and smoothly to integrate it into projects and GitHub Actions! I've already added Kowainik CACHIX_SIGNING_KEY to the secrets of this repository. So GitHub actions can be added without any problems 👌

[srid]

Sounds good. I'm not at all familiar with Github Actions, but looking forward to how this project will use it ...