Feature/nix derivation #78

Enteee · 2019-11-05T09:05:48Z

Hi folks, NixOS support! 😄 🚀

The things that I added are:

default.nix: a reproducible and pure builds with nix-build
shell.nix: a managed development environment with nix-shell
README.md: documentation on how to enter the development environment
deps.nix: freezes all build dependencies managed in go.mod
.github/workflows/nix.yml GitHub workflow which:
- updates deps.nix if go.mod changes (automatically creates a pull request)
- fails the build if deps.nix is not in sync with go.mod
- runs nix-build which does build the binary & runs tests
derivation.nix: allows for easy integration into the nixpkgs repository with something like this:

{ callPackage, fetchFromGitHub, buildGoPackage, buildEnv, stdenv}:

let
  pname = "rmapi";
  repo = fetchFromGitHub {
    owner = "Enteee";
    repo = pname;
    rev = "65c869f7566ce66a517509219c8ba9f58483808e";
    sha256 = "sha256:0limyj7dim0vr3c357zbhvhrm3jg3jjiv95fb0bgif3ri9f03akj";
  };
  drv = callPackage "${repo}/derivation.nix" {
    buildGoPackage = super: buildGoPackage (super // {
      meta = super.meta // {
        maintainer = stdenv.lib.maintainers.Enteee;
      };
    });
  };
in
  buildEnv { name = drv.name; paths = [ drv ]; }

I did this mainly for myself, but thought maybe you would be interested in this as well. If you are interested in this PR and we merge this, i will then take it a step further an create a derivation in the nixpkgs repository so that all NixOS users can use this by just: nix-shell -p rmapi.

If not I will probably have to create a very similar pull request in the nixpkgs repository.
No hard feelings if you decide to reject this 😄 🦆

To Do

I will keep my fork open and try to get that into nixpkgs using the approach documented in the description. This will hopefully trigger a code review of this pull request as well including nix contributors.
~~If I do succeed in getting this to nixpkgs and you are ok with the changes suggested, we merge this PR.~~
~~I will then create a new PR for nixpkgs which basically only changes the source repository from my fork to this repository.~~
🍺

I am currently not a 100% sure how flexible and open nixpkgs contributors are on merging pull request which basically just references a derivation stored in an other repository. It should not really matter because nix is side effect free but you never know. For easy maintenance of deps.nix, I would prefer it if all of the nix stuff was stored here instead of nixpkgs...

So if this is of any interest to the project, I would actually recommend the following for this pull request:

I will keep my fork open and try to get that into nixpkgs using the approach documented in the description. This will hopefully trigger a code review of this pull request as well including nix contributors.
If I do succeed in getting this to nixpkgs and you are ok with the changes suggested, we merge this PR.
I will then create a new PR for nixpkgs which basically only changes the source repository from my fork to this repository.
🍺

lobre · 2019-11-26T06:49:20Z

Hello,

Thanks for the PR!

I have plans for finally try and dig NixOS. I am really beginning to learn it to be able to switch to it as my main OS. Maybe I can review as soon as I have enough knowledge...

In the meantime, if any other Nix user feels comfortable for reviewing this, I would not see any problems to merge to have it merged...

Loric

…ation

Enteee · 2019-12-04T12:00:45Z

Related question on discourse: Is importing a derivation from another repository bad practice?

lobre

Hey, sorry this will feel more like questions than a proper review because I am really starting to dig the Nix world (which by the way is awesome!). If you can help me understanding this by answering my questions, that would be super helpful to me.

lobre · 2020-01-17T17:05:23Z

derivation.nix

+{stdenv, buildGoModule}:
+
+let
+in buildGoModule rec {


Does this pattern mean that someone using this derivation.nix can easily inject custom attributes to the set?

No, this is a custom builder provided by nixpkgs for builiding go programs managed with go modules. I tried to follow the official nixpkgs documentation, which suggest using buildGomodule. But yes, you could create an overlay for this if you wanted to 'inject' things.

lobre · 2020-01-17T17:08:16Z

shell.nix

+{ pkgs ? import <nixpkgs> {} }:
+with pkgs;
+
+let


What is the purpose of an empty let / in statement? Is it just by convention for potential future values?

Nah that's merely just a leftover. I used to have stuff in there which disappeared over time. I will remove this.

lobre · 2020-01-17T17:28:54Z

shell.nix

+
+let
+in pkgs.callPackage ./derivation.nix {
+  buildGoModule = super: buildGoModule (super // {


As I am quite new in functional languages and nix in general, I might need a little extra help to understand these two lines 5 and 6! So of what I understand, buildGoModule is a function that should return a set that will represent the final derivation. So defining it like this woud allow the caller to provide a super set argument that would be merged with the body of the function here.

But what is the difference with having maybe something a little bit simpler like the following:

derivation.nix

{ stdend, overrides }: overrides // { name = "rMAPI"; src = ./.; goDeps = ./deps.nix ... }

shell.nix

{ pkgs ? import <nixpkgs> {} }: pkgs.callPackage ./derivation.nix { overrides = { name = "rMAPI-env"; buildInputs = ...; }; }

Thanks for shedding light!

I do think, technically this would work. But I can see some possible downsides of this approach:

I tried to to make the inputs reflect dependencies only

When used in <nixpkgs>, callPackage will resolve overrides to whatever this is defined to in the scope of the calling function.

overriding describes the process of changing parameters of a function and therefore overrides is not a good name for this. See this discussion for some ideas around the topic.

Also I liked the readability and simplicity of the chosen approach. But that's just my opinion, maybe i am completely wrong about this.

lobre · 2020-01-17T17:30:44Z

deps.nix

@@ -0,0 +1,192 @@
+# file generated from go.mod using vgo2nix (https://github.com/adisbladis/vgo2nix)


By using nix to maintain dependencies, I understand that it feels more nix-friendly and pure but do you think that it is more robust than just keeping dependencies with go.mod?

In order to ensure purity of the build, we have to use fetchers from <nixpkgs>. To my knowledge there is no fetcher which does understand go.mod directly. I am unsure about the exact reasons for this but I do think go.mod was not considered sufficient to ensure a purity by nixpkgs. Maybe there is some loophole somewhere which could break purity.

This is why we have to convert go.mod into something nixpkgs understands.

lobre · 2020-01-17T17:35:30Z

Hi @Enteee,

I have started to have a look and try to grasp all of these nix concepts! I have reviewed and added some questions.

Maybe a last more global question here for you. Is it a common pattern to have the nix derivation building scripts inside the repository of the project and then only a simple fetchFromGitHub in nixpkgs?

Thanks,
Loric

Enteee · 2020-01-18T12:05:39Z

Thank you for your comments @lobre. I tried to answer your questions to my best knowledge.

Maybe a last more global question here for you. Is it a common pattern to have the nix derivation building scripts inside the repository of the project and then only a simple fetchFromGitHub in nixpkgs?

I try to do it in all of my projects, mostly for the sake of having a simple and well defined build workflow, a standardized and easy to set up development environment and a CI/CD pipeline support for keeping build inputs automatically in sync (i.e. deps.nix). But in general, I have not seen many projects doing the same.

My initial plan was to keep all the build relevant files here and just reference it then in nixpkgs. Sadly, this does not work because hydra (the nixpkgs build server) does not support IFDs. See my comment here and the linked discussion for more information.

On a side note, I wrote a few words about my experience with reMarkable including this project on my blog. I hope this is ok for you guys.

Enteee · 2020-05-26T07:30:01Z

With NixOS/nixpkgs#85898 rmapi is now maintained in nixpkgs. My original pull request (NixOS/nixpkgs#74657) was never merged. Therefore, I don't think there is much value in keeping this pull request open. I can do small changes to the derivation as well, but the official maintainer of the rmapi derivation in nixpkgs is @NickHu. So please get in touch with him in case you do need something changed.

Enteee added 9 commits November 3, 2019 18:31

Add nixos derivation

77949fc

* github workflow which updates deps.nix

Add nix-build

e060ca5

Remove deps

f410f3d

Better pull request message

daa7d01

Update deps.nix

33af4c7

Fail on outdated deps.nix

13d62de

Merge pull request #3 from Enteee/create-pull-request/patch-daa7d01

ebc8ee4

Update deps.nix

Merge branch 'feature/nix-derivation' of github.com:Enteee/rmapi into…

09472c7

… feature/nix-derivation

Development environment

65c869f

Enteee force-pushed the feature/nix-derivation branch 2 times, most recently from 68eabf6 to 656bc94 Compare November 6, 2019 07:21

only create pull request on push event

3d82895

Enteee force-pushed the feature/nix-derivation branch from 656bc94 to 3d82895 Compare November 6, 2019 07:24

Enteee added 6 commits November 6, 2019 08:29

test pr workflow action: remove deps.nix

a4a65c4

Update deps.nix

0a80056

Merge pull request #4 from Enteee/create-pull-request/patch-a4a65c4

75c7fd7

Update deps.nix

test workflow: outdated deps.nix

0301e82

Update deps.nix

62f365f

Merge pull request #5 from Enteee/create-pull-request/patch-0301e82

294ed43

Update deps.nix

juruen suggested changes Nov 6, 2019

View reviewed changes

README.md Outdated Show resolved Hide resolved

.github/workflows/nix.yml Show resolved Hide resolved

Update README.md

3994c0a

Co-Authored-By: Javier Uruen Val <juruen@github.com>

Merge remote-tracking branch 'upstream/master' into feature/nix-deriv…

92cdf49

…ation

Enteee mentioned this pull request Nov 29, 2019

rmapi: init at v0.0.7 NixOS/nixpkgs#74657

Closed

10 tasks

use buildGoModule

2b4653d

lobre reviewed Jan 17, 2020

View reviewed changes

doronbehar mentioned this pull request May 25, 2020

rmapi: init at 0.0.11 NixOS/nixpkgs#85898

Merged

10 tasks

Enteee closed this May 26, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature/nix derivation #78

Feature/nix derivation #78

Enteee commented Nov 5, 2019 •

edited

Loading

Enteee commented Nov 6, 2019

juruen left a comment

Enteee commented Nov 6, 2019 •

edited

Loading

lobre commented Nov 26, 2019

Enteee commented Dec 4, 2019

lobre left a comment

lobre Jan 17, 2020

Enteee Jan 18, 2020

lobre Jan 17, 2020

Enteee Jan 18, 2020

lobre Jan 17, 2020 •

edited

Loading

Enteee Jan 18, 2020 •

edited

Loading

lobre Jan 17, 2020

Enteee Jan 18, 2020

lobre commented Jan 17, 2020

Enteee commented Jan 18, 2020

Enteee commented May 26, 2020 •

edited

Loading

		@@ -0,0 +1,192 @@
		# file generated from go.mod using vgo2nix (https://github.com/adisbladis/vgo2nix)

Feature/nix derivation #78

Feature/nix derivation #78

Conversation

Enteee commented Nov 5, 2019 • edited Loading

To Do

Related

Enteee commented Nov 6, 2019

juruen left a comment

Choose a reason for hiding this comment

Enteee commented Nov 6, 2019 • edited Loading

lobre commented Nov 26, 2019

Enteee commented Dec 4, 2019

lobre left a comment

Choose a reason for hiding this comment

lobre Jan 17, 2020

Choose a reason for hiding this comment

Enteee Jan 18, 2020

Choose a reason for hiding this comment

lobre Jan 17, 2020

Choose a reason for hiding this comment

Enteee Jan 18, 2020

Choose a reason for hiding this comment

lobre Jan 17, 2020 • edited Loading

Choose a reason for hiding this comment

Enteee Jan 18, 2020 • edited Loading

Choose a reason for hiding this comment

lobre Jan 17, 2020

Choose a reason for hiding this comment

Enteee Jan 18, 2020

Choose a reason for hiding this comment

lobre commented Jan 17, 2020

Enteee commented Jan 18, 2020

Enteee commented May 26, 2020 • edited Loading

Enteee commented Nov 5, 2019 •

edited

Loading

Enteee commented Nov 6, 2019 •

edited

Loading

lobre Jan 17, 2020 •

edited

Loading

Enteee Jan 18, 2020 •

edited

Loading

Enteee commented May 26, 2020 •

edited

Loading