Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/nix derivation #78

Open
wants to merge 19 commits into
base: master
from
Open

Conversation

@Enteee
Copy link

Enteee commented Nov 5, 2019

Hi folks, NixOS support! 馃槃 馃殌

The things that I added are:

  • default.nix: a reproducible and pure builds with nix-build
  • shell.nix: a managed development environment with nix-shell
  • README.md: documentation on how to enter the development environment
  • deps.nix: freezes all build dependencies managed in go.mod
  • .github/workflows/nix.yml GitHub workflow which:
    • updates deps.nix if go.mod changes (automatically creates a pull request)
    • fails the build if deps.nix is not in sync with go.mod
    • runs nix-build which does build the binary & runs tests
  • derivation.nix: allows for easy integration into the nixpkgs repository with something like this:
{ callPackage, fetchFromGitHub, buildGoPackage, buildEnv, stdenv}:

let
  pname = "rmapi";
  repo = fetchFromGitHub {
    owner = "Enteee";
    repo = pname;
    rev = "65c869f7566ce66a517509219c8ba9f58483808e";
    sha256 = "sha256:0limyj7dim0vr3c357zbhvhrm3jg3jjiv95fb0bgif3ri9f03akj";
  };
  drv = callPackage "${repo}/derivation.nix" {
    buildGoPackage = super: buildGoPackage (super // {
      meta = super.meta // {
        maintainer = stdenv.lib.maintainers.Enteee;
      };
    });
  };
in
  buildEnv { name = drv.name; paths = [ drv ]; }

I did this mainly for myself, but thought maybe you would be interested in this as well. If you are interested in this PR and we merge this, i will then take it a step further an create a derivation in the nixpkgs repository so that all NixOS users can use this by just: nix-shell -p rmapi.

If not I will probably have to create a very similar pull request in the nixpkgs repository.
No hard feelings if you decide to reject this 馃槃 馃

To Do

  • I will keep my fork open and try to get that into nixpkgs using the approach documented in the description. This will hopefully trigger a code review of this pull request as well including nix contributors.
  • If I do succeed in getting this to nixpkgs and you are ok with the changes suggested, we merge this PR.
  • I will then create a new PR for nixpkgs which basically only changes the source repository from my fork to this repository.
  • 馃嵑

Related

Enteee added 9 commits Nov 3, 2019
* github workflow which updates deps.nix
Update deps.nix
@Enteee Enteee force-pushed the Enteee:feature/nix-derivation branch 2 times, most recently from 68eabf6 to 656bc94 Nov 6, 2019
@Enteee Enteee force-pushed the Enteee:feature/nix-derivation branch from 656bc94 to 3d82895 Nov 6, 2019
Enteee added 6 commits Nov 6, 2019
Update deps.nix
Update deps.nix
@Enteee

This comment has been minimized.

Copy link
Author

Enteee commented Nov 6, 2019

The last few commits and builds illustrate how the pull request functionality works.

The corresponding pull requests are:

Copy link
Owner

juruen left a comment

I've never used Nix myself so I can't review those bits

README.md Outdated Show resolved Hide resolved
.github/workflows/nix.yml Show resolved Hide resolved
Co-Authored-By: Javier Uruen Val <juruen@github.com>
@Enteee

This comment has been minimized.

Copy link
Author

Enteee commented Nov 6, 2019

I am currently not a 100% sure how flexible and open nixpkgs contributors are on merging pull request which basically just references a derivation stored in an other repository. It should not really matter because nix is side effect free but you never know. For easy maintenance of deps.nix, I would prefer it if all of the nix stuff was stored here instead of nixpkgs...

So if this is of any interest to the project, I would actually recommend the following for this pull request:

  1. I will keep my fork open and try to get that into nixpkgs using the approach documented in the description. This will hopefully trigger a code review of this pull request as well including nix contributors.
  2. If I do succeed in getting this to nixpkgs and you are ok with the changes suggested, we merge this PR.
  3. I will then create a new PR for nixpkgs which basically only changes the source repository from my fork to this repository.
  4. 馃嵑
@lobre

This comment has been minimized.

Copy link
Collaborator

lobre commented Nov 26, 2019

Hello,

Thanks for the PR!

I have plans for finally try and dig NixOS. I am really beginning to learn it to be able to switch to it as my main OS. Maybe I can review as soon as I have enough knowledge...

In the meantime, if any other Nix user feels comfortable for reviewing this, I would not see any problems to merge to have it merged...

Loric

@Enteee Enteee mentioned this pull request Nov 29, 2019
5 of 10 tasks complete
@Enteee

This comment has been minimized.

Copy link
Author

Enteee commented Dec 4, 2019

Copy link
Collaborator

lobre left a comment

Hey, sorry this will feel more like questions than a proper review because I am really starting to dig the Nix world (which by the way is awesome!). If you can help me understanding this by answering my questions, that would be super helpful to me.

{stdenv, buildGoModule}:

let
in buildGoModule rec {

This comment has been minimized.

Copy link
@lobre

lobre Jan 17, 2020

Collaborator

Does this pattern mean that someone using this derivation.nix can easily inject custom attributes to the set?

This comment has been minimized.

Copy link
@Enteee

Enteee Jan 18, 2020

Author

No, this is a custom builder provided by nixpkgs for builiding go programs managed with go modules. I tried to follow the official nixpkgs documentation, which suggest using buildGomodule. But yes, you could create an overlay for this if you wanted to 'inject' things.

{ pkgs ? import <nixpkgs> {} }:
with pkgs;

let

This comment has been minimized.

Copy link
@lobre

lobre Jan 17, 2020

Collaborator

What is the purpose of an empty let / in statement? Is it just by convention for potential future values?

This comment has been minimized.

Copy link
@Enteee

Enteee Jan 18, 2020

Author

Nah that's merely just a leftover. I used to have stuff in there which disappeared over time. I will remove this.


let
in pkgs.callPackage ./derivation.nix {
buildGoModule = super: buildGoModule (super // {

This comment has been minimized.

Copy link
@lobre

lobre Jan 17, 2020

Collaborator

As I am quite new in functional languages and nix in general, I might need a little extra help to understand these two lines 5 and 6! So of what I understand, buildGoModule is a function that should return a set that will represent the final derivation. So defining it like this woud allow the caller to provide a super set argument that would be merged with the body of the function here.

But what is the difference with having maybe something a little bit simpler like the following:

derivation.nix

{ stdend, overrides }:

overrides // {
    name = "rMAPI";
    src = ./.;
    goDeps = ./deps.nix
    ...
}

shell.nix

{ pkgs ? import <nixpkgs> {} }:

pkgs.callPackage ./derivation.nix {
    overrides = {
        name = "rMAPI-env";
        buildInputs = ...;
    };
}

Thanks for shedding light!

This comment has been minimized.

Copy link
@Enteee

Enteee Jan 18, 2020

Author

I do think, technically this would work. But I can see some possible downsides of this approach:

Also I liked the readability and simplicity of the chosen approach. But that's just my opinion, maybe i am completely wrong about this.

@@ -0,0 +1,192 @@
# file generated from go.mod using vgo2nix (https://github.com/adisbladis/vgo2nix)

This comment has been minimized.

Copy link
@lobre

lobre Jan 17, 2020

Collaborator

By using nix to maintain dependencies, I understand that it feels more nix-friendly and pure but do you think that it is more robust than just keeping dependencies with go.mod?

This comment has been minimized.

Copy link
@Enteee

Enteee Jan 18, 2020

Author

In order to ensure purity of the build, we have to use fetchers from <nixpkgs>. To my knowledge there is no fetcher which does understand go.mod directly. I am unsure about the exact reasons for this but I do think go.mod was not considered sufficient to ensure a purity by nixpkgs. Maybe there is some loophole somewhere which could break purity.

This is why we have to convert go.mod into something nixpkgs understands.

@lobre

This comment has been minimized.

Copy link
Collaborator

lobre commented Jan 17, 2020

Hi @Enteee,

I have started to have a look and try to grasp all of these nix concepts! I have reviewed and added some questions.

Maybe a last more global question here for you. Is it a common pattern to have the nix derivation building scripts inside the repository of the project and then only a simple fetchFromGitHub in nixpkgs?

Thanks,
Loric

@Enteee

This comment has been minimized.

Copy link
Author

Enteee commented Jan 18, 2020

Thank you for your comments @lobre. I tried to answer your questions to my best knowledge.

Maybe a last more global question here for you. Is it a common pattern to have the nix derivation building scripts inside the repository of the project and then only a simple fetchFromGitHub in nixpkgs?

I try to do it in all of my projects, mostly for the sake of having a simple and well defined build workflow, a standardized and easy to set up development environment and a CI/CD pipeline support for keeping build inputs automatically in sync (i.e. deps.nix). But in general, I have not seen many projects doing the same.

My initial plan was to keep all the build relevant files here and just reference it then in nixpkgs. Sadly, this does not work because hydra (the nixpkgs build server) does not support IFDs. See my comment here and the linked discussion for more information.

On a side note, I wrote a few words about my experience with reMarkable including this project on my blog. I hope this is ok for you guys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can鈥檛 perform that action at this time.