Gogs: init at 20160304-d57a2b9 #13566

valeriangalliat · 2016-02-29T04:23:12Z

The default.nix is generated using go2nix and applying a couple of patches to make the SQLite support configurable and wrapping the program.

Gogs behavior is a bit modified by the wrapper program because original Gogs is impure (it expects to be run only in its source Git repository and needs to be able to write files inside. To avoid this, I define the GOGS_WORK_DIR to be the CWD if not defined, and cd inside this directory before executing Gogs (if Gogs is not run from inside the GOGS_WORK_DIR, some parts still rely on the CWD to be the work directory).

Also the public and templates directories from the Go package directory are symlinked inside the work directory since Gogs expect them to be here.

This PR comes with a maintenance script, update-gogs-definition, that is basically doing an impure go get of chosen Gogs revision and its dependencies, and runs go2nix over this tree (plus some specific patches).

For information, I also have a WIP NixOS module integration for Gogs but I'll add it later, or in another PR, once I'm done with it.

Dependencies

I'm not sure about the way dependencies are handled here. It's plain go2nix but it's not making those dependencies shareable... but it would be a huge work to go through those 46 dependencies, and convert them in nixpkgs.goPackages format, see if they exist already and in a compatible version or add the definition, and requiring to manually recreate the dependency graph.

I'd really like some input about the manner to handle this.

Things done:

Tested using sandboxing (nix-build --option build-use-chroot true or nix.useChroot on NixOS)
Built on platform(s): NixOS
- Builds on OSX but the update-gogs-definition script does not work because go get on OSX in unable to verify the certificates of gopkg.in and I couldn't find a way to fix this.
Tested execution of all binary files (usually in ./result/bin/)

valeriangalliat · 2016-02-29T04:24:58Z

pkgs/applications/version-management/gogs/default.nix

+  postInstall = ''
+    wrapProgram $bin/bin/gogs \
+      --prefix PATH : ${git}/bin \
+      --run 'export GOGS_WORK_DIR=${"$" + "{"}GOGS_WORK_DIR:-$PWD}' \


This line looks odd, because I didn't find any way to escape ${ inside a Nix string.

The documentation says to put a \ in front of it but this did not work for me, still yielding unexpected $undefined error.

Use the nix string open / close symbol to escape, ex ''${}
On Sun, Feb 28, 2016 at 8:25 PM Valérian Galliat notifications@github.com
wrote:

In pkgs/applications/version-management/gogs/default.nix
#13566 (comment):

+with goPackages;
+
+buildGoPackage rec {

name = "gogs-${version}";

version = "20160227-${stdenv.lib.strings.substring 0 7 rev}";

rev = "d320915ad2a7b4bbab075b98890aa50f91f0ced5";

buildInputs = [ makeWrapper go ];

buildFlags = lib.optional (sqliteSupport) "-tags sqlite";

goPackagePath = "github.com/gogits/gogs";

postInstall = ''

wrapProgram $bin/bin/gogs \

--prefix PATH : ${git}/bin \

--run 'export GOGS_WORK_DIR=${"$" + "{"}GOGS_WORK_DIR:-$PWD}' \

This line looks odd, because I didn't find any way to escape ${ inside a
Nix string.

The documentation http://nixos.org/nix/manual/#idm46912467695696 says
to put a \ in front of it but this did not work for me, still yielding unexpected
$undefined error.

—
Reply to this email directly or view it on GitHub
https://github.com/NixOS/nixpkgs/pull/13566/files#r54368220.

jagajaga · 2016-02-29T15:20:40Z

Please properly name commit to follow our rules in CONTRIBUTING.md.

valeriangalliat · 2016-02-29T15:44:12Z

@jagajaga Thanks, updated.

jgillich · 2016-02-29T16:16:13Z

I'm not too familiar with the go stuff, but why isn't this part of goPackages?

Also, the correct commit message would be gogs: init at 20160227

valeriangalliat · 2016-02-29T16:35:13Z

@jgillich Oh sorry I didn't get it for the commit message. Updated.

About the goPackages, that's the doubt/question I have in the Dependencies part of my PR message; I used go2nix to generate this derivation but it makes a single derivation for the Gogs package and all its dependencies at an exact version merged together, instead of a shared dependencies tree like in goPackages. I'm aware of no tooling to do this yet, and doing it manually would take much time, both now and eveytime we update Gogs (compared to the current fully automated update-gogs-definition script).

joachifm · 2016-03-05T13:28:17Z

Looks like a legit build failure:

# github.com/gogits/gogs/models
go/src/github.com/gogits/gogs/models/repo.go:374: cannot use true (type bool) as type git.PullRemoteOptions in argument to git.Pull
builder for ‘/nix/store/7bbw82k9zd4fibzgz7g9z22dy40ilxl1-go1.5-gogs-20160227-d320915.drv’ failed with exit code 44
error: build of ‘/nix/store/7bbw82k9zd4fibzgz7g9z22dy40ilxl1-go1.5-gogs-20160227-d320915.drv’ failed
The invocation of "nix-build -A gogs /home/travis/.nox/nixpkgs" failed

valeriangalliat · 2016-03-05T14:28:16Z

Oops right, thanks, related to gogs/gogs#2728, I just updated Gogs and its dependencies.

joachifm · 2016-03-05T15:16:42Z

pkgs/applications/version-management/gogs/default.nix.patch

@@ -0,0 +1,22 @@
+2,3c2


I don't understand why this exists? Patching default.nix is a bit odd.

@joachifm default.nix is generated by go2nix, but it cannot guess the postInstall hook required to wrap Gogs (because it's specific to Gogs), nor the optional SQLite support with an argument.

That's why, when auto-generating default.nix using gogs-derivation.nix, I patch the output of go2nix to include the sqliteSupport argument, and postInstall hook, which makes the final default.nix.

All of this is done inside nix-build --no-out-link gogs-definition.nix.

Maybe I could achieve the same by having the raw output of go2nix in default-generated.nix, and overriding it in default.nix to add sqliteSupport and postInstall?

This seems a bit complex for my tastes, but I'm simply not qualified to comment on go packaging, sorry.

cc @vcunat @domenkozar

I immediately don't see any better way than this.

I also didn't know about go2nix, so perhaps its users might have a better idea how to handle such a situation. Maybe @cstrahan?

Thanks a lot @cstrahan, I think it's possible to override the go2nix output to add the postInstall hook, and tweak the buildFlags, it's indeed a lot cleaner than patching default.nix.

However the conditional sqliteSupport is a bit trickier, since it needs to remove an entry from extraSrcs if sqliteSupport is false, and extraSrcs is an argument to buildGoPackage, but not an attribute of the actual derivation, so I think that lib.overrideDerivation will not help. I can just ignore this part, it will result in downloading an unnecessary Go package if sqliteSupport is false which is not that bad.

But if anyone have an idea how to gracefully handle this with overriding, I would happily take it. 😄

I agree that generated defaults should be easily overridable. I'll look at haskell stuff and try to make it better.

I tried to separate default.nix from generated one in kamilchm/go2nix#10 but it looks like the allowGoReference flag doesn't work :/
Is there a better way to override generated file than import?

@kamilchm Oh, nice! But yes allowGoReference is an argument of buildGoPackage and lib.overrideDerivation allows only to custom the derivation attributes.

Is there some kind of override mechanism in Nix that would allow to customize the arguments to the buildGoPackage call?

An idea would be to tweak the generated.nix like this:

{ stdenv, lib, goPackages, fetchgit, fetchhg, fetchbzr, fetchsvn, args }: with goPackages; buildGoPackage (rec { name = "go2nix-${version}"; # ... } // args)

And call it like this:

callPackage ./generated.nix { args = { allowGoReference = true; # ... }; }

Or even allow to pass these arguments in the top-level function instead of the args set, like @cstrahan suggested, if I understood well? Something like the following example would avoid listing explicitly all possible overridable attributes, but I don't like this filterAttrs call, maybe there's a better way to get all the remaining args).

{ stdenv, lib, goPackages, fetchgit, fetchhg, fetchbzr, fetchsvn, ... } @ args: with goPackages; buildGoPackage (rec { name = "go2nix-${version}"; # ... } // (lib.filterAttrs (n: v: n != "lib" && n != "stdenv" && n != "goPackages" && n != "fetchbzr" && n != "fetchgit" && n != "fetchsvn" && n != "fetchhg") args))

@cstrahan what do you think about these options?

valeriangalliat · 2016-03-07T20:03:12Z

Quick update of gogs-derivation.nix to remove usage of my fork of go2nix and just use goPackages.go2nix since #13741, #13564, and #13533 are merged.

kamilchm · 2016-06-11T14:20:58Z

@valeriangalliat see new go2nix from master and how i rework goPackages in #16017

garbas · 2016-07-23T00:05:12Z

@valeriangalliat ping

Introduced with #16017.

valeriangalliat · 2016-07-28T17:14:34Z

Thanks a lot @kamilchm, I updated the PR to use new go2nix generated default.nix and deps.json.

I can't build it though:

stripping (with flags -S) in /nix/store/wca3v40cfbrman7xxnmaww7i925ha6qy-go1.6-gogs-20160728-ad7ea88-bin/bin
strip: can't process non-object and non-archive file: /nix/store/wca3v40cfbrman7xxnmaww7i925ha6qy-go1.6-gogs-20160728-ad7ea88-bin/bin/gogs
patching script interpreter paths in /nix/store/wca3v40cfbrman7xxnmaww7i925ha6qy-go1.6-gogs-20160728-ad7ea88-bin
cycle detected in the references of ‘/nix/store/3kqbi3as85hshj1kxifhqsdmlprqnad7-go1.6-gogs-20160728-ad7ea88’
error: build of ‘/nix/store/hhr8wdwfyxiz2ccwny9jbalg4gqwwvxw-go1.6-gogs-20160728-ad7ea88.drv’ failed

Maybe something doesn't like the fact that because of wrapProgram, the Gogs wrapper shell script references the Gogs package to actually access the binary?

kamilchm · 2016-07-29T05:35:51Z

What about last line with ln in postInstall. It looks like it could generate a cycle?

valeriangalliat · 2016-07-29T14:38:24Z

I removed whole postInstall and still have the same problem, not sure how to investigate on this.

grep -R /nix/store/ipgqsf1myv4fk5dd9r5pv0lgpkf5kps9-go1.6-gogs-20160728-ad7ea88 /nix/store/zmama6l4ai64adzix0bfc8lwkc81bnz9-go1.6-gogs-20160728-ad7ea88-bin
Binary file /nix/store/zmama6l4ai64adzix0bfc8lwkc81bnz9-go1.6-gogs-20160728-ad7ea88-bin/bin/gogs matches
Binary file /nix/store/zmama6l4ai64adzix0bfc8lwkc81bnz9-go1.6-gogs-20160728-ad7ea88-bin/bin/gogs-ad7ea88 matches

Not sure if this is OK or if this is the problem?

kamilchm · 2016-07-29T16:21:03Z

I think it's related to NixOS/nix#494
It could be a lot easier to debug with NixOS/nix#925
It builds fine for me when I remove the ln so I suspect that you can't do wrapProgram in $bin output when the wrapper needs anything from $out output.
I would try to install data files into a separate output and link it then from $bin

kamilchm · 2016-07-29T16:38:43Z

It works fine when I do kamilchm@c491474
But I wonder if there could be a better way to use templates and public than linking it in PWD?
Maybe we should use go-bindata to embed static files into the binary? https://github.com/gogits/gogs/blob/master/Makefile#L46

PS. Don't forget to add metadata https://nixos.org/nixpkgs/manual/#sec-standard-meta-attributes ;)

valeriangalliat · 2016-07-29T18:28:28Z

Oh thanks, I'll use this then, I'll update the PR this week-end.

Other than modifying Gogs source code, last time I checked, I could think of any other way than linking templates in GOGS_WORK_DIR and also running the program from GOGS_WORK_DIR.

Because Gogs expects to be run from its Go repository, they import / read some paths relative to CWD, and some others from GOGS_WORK_DIR.

yorickvP · 2016-09-11T14:55:03Z

@valeriangalliat what's the status of this?

valeriangalliat · 2016-09-12T19:14:53Z

Forgot about it, sorry, just pulled @kamilchm patch, hopefully the tests will pass.

yorickvP · 2016-09-12T20:25:14Z

Don't pay too much attention to the tests #6652
But do pay some attention, it seems like it has something to do with it this time.

domenkozar · 2016-09-13T09:03:04Z

In this care the error on OSX is legit.

Mic92 · 2016-10-22T18:25:57Z

This pull request is superseded by #19361

fpletz · 2016-11-28T03:43:43Z

Closing in favor of #19361. Thanks for your contribution!

valeriangalliat reviewed Feb 29, 2016
View reviewed changes

valeriangalliat changed the title ~~Add Gogs package~~ Gogs: add derivation Feb 29, 2016

valeriangalliat changed the title ~~Gogs: add derivation~~ Gogs: init at 20160227-d320915 Feb 29, 2016

joachifm added the 8.has: package (new) label Mar 2, 2016

valeriangalliat changed the title ~~Gogs: init at 20160227-d320915~~ Gogs: init at 20160304-d57a2b9 Mar 5, 2016

joachifm reviewed Mar 5, 2016
View reviewed changes

joachifm added the 6.topic: golang label Mar 6, 2016

lucabrunox added the 2.status: work-in-progress label Mar 11, 2016

rushmorem mentioned this pull request Jun 9, 2016

Rework goPackages #16017

Merged

7 tasks

valeriangalliat added 2 commits July 28, 2016 11:20

Gogs: init at 20160227-d320915

9dac4d3

Gogs: use new go2nix and goPackages format

dd5400a

Introduced with #16017.

gogs: FIX public and templates use from bin

c491474

schneefux mentioned this pull request Oct 8, 2016

gogs: init at v0.9.97 #19361

Merged

6 tasks

fpletz closed this Nov 28, 2016

schneefux mentioned this pull request Aug 12, 2017

gogs: Wrapper causes symlinks to be created all over the place #27866

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gogs: init at 20160304-d57a2b9 #13566

Gogs: init at 20160304-d57a2b9 #13566

valeriangalliat commented Feb 29, 2016

valeriangalliat Feb 29, 2016

wkennington Feb 29, 2016

valeriangalliat Feb 29, 2016

jagajaga commented Feb 29, 2016

valeriangalliat commented Feb 29, 2016

jgillich commented Feb 29, 2016

valeriangalliat commented Feb 29, 2016

joachifm commented Mar 5, 2016

valeriangalliat commented Mar 5, 2016

joachifm Mar 5, 2016

valeriangalliat Mar 5, 2016

joachifm Mar 6, 2016

jagajaga Mar 6, 2016

vcunat Mar 6, 2016

valeriangalliat Mar 7, 2016

kamilchm Mar 7, 2016

kamilchm Mar 8, 2016

valeriangalliat Mar 8, 2016

kamilchm Mar 9, 2016

valeriangalliat commented Mar 7, 2016

kamilchm commented Jun 11, 2016

garbas commented Jul 23, 2016

valeriangalliat commented Jul 28, 2016

kamilchm commented Jul 29, 2016

valeriangalliat commented Jul 29, 2016

kamilchm commented Jul 29, 2016 •

edited

kamilchm commented Jul 29, 2016 •

edited

valeriangalliat commented Jul 29, 2016

yorickvP commented Sep 11, 2016

valeriangalliat commented Sep 12, 2016

yorickvP commented Sep 12, 2016 •

edited

domenkozar commented Sep 13, 2016

Mic92 commented Oct 22, 2016

fpletz commented Nov 28, 2016

Gogs: init at 20160304-d57a2b9 #13566

Gogs: init at 20160304-d57a2b9 #13566

Conversation

valeriangalliat commented Feb 29, 2016

Dependencies

Things done:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jagajaga commented Feb 29, 2016

valeriangalliat commented Feb 29, 2016

jgillich commented Feb 29, 2016

valeriangalliat commented Feb 29, 2016

joachifm commented Mar 5, 2016

valeriangalliat commented Mar 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

valeriangalliat commented Mar 7, 2016

kamilchm commented Jun 11, 2016

garbas commented Jul 23, 2016

valeriangalliat commented Jul 28, 2016

kamilchm commented Jul 29, 2016

valeriangalliat commented Jul 29, 2016

kamilchm commented Jul 29, 2016 • edited

kamilchm commented Jul 29, 2016 • edited

valeriangalliat commented Jul 29, 2016

yorickvP commented Sep 11, 2016

valeriangalliat commented Sep 12, 2016

yorickvP commented Sep 12, 2016 • edited

domenkozar commented Sep 13, 2016

Mic92 commented Oct 22, 2016

fpletz commented Nov 28, 2016

kamilchm commented Jul 29, 2016 •

edited

kamilchm commented Jul 29, 2016 •

edited

yorickvP commented Sep 12, 2016 •

edited