feat: use nix flake, std, tullia and cicero #176

jbgi · 2022-10-12T11:36:08Z

std-based alternative take to #175.

hamishmack · 2022-10-13T09:24:48Z

This looks really good. Is there a way to pass evalSystem to everything? I wanted to see if I could run nix flake show on macOS without using remote builder:

I tried the following, but I think the cross compiled versions of the project are still missing the evalSystem somehow.

diff --git a/cells/_automation/hydraJobs.nix b/cells/_automation/hydraJobs.nix
index 5d101b8..ceb7d5c 100644
--- a/cells/_automation/hydraJobs.nix
+++ b/cells/_automation/hydraJobs.nix
@@ -4,7 +4,7 @@
 }: let
   inherit (inputs.nixpkgs) lib;
   inherit (inputs.cells.prelude.library) project extra-compilers;
-  evalSystem = "x86_64-linux";
+  evalSystem = "x86_64-darwin";

   ciProject = project.appendModule {
     inherit evalSystem;
diff --git a/cells/prelude/library.nix b/cells/prelude/library.nix
index d949a48..9ed335a 100644
--- a/cells/prelude/library.nix
+++ b/cells/prelude/library.nix
@@ -41,6 +41,7 @@
     name = "cardano-prelude";
     inherit src;
     compiler-nix-name = lib.mkDefault default-compiler;
+    evalSystem = "x86_64-darwin";
     # Cross compilation support:
     crossPlatforms = p:
       lib.optionals pkgs.stdenv.hostPlatform.isLinux (
@@ -59,7 +60,7 @@
       # deduce package names from the cabal project to avoid hard-coding them:
       projectPackageNames = builtins.attrNames (haskellLib.selectProjectPackages
         (haskell-nix.cabalProject' {
-          inherit (config) name src compiler-nix-name cabalProjectLocal;
+          inherit (config) name src compiler-nix-name evalSystem cabalProjectLocal;
         })
         .hsPkgs);
     in [

jbgi · 2022-10-13T10:23:34Z

@hamishmack right, forgot to pass evalSystem here: bb6e26a
nix flake show should work now (with your local modifications).

cells/_automation/hydraJobs.nix

cells/_automation/tasks.nix

cells/prelude/apps.nix

jbgi · 2022-10-13T17:48:52Z

@dermetfan tried your suggestion in last commit, but it is still failing:

$ tullia run ci/hydraJobs.x86_64-linux.roots
...
[✗] error  ci/hydraJobs.x86_64-linux.roots                                                                9m11.235325358s
warning: Git tree '/home/jbgi/Dev/iohk/cardano-prelude' is dirty                                                         
warning: Using saved setting for 'allow-import-from-derivation = true' from ~/.local/share/nix/trusted-settings.json.    
warning: Using saved setting for 'extra-substituters = https://cache.iog.io' from ~/.local/share/nix/trusted-settings.jso
n.                                                                                                                       
warning: Using saved setting for 'extra-trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ='
from ~/.local/share/nix/trusted-settings.json.                                                                           
error (ignored): error: SQLite database '/home/jbgi/.cache/nix/eval-cache-                                               
v4/1553df62b97b634bce22160eed0bb4a05e9e574b5e5e752ad9c22245c16fd86e.sqlite' is busy                                      
error: flake 'git+file:///home/jbgi/Dev/iohk/cardano-prelude' does not provide attribute 'packages.x86_64-               
linux.tullia.x86_64-linux.task.ci/hydraJobs.x86_64-linux.roots.nsjail.run', 'legacyPackages.x86_64-linux.tullia.x86_64-  
linux.task.ci/hydraJobs.x86_64-linux.roots.nsjail.run' or 'tullia.x86_64-linux.task.ci/hydraJobs.x86_64-                 
linux.roots.nsjail.run'                                                                                                  
Failed to run /nix/store/i7zh5x24vsbv471zk7vadkgsa55qianz-nix-2.11.0/bin/nix build --json --no-link .#tullia.x86_64-     
linux.task.ci/hydraJobs.x86_64-linux.roots.nsjail.run: exit status 1

Also another issue is that even though I specify only one task, tullia execute all tasks... (almost crashing my machine: is there a way to limit concurrent jobs?).

dermetfan · 2022-10-17T14:26:14Z

There was an escaping issue in tullia. Fixed that and updated the tullia flake input.

is there a way to limit concurrent jobs?

I added a tullia task for each hydra job so you can now run them separately like this (without ci/ prefix):

nix run github:input-output-hk/tullia -- run hydraJobs.x86_64-linux.ghc924.plan-nix

jbgi · 2022-10-17T14:57:38Z

@dermetfan I'm now getting

$ nix run github:input-output-hk/tullia -- run x86_64-linux._automation.ciJobs.ghc924.plan-nix
warning: Using saved setting for 'allow-import-from-derivation = true' from ~/.local/share/nix/trusted-settings.json.
warning: Using saved setting for 'extra-substituters = https://cache.iog.io' from ~/.local/share/nix/trusted-settings.json.
warning: Using saved setting for 'extra-trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ=' from ~/.local/share/nix/trusted-settings.json.
[✗] error  x86_64-linux._automation.ciJobs.ghc924.plan-nix                                                  12.778438199s
[E][2022-10-17T14:55:32+0000][1] void subproc::subprocNewProc(nsjconf_t*, int, int, int, int, int)():205                 
execve('/nix/store/zspimn40hjy77c6bdmk24lwam87zwmy2-x86_64-linux._automation.ciJobs.ghc924.plan-nix/bin/'x86_64-         
linux._automation.ciJobs.ghc924.plan-nix'') failed: No such file or directory                                            
[F][2022-10-17T14:55:32+0000][1] bool subproc::runChild(nsjconf_t*, int, int, int, int)():448 Launching child process    
failed                                                                                                                   
[W][2022-10-17T16:55:32+0200][802842] bool subproc::runChild(nsjconf_t*, int, int, int, int)():478 Received error message
from the child process before it has been executed                                                                       
[E][2022-10-17T16:55:32+0200][802842] int nsjail::standaloneMode(nsjconf_t*)():256 Couldn't launch the child process     
Failed to run /nix/store/6cka89h4lly1m068a2lv8i99gvp8a0x6-x86_64-linux._automation.ciJobs.ghc924.plan-nix-nsjail/bin/x86_
64-                                                                                                                      
linux._automation.ciJobs.ghc924.plan-nix-nsjail: exit status 255                                                         
4:55PM FTL starting supervisor error="running x86_64-linux._automation.ciJobs.ghc924.plan-nix: Failed to run /nix/store/6cka89h4lly1m068a2lv8i99gvp8a0x6-x86_64-linux._automation.ciJobs.ghc924.plan-nix-nsjail/bin/x86_64-linux._automation.ciJobs.ghc924.plan-nix-nsjail: exit status 255"

dermetfan · 2022-10-18T09:19:55Z

That's just nsjail having trouble with cgroups when it tries to actually run the task (not yet well tested). Evaluation and building worked fine so it works on cicero, as for cicero the nomad + oci runtime is used instead of nsjail.

.envrc

cardano-prelude-test/cardano-prelude-test.cabal

cells/_automation/ciJobs.nix

cells/_automation/devshells.nix

cells/_automation/nixago.nix

cells/_automation/tasks.nix

cells/prelude/library.nix

release.nix

cells/prelude/library.nix

michaelpj · 2022-10-19T18:44:20Z

On reflection, a high-level thought. This PR contains hundreds of lines of nix code providing lots of features, but explains nothing about how to use it or work with it. To put it another way: cardano-prelude needs a CONTRIBUTING, and if we're going to add all this stuff it desperately needs one. What is all this stuff? What is lefthook? What is nixago? What's up with this editorconfig thing? How do I get a shell? How do I update things? What's the CI setup? How do I change the compilers it's testing against? At the moment nobody will know any of this except @jbgi

Yes, that's a lot to write about, but this PR adds a lot of stuff. If we want to keep it minimal instead that would be fine but if we're going to go big IMO we have to document things.

jbgi · 2022-10-19T20:31:59Z

Sure. Most std stuff can be documented with links to the appropriate std doc page.
I'll work on it next week.

michaelpj · 2022-10-20T11:40:50Z

I think that's good for maintainers of the nix code, but we also need to document things in a form that's comprehensible by people working on the Haskell code, hence CONTRIBUTING.

jbgi · 2022-10-26T15:10:00Z

@michaelpj added CONTRIBUTING.md in last commit (albeit rather minimal for now, but should be enough to not block any dev workflow).

dermetfan · 2023-01-09T14:53:38Z

I pushed a fix for the latest tullia version, maybe squash again @jbgi (didn't want to squash and force push to not confuse anybody)

bors.toml

cells/_automation/tasks.nix

coot · 2023-01-13T08:21:13Z

@erikd, @newhoggy, @lehins are you ok with requiring fourmolu formatter?

lehins · 2023-01-13T11:46:30Z

are you ok with requiring fourmolu formatter?

@coot I encourage requiring it!

blaggacao

For the formatter-only changes this will be handy: https://docs.github.com/en/repositories/working-with-files/using-files/viewing-a-file#ignore-commits-in-the-blame-view

Please let me know or open an upstream ticket if you need anything from Standard. Even the ever so slightest improvement ideas are welcome!

blaggacao · 2022-12-22T12:59:59Z

flake.nix

+  # This is a template created by `hix init`
+  inputs = {
+    nixpkgs = {
+      follows = "std/nixpkgs";


Depending on the circumstances, it may be better to set, not follow nixpkgs, here. For example, it gives you more control over cache invalidations and decouples them from std.

You still then do std.inputs.nixpkgs.follows.

blaggacao · 2023-01-13T13:49:37Z

CONTRIBUTING.md

+
+### Commit message
+
+Commit messages should ideally be of the form `<type>[(<scope>)]: <short description>]` ([conventionnal](https://www.conventionalcommits.org/) commit message). Available commit types are from [std presets](https://std.divnix.com/reference/presets/nixago/conform.html).


Heads up: In my experience, the default scopes (one scope per cell) is often times not enough. So as this is being trained into muscle memory and used by people, I'd suggest at least a refinement on the scopes.

jbgi · 2023-01-18T10:57:04Z

bors r+

jbgi requested review from dermetfan and hamishmack October 12, 2022 11:48

jbgi force-pushed the jbgi/nix-flake-std branch 3 times, most recently from 570476b to 18e3612 Compare October 12, 2022 22:40

dermetfan reviewed Oct 13, 2022

View reviewed changes

cells/_automation/hydraJobs.nix Outdated Show resolved Hide resolved

cells/_automation/tasks.nix Outdated Show resolved Hide resolved

cells/_automation/tasks.nix Outdated Show resolved Hide resolved

cells/prelude/apps.nix Outdated Show resolved Hide resolved

hamishmack mentioned this pull request Oct 14, 2022

Use nix flake, tullia and cicero #175

Closed

dermetfan force-pushed the jbgi/nix-flake-std branch from 76c6bee to 2dbea5a Compare October 17, 2022 14:21

jbgi force-pushed the jbgi/nix-flake-std branch 2 times, most recently from 5e107d2 to 5431e2b Compare October 18, 2022 13:29

jbgi requested a review from michaelpj October 18, 2022 13:37

michaelpj approved these changes Oct 18, 2022

View reviewed changes

jbgi force-pushed the jbgi/nix-flake-std branch from 5431e2b to 19e5cdb Compare October 19, 2022 18:53

jbgi force-pushed the jbgi/nix-flake-std branch from 19e5cdb to affe919 Compare October 20, 2022 08:29

jbgi mentioned this pull request Oct 20, 2022

colons in packages/apps names break std cli divnix/std#167

Closed

jbgi force-pushed the jbgi/nix-flake-std branch 5 times, most recently from 82cdc3f to ce8aee6 Compare October 26, 2022 15:08

dermetfan approved these changes Jan 9, 2023

View reviewed changes

bors.toml Outdated Show resolved Hide resolved

dermetfan reviewed Jan 9, 2023

View reviewed changes

cells/_automation/tasks.nix Show resolved Hide resolved

jbgi force-pushed the jbgi/nix-flake-std branch 2 times, most recently from 8de0368 to e3434a6 Compare January 9, 2023 15:45

dermetfan force-pushed the jbgi/nix-flake-std branch from 7ace23c to dfe062c Compare January 9, 2023 16:44

jbgi force-pushed the jbgi/nix-flake-std branch 7 times, most recently from f6ccd17 to c8cf8e6 Compare January 12, 2023 14:43

jbgi force-pushed the jbgi/nix-flake-std branch from bd64001 to 46c1c23 Compare January 13, 2023 13:34

blaggacao reviewed Jan 13, 2023

View reviewed changes

hamishmack and others added 2 commits January 17, 2023 13:04

feat: use nix flake, std, tullia and cicero

1babc2b

docs: add CONTRIBUTING.md

5a05ab7

dermetfan force-pushed the jbgi/nix-flake-std branch from 46c1c23 to 27bf675 Compare January 17, 2023 12:05

jbgi force-pushed the jbgi/nix-flake-std branch 2 times, most recently from e3e90e4 to 7ae03d3 Compare January 17, 2023 13:07

feat: activate fourmolu formatting of haskell file

be48b95

jbgi force-pushed the jbgi/nix-flake-std branch 2 times, most recently from cc6b4a0 to 32d8f0f Compare January 18, 2023 09:58

actions: separate required/nonrequired (do not use after for now)

53d8059

jbgi force-pushed the jbgi/nix-flake-std branch from 32d8f0f to 53d8059 Compare January 18, 2023 10:35

jbgi merged commit 94b3748 into master Jan 18, 2023

dermetfan deleted the jbgi/nix-flake-std branch January 19, 2023 09:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: use nix flake, std, tullia and cicero #176

feat: use nix flake, std, tullia and cicero #176

jbgi commented Oct 12, 2022

hamishmack commented Oct 13, 2022

jbgi commented Oct 13, 2022

jbgi commented Oct 13, 2022 •

edited

Loading

dermetfan commented Oct 17, 2022

jbgi commented Oct 17, 2022 •

edited

Loading

dermetfan commented Oct 18, 2022

michaelpj commented Oct 19, 2022

jbgi commented Oct 19, 2022 •

edited

Loading

michaelpj commented Oct 20, 2022

jbgi commented Oct 26, 2022 •

edited

Loading

dermetfan commented Jan 9, 2023

coot commented Jan 13, 2023

lehins commented Jan 13, 2023

blaggacao left a comment

blaggacao Dec 22, 2022

blaggacao Jan 13, 2023

jbgi commented Jan 18, 2023


		### Commit message

		Commit messages should ideally be of the form `<type>[(<scope>)]: <short description>]` ([conventionnal](https://www.conventionalcommits.org/) commit message). Available commit types are from [std presets](https://std.divnix.com/reference/presets/nixago/conform.html).

feat: use nix flake, std, tullia and cicero #176

feat: use nix flake, std, tullia and cicero #176

Conversation

jbgi commented Oct 12, 2022

hamishmack commented Oct 13, 2022

jbgi commented Oct 13, 2022

jbgi commented Oct 13, 2022 • edited Loading

dermetfan commented Oct 17, 2022

jbgi commented Oct 17, 2022 • edited Loading

dermetfan commented Oct 18, 2022

michaelpj commented Oct 19, 2022

jbgi commented Oct 19, 2022 • edited Loading

michaelpj commented Oct 20, 2022

jbgi commented Oct 26, 2022 • edited Loading

dermetfan commented Jan 9, 2023

coot commented Jan 13, 2023

lehins commented Jan 13, 2023

blaggacao left a comment

Choose a reason for hiding this comment

blaggacao Dec 22, 2022

Choose a reason for hiding this comment

blaggacao Jan 13, 2023

Choose a reason for hiding this comment

jbgi commented Jan 18, 2023

jbgi commented Oct 13, 2022 •

edited

Loading

jbgi commented Oct 17, 2022 •

edited

Loading

jbgi commented Oct 19, 2022 •

edited

Loading

jbgi commented Oct 26, 2022 •

edited

Loading