Skip to content

Commit

Permalink
Migrate to nix flake
Browse files Browse the repository at this point in the history
  • Loading branch information
jbgi committed May 31, 2023
1 parent 78ed69a commit aec6b78
Show file tree
Hide file tree
Showing 74 changed files with 1,789 additions and 1,597 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/bench.yml
Expand Up @@ -22,7 +22,7 @@ jobs:
trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= loony-tools:pr9m4BkM/5/eSTZlkQyRt57Jz7OMBxNSUiMC4FkcNfk=
substituters = https://cache.iog.io https://cache.nixos.org/ https://cache.zw3rk.com
- name: Build benchmarks
run: nix-build -A haskellPackages.cardano-ledger-test.components.benchmarks
run: nix build .#cardano-ledger-test:bench:bench
- name: Run benchmark
run: |
cd libs/cardano-ledger-test
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/gh-pages.yml
Expand Up @@ -20,7 +20,7 @@ jobs:
trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= loony-tools:pr9m4BkM/5/eSTZlkQyRt57Jz7OMBxNSUiMC4FkcNfk=
substituters = https://cache.iog.io https://cache.nixos.org/ https://cache.zw3rk.com
- name: Build projects and haddocks
run: nix-shell --command "cabal update && cabal build --enable-documentation all && ./scripts/haddocks.sh"
run: nix develop --command bash -c 'cabal update && cabal build --enable-documentation all && ./scripts/haddocks.sh'
- name: Add files
run: |
git config --local user.name ${{ github.actor }}
Expand All @@ -37,4 +37,3 @@ jobs:
branch: gh-pages
force: true
directory: .

6 changes: 2 additions & 4 deletions .github/workflows/haskell.yml
Expand Up @@ -198,7 +198,7 @@ jobs:
run: ./scripts/gen-hie.sh

nix-build:
name: Build nix derivations and enter nix-shell
name: Build nix required derivations
if: ${{ github.event_name == 'schedule' }}
runs-on: ubuntu-latest
steps:
Expand All @@ -210,6 +210,4 @@ jobs:
trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= loony-tools:pr9m4BkM/5/eSTZlkQyRt57Jz7OMBxNSUiMC4FkcNfk=
substituters = https://cache.iog.io https://cache.nixos.org/ https://cache.zw3rk.com
- name: nix-build
run: nix-build
- name: nix-shell
run: nix-shell shell.nix
run: nix build .#hydraJobs.required
3 changes: 1 addition & 2 deletions .github/workflows/push-specs.yml
Expand Up @@ -20,7 +20,7 @@ jobs:
trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
substituters = https://cache.iog.io https://cache.nixos.org/
- name: Build specs
run: nix-build -A specs -o output/spec
run: nix build .#specs -o output/spec
- name: Collect dereferenced docs
run: |
mkdir -p deref
Expand All @@ -32,4 +32,3 @@ jobs:
uses: ncipollo/release-action@v1
with:
artifacts: "deref/*"

42 changes: 18 additions & 24 deletions CONTRIBUTING.md
Expand Up @@ -38,10 +38,11 @@ We are transitioning to use GHC 9.2 rather than GHC 8.10.
We need to retain 8.10 compatibility until we are sure that the Cardano node can switch over to 9.2 without any problems.
At that point we can drop it.

The main `nix-shell` will now give you a GHC 9.2 compiler, but you can get a GHC 8.10 shell by calling
The main `nix develop` shell will now give you a GHC 9.2 compiler, but you can get a GHC 8.10 shell by calling
```
nix-shell --arg config '{ haskellNix.compiler = "ghc8107"; }'
nix develop .#ghc8107
```
(this pattern can also be used to test any supported GHC version)

## Updating dependencies

Expand Down Expand Up @@ -189,15 +190,15 @@ NIGHTLY=true cabal test cardano-ledger-shelley-test

We have support for running
[ghcid](https://github.com/ndmitchell/ghcid)
from inside of `nix-shell`.
Enter `nix-shell` from the base directory of the repository,
from inside of `nix develop`.
Enter `nix develop` from the base directory of the repository,
change directories to the cabal package that you wish to check,
then run `ghcid`.

For example:

```shell
nix-shell
nix develop
cd eras/shelley/impl/
ghcid
```
Expand All @@ -214,12 +215,12 @@ Error: cabal: Cannot open a repl for multiple components at once. The target ''
Specifying the component solves this problem:

```shell
nix-shell
nix develop
cd libs/cardano-ledger-binary/
ghcid testlib # or `ghcid cardano-ledger-binary`
```

## nix-build Infrastructure
## nix build Infrastructure

The artifacts in this repository can be built and tested using nix. This is
additionally used by the Hydra CI to test building, including cross-compilation
Expand All @@ -231,11 +232,11 @@ To add a new Haskell package, you should do the following:

1. Create the project in the usual way. It should have an appropriate `.cabal` file.
2. Test that you can build your new project by running the following: `nix build
-f default.nix libs.<project_name>`. If you have executables, then
you may also try building these using the `exes.<executable_name>`
attribute path. A good way to see what's available is to execute `:l
default.nix` in `nix repl`. This will allow you to explore the potential
attribute names by using tab completion on "libs.".
.#<project_name>:lib:<lib_name>`. If you have executables, then
you may also try building these using the `.#<project_name>:exe:<exe_name>`
attribute path. A good way to see what's available is to execute `:lf .`
in `nix repl`. This will allow you to explore the potential
attribute names by using tab completion on "packages.<your_system>".

### To add a new LaTeX specification

Expand All @@ -246,16 +247,11 @@ from [the Shelley ledger spec](./eras/shelley/formal-spec)).
1. Copy these files into the root of your new LaTeX specification.
2. Modify the `DOCNAME` in the `Makefile`.
3. Update `default.nix` to:
1. Make sure that the relative path in the first line is pointing to
(default.nix)[./default.nix]. This is used to pin the
`nixpkgs` version used to build the LaTeX specifications.
2. Update the `buildInputs` to add in any LaTeX packages you need in your
1. Update the `buildInputs` to add in any LaTeX packages you need in your
document, and remove any unneeded ones.
3. Alter the `meta` description field to reflect the nature of this document.
4. Add a link to the package at the bottom of [default.nix](./default.nix),
2. Alter the `meta` description field to reflect the nature of this document.
4. Add a link to the package near the bottom of [flake.nix](./flake.nix),
following the existing examples.
5. To require that your specification be built in CI, add it at the end of the
list in [default.nix](./default.nix) following the existing examples.

### Additional documentation

Expand Down Expand Up @@ -357,11 +353,9 @@ cabal configure --enable-profiling --profiling-detail=all-functions
```

Now we need to run a node to build up the dataabase.
This can be done in the cardano-node repository by
opening a nix-shell and running:
This can be done in the cardano-node repository by running:
```
nix build -f default.nix scripts.mainnet.node
./result/bin/cardano-node-mainnet
nix run .#mainnet/node
```
This will take a very long time.
You can stop the node once it is past any slots that you care about.
Expand Down
37 changes: 19 additions & 18 deletions HowToProfileLedger.md
Expand Up @@ -23,8 +23,8 @@ greater detail about each step

1. Decide what you want to profile, and arrange the code so this is possible
2. Choreograph the dance between nix, cabal, and ghc. This has two parts
a. Adding stuff to files like cabal.project, cabal.project.local, and nix/haskel.nix
b. Passing the right flags to nix-shell, cabal, and ghc
a. Adding stuff to files like cabal.project, cabal.project.local
b. Passing the right flags to cabal, and ghc, running the right `nix develop` shell,
3. Recompiling everything so it can be profiled. This takes a very long time (greater than
30 minutes when I did it)
4. Start up the profiling. I used "cabal test" with just the right command line arguments.
Expand Down Expand Up @@ -103,53 +103,55 @@ package plutus-core
---------------------

Supposedly this makes things work. But I could never get the nix-shell to complete with this approach.
To get that to happen I had to add the following line to the modules section of the nix/haskell.nix file
To get that to happen I had to add the following line to the modules section of the flake.nix file
-----------------------------
packages.plutus-core.components.library.ghcOptions = [ "-fexternal-interpreter" ];
-----------------------------
There are probaly more that a dozen lines just like this, so it is easy to figure out where to put it.

The final step is to pass the right flags to nix-shell, cabal, and ghc. Here is a summary.
The final step is to pass the right flags to cabal, ghc, and running the right `nix develop` shell. Here is a summary.

1) to start nix, we must use
nix-shell --arg config "{ haskellNix.profiling = true; }"
`nix develop .#profiling`
2) to build with cabal, we must use
cabal build --enable-profiling
`cabal build --enable-profiling`
3) to run the test, we must pass extra flags to ghc, so we must use
cabal test --test-options="+RTS -i60 -p"
`cabal test --test-options="+RTS -i60 -p"`

How to build the system for profiling

Be sure you have set up the files cabal.project.local and nix/haskell.nix as described above.
Exit the nix-shell, if you are running it. Now change directories to the root of the Ledger repository.
Exit the nix shell, if you are running it. Now change directories to the root of the Ledger repository.

Now to start nix type
```
nix develop .#profiling
```
(or, eg. `nix develop .#ghc8107.profiling` for alternative compiler)

nix-shell --arg config "{ haskellNix.profiling = true; }"

When the nix-shell completes (this can take a long time, since it must make sure
When the `nix develop` shell completes (this can take a long time, since it must make sure
every file in the Ledger is compiled with profiling enabled). This might take a
while. Be patient. Take the dogs for a walk.

Now change directories to the root directory of the modlue that contains your
modified Test file, and type

```
cabal build --enable-profiling

```
This might also take a while. Take the dogs for second walk. When this completes
you are ready to start profiing!

How to run a profile.

In the same directory where you did (cabal build --enable-profiling) type

```
cabal test --test-options="+RTS -i60 -p"

```
This should take slightly longer than running the test without profiling.
When it is done, there will be a file in this same directory with extension .prof
When I did it, the file was called cardano-ledger-test.prof . It is a big file
Here are the first few lines.

```
----------------------------------------------------------------------------------------------------------
Fri May 6 14:02 2022 Time and Allocation Profiling Report (Final)
Expand Down Expand Up @@ -177,11 +179,10 @@ genKeyHash.\ Test.Cardano.Ledger.Generic.GenState src/Test/Cardano
serializeEncoding Cardano.Binary.Serialize src/Cardano/Binary/Serialize.hs:(61,1)-(67,49) 0.8 2.1
toLazyByteString Codec.CBOR.Write src/Codec/CBOR/Write.hs:86:1-49 0.7 4.9
---------------------------------------------------------------------------------------------------------------------
```

The problem with my test, was that evalScripts was inadvertantly showing a large data structure
using tellEvent. This was added when debugging and never removed. After fixing this, we had much better results.
I hope you experience is just as rewarding.

Don't forget to revert cabal.project.local, nix/haskell.nix and your Test file to their original state.


18 changes: 9 additions & 9 deletions README.md
Expand Up @@ -106,17 +106,17 @@ When using `nix` the documents and Haskell code can be readily
built by running:

```shell
nix-build
nix build .#specs
```

The LaTeX documents will be placed inside directories named `result*`, e.g.:
The LaTeX documents will be placed inside a directory named `result`, e.g.:

```shell
result-2/byron-ledger.pdf
result-3/shelley-delegation.pdf
result-4/non-integer-calculations.pdf
result-5/small-step-semantics.pdf
result-6/shelley-ledger.pdf
result/byron-ledger.pdf
result/shelley-delegation.pdf
result/non-integer-calculations.pdf
result/small-step-semantics.pdf
result/shelley-ledger.pdf
result/byron-blockchain.pdf
```

Expand All @@ -131,14 +131,14 @@ the Byron release). Then, build the latex document by running:

```shell
cd <myLaTexDir>
nix-shell --pure --run make
nix develop --command make
```

For a continuous compilation of the `LaTeX` file run:

```shell
cd <myLaTexDir>
nix-shell --pure --run "make watch"
nix develop --command make watch
```

# Testing
Expand Down
92 changes: 10 additions & 82 deletions default.nix
@@ -1,82 +1,10 @@
{ system ? builtins.currentSystem
, crossSystem ? null
# allows to cutomize haskellNix (ghc and profiling, see ./nix/haskell.nix)
, config ? {}
# allows to override dependencies of the project without modifications,
# eg. to test build against local checkout of iohk-nix:
# nix build -f default.nix cardano-node --arg sourcesOverride '{
# iohk-nix = ../iohk-nix;
# }'
, sourcesOverride ? {}
# pinned version of nixpkgs augmented with overlays (iohk-nix and our packages).
, pkgs ? import ./nix { inherit system crossSystem config sourcesOverride; }
, gitrev ? pkgs.iohkNix.commitIdFromGitRepoOrZero ./.git
}:
with pkgs; with commonLib;

let
haskellPackages = recRecurseIntoAttrs
# we are only interested in listing the project packages:
(selectProjectPackages cardanoLedgerSpecsHaskellPackages);

self = {
inherit haskellPackages hydraEvalErrors;

# `tests` are the test suites which have been built.
tests = collectComponents' "tests" haskellPackages;
# `benchmarks` (only built, not run).
benchmarks = collectComponents' "benchmarks" haskellPackages;

libs = collectComponents' "library" haskellPackages;

exes = collectComponents' "exes" haskellPackages;

checks = recurseIntoAttrs {
# `checks.tests` collect results of executing the tests:
tests = collectChecks' haskellPackages;
};

shell = import ./shell.nix {
inherit pkgs;
withHoogle = true;
};

roots = cardanoLedgerSpecsHaskellPackages.roots;

#
# PDF builds of LaTeX documentation.
#
# To download the latest PDF build from Hydra, use this link:
# https://github.com/input-output-hk/cardano-ledger/releases/latest/download/NAME.pdf
#
# To get a shell where you can run pdflatex to build it yourself, use:
# nix-shell default.nix -A specs.NAME
#
# To build all specs locally with Nix:
# nix-build -A specs -o spec
#
specs = recurseIntoAttrs {
byron-ledger = pkgs.callPackage ./eras/byron/ledger/formal-spec/default.nix {};
byron-chain = pkgs.callPackage ./eras/byron/chain/formal-spec/default.nix {};
shelley-ledger = pkgs.callPackage ./eras/shelley/formal-spec/default.nix {};
shelley-ma = pkgs.callPackage ./eras/shelley-ma/formal-spec/default.nix {};
alonzo-ledger = pkgs.callPackage ./eras/alonzo/formal-spec/default.nix {};
babbage-ledger = pkgs.callPackage ./eras/babbage/formal-spec/default.nix {};
delegation-design = pkgs.callPackage ./eras/shelley/design-spec/default.nix {};
small-step-semantics = pkgs.callPackage ./docs/small-step-semantics/default.nix {};
pool-ranking = pkgs.callPackage ./docs/pool-ranking/default.nix {};
non-integer-calculations = pkgs.callPackage ./docs/non-integer-calculations/default.nix {};
blocks-cddl = pkgs.callPackage ./eras/byron/cddl-spec/default.nix {};
};

doc = {
site =
let
sphinx-markdown-tables = pkgs.python3Packages.callPackage ./nix/python/sphinx-markdown-tables.nix {};
sphinxemoji = pkgs.python3Packages.callPackage ./nix/python/sphinxemoji.nix {};
in pkgs.callPackage ./doc { inherit sphinx-markdown-tables sphinxemoji; pythonPackages = pkgs.python3Packages; };
};
};

in
self
# a non-flake nix compat wrapper using https://github.com/edolstra/flake-compat
# DO NOT EDIT THIS FILE
__trace
'' ************************************************************************************
Hi there! This project has been moved to nix flakes. You are using the default.nix
compatibility layer. Please consider using flake commands like `nix build .#...`,
`nix repl` with `:lf .` and similar instead.
************************************************************************************
''
(import ./nix/flake-compat.nix).defaultNix

0 comments on commit aec6b78

Please sign in to comment.