Skip to content

Commit

Permalink
Migrate to nix flake
Browse files Browse the repository at this point in the history
  • Loading branch information
jbgi committed Jun 5, 2023
1 parent 6ae2dc3 commit 7b9c1c1
Show file tree
Hide file tree
Showing 74 changed files with 1,820 additions and 1,600 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/bench.yml
Expand Up @@ -22,7 +22,7 @@ jobs:
trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= loony-tools:pr9m4BkM/5/eSTZlkQyRt57Jz7OMBxNSUiMC4FkcNfk=
substituters = https://cache.iog.io https://cache.nixos.org/ https://cache.zw3rk.com
- name: Build benchmarks
run: nix-build -A haskellPackages.cardano-ledger-test.components.benchmarks
run: nix build .#cardano-ledger-test:bench:bench
- name: Run benchmark
run: |
cd libs/cardano-ledger-test
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/gh-pages.yml
Expand Up @@ -20,7 +20,7 @@ jobs:
trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= loony-tools:pr9m4BkM/5/eSTZlkQyRt57Jz7OMBxNSUiMC4FkcNfk=
substituters = https://cache.iog.io https://cache.nixos.org/ https://cache.zw3rk.com
- name: Build projects and haddocks
run: nix-shell --command "cabal update && cabal build --enable-documentation all && ./scripts/haddocks.sh"
run: nix develop --command bash -c 'cabal update && cabal build --enable-documentation all && ./scripts/haddocks.sh'
- name: Add files
run: |
git config --local user.name ${{ github.actor }}
Expand All @@ -37,4 +37,3 @@ jobs:
branch: gh-pages
force: true
directory: .

6 changes: 2 additions & 4 deletions .github/workflows/haskell.yml
Expand Up @@ -198,7 +198,7 @@ jobs:
run: ./scripts/gen-hie.sh

nix-build:
name: Build nix derivations and enter nix-shell
name: Build nix required derivations
if: ${{ github.event_name == 'schedule' }}
runs-on: ubuntu-latest
steps:
Expand All @@ -210,6 +210,4 @@ jobs:
trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= loony-tools:pr9m4BkM/5/eSTZlkQyRt57Jz7OMBxNSUiMC4FkcNfk=
substituters = https://cache.iog.io https://cache.nixos.org/ https://cache.zw3rk.com
- name: nix-build
run: nix-build
- name: nix-shell
run: nix-shell shell.nix
run: nix build .#hydraJobs.required
3 changes: 1 addition & 2 deletions .github/workflows/push-specs.yml
Expand Up @@ -20,7 +20,7 @@ jobs:
trusted-public-keys = hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
substituters = https://cache.iog.io https://cache.nixos.org/
- name: Build specs
run: nix-build -A specs -o output/spec
run: nix build .#specs -o output/spec
- name: Collect dereferenced docs
run: |
mkdir -p deref
Expand All @@ -32,4 +32,3 @@ jobs:
uses: ncipollo/release-action@v1
with:
artifacts: "deref/*"

48 changes: 21 additions & 27 deletions CONTRIBUTING.md
Expand Up @@ -38,10 +38,11 @@ We are transitioning to use GHC 9.2 rather than GHC 8.10.
We need to retain 8.10 compatibility until we are sure that the Cardano node can switch over to 9.2 without any problems.
At that point we can drop it.

The main `nix-shell` will now give you a GHC 9.2 compiler, but you can get a GHC 8.10 shell by calling
The main `nix develop` shell will now give you a GHC 9.2 compiler, but you can get a GHC 8.10 shell by calling
```
nix-shell --arg config '{ haskellNix.compiler = "ghc8107"; }'
nix develop .#ghc8107
```
(this pattern can also be used to test any supported GHC version)

## Updating dependencies

Expand All @@ -58,10 +59,10 @@ That typically just means that we need to fix the breakage (and add a lower-boun
Note that `cabal` itself keeps track of what index states it knows about, so when you bump the pinned index state you may need call `cabal update` in order for `cabal` to be happy.

The Nix code which builds our packages also cares about the index state.
This is represented by inputs managed by `niv`:
This is represented by inputs managed by `nix flake`:
You can update these by running:
- `niv update hackage.nix` for Hackage
- `niv update cardano-haskell-packages` for CHaP
- `nix flake lock --update-input haskellNix/hackage` for Hackage
- `nix flake lock --update-input CHaP` for CHaP (Cardano Haskell Packages)

If you fail to do this you may get an error like this from Nix:
```
Expand Down Expand Up @@ -189,15 +190,15 @@ NIGHTLY=true cabal test cardano-ledger-shelley-test

We have support for running
[ghcid](https://github.com/ndmitchell/ghcid)
from inside of `nix-shell`.
Enter `nix-shell` from the base directory of the repository,
from inside of `nix develop`.
Enter `nix develop` from the base directory of the repository,
change directories to the cabal package that you wish to check,
then run `ghcid`.

For example:

```shell
nix-shell
nix develop
cd eras/shelley/impl/
ghcid
```
Expand All @@ -214,12 +215,12 @@ Error: cabal: Cannot open a repl for multiple components at once. The target ''
Specifying the component solves this problem:

```shell
nix-shell
nix develop
cd libs/cardano-ledger-binary/
ghcid testlib # or `ghcid cardano-ledger-binary`
```

## nix-build Infrastructure
## nix build Infrastructure

The artifacts in this repository can be built and tested using nix. This is
additionally used by the Hydra CI to test building, including cross-compilation
Expand All @@ -231,11 +232,11 @@ To add a new Haskell package, you should do the following:

1. Create the project in the usual way. It should have an appropriate `.cabal` file.
2. Test that you can build your new project by running the following: `nix build
-f default.nix libs.<project_name>`. If you have executables, then
you may also try building these using the `exes.<executable_name>`
attribute path. A good way to see what's available is to execute `:l
default.nix` in `nix repl`. This will allow you to explore the potential
attribute names by using tab completion on "libs.".
.#<project_name>:lib:<lib_name>`. If you have executables, then
you may also try building these using the `.#<project_name>:exe:<exe_name>`
attribute path. A good way to see what's available is to execute `:lf .`
in `nix repl`. This will allow you to explore the potential
attribute names by using tab completion on "packages.<your_system>".

### To add a new LaTeX specification

Expand All @@ -246,16 +247,11 @@ from [the Shelley ledger spec](./eras/shelley/formal-spec)).
1. Copy these files into the root of your new LaTeX specification.
2. Modify the `DOCNAME` in the `Makefile`.
3. Update `default.nix` to:
1. Make sure that the relative path in the first line is pointing to
(default.nix)[./default.nix]. This is used to pin the
`nixpkgs` version used to build the LaTeX specifications.
2. Update the `buildInputs` to add in any LaTeX packages you need in your
1. Update the `buildInputs` to add in any LaTeX packages you need in your
document, and remove any unneeded ones.
3. Alter the `meta` description field to reflect the nature of this document.
4. Add a link to the package at the bottom of [default.nix](./default.nix),
2. Alter the `meta` description field to reflect the nature of this document.
4. Add a link to the package near the bottom of [flake.nix](./flake.nix),
following the existing examples.
5. To require that your specification be built in CI, add it at the end of the
list in [default.nix](./default.nix) following the existing examples.

### Additional documentation

Expand Down Expand Up @@ -357,11 +353,9 @@ cabal configure --enable-profiling --profiling-detail=all-functions
```

Now we need to run a node to build up the dataabase.
This can be done in the cardano-node repository by
opening a nix-shell and running:
This can be done in the cardano-node repository by running:
```
nix build -f default.nix scripts.mainnet.node
./result/bin/cardano-node-mainnet
nix run .#mainnet/node
```
This will take a very long time.
You can stop the node once it is past any slots that you care about.
Expand Down
37 changes: 19 additions & 18 deletions HowToProfileLedger.md
Expand Up @@ -23,8 +23,8 @@ greater detail about each step

1. Decide what you want to profile, and arrange the code so this is possible
2. Choreograph the dance between nix, cabal, and ghc. This has two parts
a. Adding stuff to files like cabal.project, cabal.project.local, and nix/haskel.nix
b. Passing the right flags to nix-shell, cabal, and ghc
a. Adding stuff to files like cabal.project, cabal.project.local
b. Passing the right flags to cabal, and ghc, running the right `nix develop` shell,
3. Recompiling everything so it can be profiled. This takes a very long time (greater than
30 minutes when I did it)
4. Start up the profiling. I used "cabal test" with just the right command line arguments.
Expand Down Expand Up @@ -103,53 +103,55 @@ package plutus-core
---------------------

Supposedly this makes things work. But I could never get the nix-shell to complete with this approach.
To get that to happen I had to add the following line to the modules section of the nix/haskell.nix file
To get that to happen I had to add the following line to the modules section of the flake.nix file
-----------------------------
packages.plutus-core.components.library.ghcOptions = [ "-fexternal-interpreter" ];
-----------------------------
There are probaly more that a dozen lines just like this, so it is easy to figure out where to put it.

The final step is to pass the right flags to nix-shell, cabal, and ghc. Here is a summary.
The final step is to pass the right flags to cabal, ghc, and running the right `nix develop` shell. Here is a summary.

1) to start nix, we must use
nix-shell --arg config "{ haskellNix.profiling = true; }"
`nix develop .#profiling`
2) to build with cabal, we must use
cabal build --enable-profiling
`cabal build --enable-profiling`
3) to run the test, we must pass extra flags to ghc, so we must use
cabal test --test-options="+RTS -i60 -p"
`cabal test --test-options="+RTS -i60 -p"`

How to build the system for profiling

Be sure you have set up the files cabal.project.local and nix/haskell.nix as described above.
Exit the nix-shell, if you are running it. Now change directories to the root of the Ledger repository.
Exit the nix shell, if you are running it. Now change directories to the root of the Ledger repository.

Now to start nix type
```
nix develop .#profiling
```
(or, eg. `nix develop .#ghc8107.profiling` for alternative compiler)

nix-shell --arg config "{ haskellNix.profiling = true; }"

When the nix-shell completes (this can take a long time, since it must make sure
When the `nix develop` shell completes (this can take a long time, since it must make sure
every file in the Ledger is compiled with profiling enabled). This might take a
while. Be patient. Take the dogs for a walk.

Now change directories to the root directory of the modlue that contains your
modified Test file, and type

```
cabal build --enable-profiling

```
This might also take a while. Take the dogs for second walk. When this completes
you are ready to start profiing!

How to run a profile.

In the same directory where you did (cabal build --enable-profiling) type

```
cabal test --test-options="+RTS -i60 -p"

```
This should take slightly longer than running the test without profiling.
When it is done, there will be a file in this same directory with extension .prof
When I did it, the file was called cardano-ledger-test.prof . It is a big file
Here are the first few lines.

```
----------------------------------------------------------------------------------------------------------
Fri May 6 14:02 2022 Time and Allocation Profiling Report (Final)
Expand Down Expand Up @@ -177,11 +179,10 @@ genKeyHash.\ Test.Cardano.Ledger.Generic.GenState src/Test/Cardano
serializeEncoding Cardano.Binary.Serialize src/Cardano/Binary/Serialize.hs:(61,1)-(67,49) 0.8 2.1
toLazyByteString Codec.CBOR.Write src/Codec/CBOR/Write.hs:86:1-49 0.7 4.9
---------------------------------------------------------------------------------------------------------------------
```

The problem with my test, was that evalScripts was inadvertantly showing a large data structure
using tellEvent. This was added when debugging and never removed. After fixing this, we had much better results.
I hope you experience is just as rewarding.

Don't forget to revert cabal.project.local, nix/haskell.nix and your Test file to their original state.


18 changes: 9 additions & 9 deletions README.md
Expand Up @@ -106,17 +106,17 @@ When using `nix` the documents and Haskell code can be readily
built by running:

```shell
nix-build
nix build .#specs
```

The LaTeX documents will be placed inside directories named `result*`, e.g.:
The LaTeX documents will be placed inside a directory named `result`, e.g.:

```shell
result-2/byron-ledger.pdf
result-3/shelley-delegation.pdf
result-4/non-integer-calculations.pdf
result-5/small-step-semantics.pdf
result-6/shelley-ledger.pdf
result/byron-ledger.pdf
result/shelley-delegation.pdf
result/non-integer-calculations.pdf
result/small-step-semantics.pdf
result/shelley-ledger.pdf
result/byron-blockchain.pdf
```

Expand All @@ -131,14 +131,14 @@ the Byron release). Then, build the latex document by running:

```shell
cd <myLaTexDir>
nix-shell --pure --run make
nix develop --command make
```

For a continuous compilation of the `LaTeX` file run:

```shell
cd <myLaTexDir>
nix-shell --pure --run "make watch"
nix develop --command make watch
```

# Testing
Expand Down
2 changes: 1 addition & 1 deletion cabal.project
Expand Up @@ -12,7 +12,7 @@ repository cardano-haskell-packages

-- See CONTRIBUTING for some Nix commands you will need to run if you
-- update either of these.
index-state: 2023-05-25T00:00:00Z
index-state: 2023-06-25T00:00:00Z
index-state:
-- Bump this if you need newer packages from Hackage
, hackage.haskell.org 2023-06-01T19:11:19Z
Expand Down

0 comments on commit 7b9c1c1

Please sign in to comment.