IntersectMBO · jorisdral · Jul 6, 2022 · Jun 23, 2022 · Jul 4, 2022
diff --git a/README.rst b/README.rst
@@ -98,6 +98,8 @@ The general synopsis is as follows:
                            [--host-ipv6-addr IPV6-ADDRESS]
                            [--port PORT]
                            [--config NODE-CONFIGURATION] [--validate-db]
+                           [ --in-memory-ledger-db-backend | --lmdb-ledger-db-backend [--lmdb-mapsize BIN]]
+
      Run the node.
 
 * ``--topology`` - Filepath to a topology file describing which peers the node should connect to.
@@ -128,6 +130,24 @@ The general synopsis is as follows:
 
 * ``--validate-db`` - Flag to revalidate all on-disk database files
 
+* ``--in-memory-ledger-db-backend`` - Use the in-memory backend to store the
+  UTxO portion of the ledger state in memory. As a result, the whole ledger
+  state will be stored in memory. Incompatible with
+  ``--lmdb-ledger-db-backend``.
+* ``--lmdb-ledger-db-backend`` - Use the LMDB backend to store the UTxO portion
+  of the ledger state on disk. The remainder of the ledger state will still be
+  stored in memory. Incompatible with ``--in-memory-ledger-db-backend``. The
+  node uses the LMDB backend by default if no ``--*-db-backend`` flags are set.
+
+  * ``--lmdb-mapsize`` - Set the maximum database size (mapsize) for the LMDB
+    backend. By default, the mapsize of the backend is set to 16 Gigabytes.
+    Warning: if the database size exceeds the given mapsize, the node will
+    abort. Therefore, the mapsize should be set to a value high enough to
+    guarantee that the maximum database size will not be reached during the
+    expected node uptime. The current default value is sufficient to offer this
+    guarantee, and setting the mapsize to anything less is therefore not
+    recommended.
+
 Configuration ``.yaml`` files
 =============================
 

diff --git a/cardano-node/src/Cardano/Node/Configuration/LedgerDB.hs b/cardano-node/src/Cardano/Node/Configuration/LedgerDB.hs
@@ -28,6 +28,7 @@ data BackingStoreSelectorFlag =
 
 -- | Recommended settings for the LMDB backing store.
 --
+-- === @'lmdbMapSize'@
 -- The default @'LMDBLimits'@ uses an @'lmdbMapSize'@ of @16_000_000_000@
 -- bytes, or 16 GigaBytes. @'lmdbMapSize'@ sets the size of the memory map
 -- that is used internally by the LMDB backing store, and is also the
@@ -42,6 +43,29 @@ data BackingStoreSelectorFlag =
 -- @'lmdbMapSize'@. If this fatal error were to occur, we would expect that
 -- the node can continue normal operation if it is restarted with a higher
 -- @'lmdbMapSize'@ configured. Nonetheless, this situation should be avoided.
+--
+-- === @'lmdbMaxDatabases'@
+-- The @'lmdbMaxDatabases'@ is set to 10, which means that the LMDB backing
+-- store will allow up @<= 10@ internal databases. We say /internal/
+-- databases, since they are not exposed outside the backing store interface,
+-- such that from the outside view there is just one /logical/ database.
+-- Two of these internal databases are reserved for normal operation of the
+-- backing store, while the remaining databases will be used to store ledger
+-- tables. At the moment, there is at most one ledger table that will be
+-- stored in an internal database: the UTxO. Nonetheless, we set
+-- @'lmdbMaxDatabases'@ to @10@ in order to future-proof these limits.
+--
+-- === @'lmdbMaxReaders'@
+-- The @'lmdbMaxReaders'@ limit sets the maximum number of threads that can
+-- read from the LMDB database. Currently, there should only be a single reader
+-- active. Again, we set @'lmdbMaxReaders'@ to @16@ in order to future-proof
+-- these limits.
+--
+-- === References
+-- For more information about LMDB limits, one should inspect:
+-- * The @lmdb-simple@ and @haskell-lmdb@ forked repositories.
+-- * The official LMDB API documentation at
+--    <http://www.lmdb.tech/doc/group__mdb.html>.
 defaultLMDBLimits :: LMDBLimits
 defaultLMDBLimits = LMDBLimits {
     lmdbMapSize = 16_000_000_000

diff --git a/cardano-node/src/Cardano/Node/Configuration/POM.hs b/cardano-node/src/Cardano/Node/Configuration/POM.hs
@@ -331,7 +331,7 @@ instance FromJSON PartialNodeConfiguration where
              maybeMapSize :: Maybe String <- v .:? "LMDBMapSize"
              mapSize <- case maybeMapSize of
                Nothing -> return Nothing
-               Just s  -> case parseValue ParseBinary s of
+               Just s  -> case parseValue ParseExact s of
                  Left e      -> fail ("Malformed LMDBMapSize: " <> e)
                  Right units -> return $ Just units
              return . Just . LMDB $ mapSize

diff --git a/cardano-node/src/Cardano/Node/Parsers.hs b/cardano-node/src/Cardano/Node/Parsers.hs
@@ -240,7 +240,7 @@ parseLedgerDBBackend = parseInMemory <|> parseLMDB <*> optional parseMapSize
 
     parseMapSize :: Parser Int
     parseMapSize =
-      option (eitherReader (parseValue ParseBinary)) (
+      option (eitherReader (parseValue ParseExact)) (
            long "lmdb-mapsize"
         <> metavar "BIN"
         <> help "The maximum database size defined as a binary \

diff --git a/doc/getting-started/building-the-node-on-windows.md b/doc/getting-started/building-the-node-on-windows.md
@@ -1,104 +1,201 @@
 # Building the development version of the Cardano node on Windows
 
-This document explains how to build a __DEVELOPMENT__ version of the `cardano-node`.
-Note that this is *not* for building a __PRODUCTION__ version of the node. 
+This document explains how to build a __DEVELOPMENT__ version of the `cardano-node`. Note that this is *not* for building a __PRODUCTION__ version of the node.
 
-To start building on `Windows`, you will need to install and configure specific tools outlined below. We recommend using [chocolatey](https://chocolatey.org), which provides the `choco` command to install these tools.  
+See [install.md](./install.md) for a more thorough explanation of the dependencies and packages that have to be installed to run a node. Although the installation instructions in that document are aimed at building on Linux, large parts of the text are interesting to Windows users as well.
 
-You can run all the instructions that invoke `choco` in PowerShell with root privileges. We recommend installing and using `git-bash` for development purposes, which is `git` when installed with `choco install git`.
+## Install the Haskell environment
 
-## Install GHC
+The recommended way to install the Haskell tools is via [GHCup](https://www.haskell.org/ghcup/). Check [this page](https://www.haskell.org/ghcup/install/) for further explanation on the installation process.
 
-The recommended way is to run the following command in PowerShell:
+Once GHCup is installed, open a new terminal (to get an updated environment) and run:
 
-```PowerShell
-choco install --version 8.10.7 ghc
+```bash
+ghcup install ghc 8.10.7
+ghcup install cabal 3.6.2.0
+ghcup set ghc 8.10.7
+ghcup set cabal 3.6.2.0
 ```
 
-## Install pkg-config
+## Install MSYS2 and basic dependencies
 
-```PowerShell
-choco install pkgconfiglite
+### GHCUP and MSYS2
+
+Note that the GHCUP comes with its own installation for MSYS2, and it can be used to build `cardano-node`. You can start the MSYS2 shell with
+```cmd
+<ghcup-path>\msys64\msys2_shell.cmd -defterm -here -no-start -mingw64
+```
+
+Make sure to enable `mingw64` subsystem. It is also possible to enable `mingw64` subsystem by
+
+```bash
+source shell mingw64
+```
+
+
+The MSYS2 shell is bundled with `pacman` package manager. `GHC` is bundled with `gcc` for `mingw`. Configure your environment to use this tool-chain.
+
+```bash
+pacman -Syu
+pacman -S --needed base-devel automake git
+export PATH=$PATH:/mingw64/bin:<ghcup-path>/bin:<ghcup-path>/ghc/<ghc-version>/mingw/bin
+```
+
+Alternatively, you can update the `mingw64` tool-chain, that adds number of packages including latest `gcc`. This may give rise to `crt` linker error. Please read the [section - global cabal configuration](#global-cabal-configuration) to mitigate the error.
+
+```bash
+pacman -S mingw-w64-x86_64-toolchain
 ```
 
-## Install vcpkg
+### Installing MSYS independently
 
-You will first need to install `vcpkg` to proceed with `libsodium` library installation (which is the next step). 
+Go to [the MSYS2 website](https://www.msys2.org/) and follow its instructions to install MSYS2. It will also install `pacman`. Then, update and upgrade all `pacman` packages and install basic dependencies:
+
+```
+pacman -Syu
+pacman -S --needed base-devel mingw-w64-x86_64-toolchain git
+```
 
-For this, use `git`(which you can also install using `choco`) and follow [these
-instructions](https://github.com/microsoft/vcpkg#quick-start-windows).
+## Installing and configuring third party libraries
+You can use `vcpkg`, or `pacman` or both to install third party libraries. Following are the dependencies for `cardano-node`.
+
+| package   | `vcpkg`   | `pacman`                   |  Remarks
+|-----------|-----------|----------------------------|--------------------------------------
+| libsodium | libsodium | mingw-w64-x86_64-lmdb      |
+| lmdb      | lmdb      | mingw-w64-x86_64-libsodium |
+| secp256k1 | secp256k1 | __not available__          | `vcpkg` package is not complete.
+| openssl   | openssl   | mingw-w64-x86_64-openssl   |
+|-----------|-----------|----------------------------|--------------------------------------
+
+### Using __vcpkg__
+- Install `vcpkg` for MSYS2 with instruction given [here](https://vcpkg.io/en/docs/users/mingw.html)
+- Set following variables for installing libraries
+  ```bash
+  export VCPKG_DEFAULT_TRIPLET=x64-mingw-static
+  export VCPKG_DEFAULT_HOST_TRIPLET=x64-mingw-dynamic
+  ```
+- You can also use `x64-mingw-dynamic` for `VCPKG_DEFAULT_TRIPLET`, if you'd like to use shared libraries
+- Install libraries
+  ```bash
+  vcpkg install libsodium
+  vcpkg install openssl
+  vcpkg install lmdb
+  ```
+- __cardano-crypto-class__ uses several features of `secp256k1` that are not compiled into `vcpkg` package. Hence it needs to be compiled separately.
+- `vcpkg` generates `pkg-config` when it can. It generates configuration for `openssl`. To avoid mixing configurations with existing system, you can use following to install packages
+  ```bash
+  PATH="${PATH/:\/usr\/bin:\/bin:/:}:/ming64/bin" ./vcpkg.exe install <pkg-name>
+  ```
+  + Note that packages will be generated at `<vc-pkg>/installed/${VCPKG_DEFAULT_TRIPLET}/lib/pkgconfig`.
+
+### Using __MSYS2 pacman__
+
+You can search for packages [here](https://packages.msys2.org/)
+
+- Install dependencies as below
+  ```bash
+  pacman -S mingw-w64-x86_64-libsodium
+  pacman -S mingw-w64-x86_64-lmdb
+  pacman -S mingw-w64-x86_64-openssl
+  ```
+
+### Install Secp256k1
+
+You can use `secp256k1` either by downloading from hydra. Or by downloading from [github](https://github.com/bitcoin-core/secp256k1)
 
-You can now install `libsodium` with the following command:
-```bash
-./vcpkg install --triplet x64-windows libsodium
+```
+pacman -S unzip
+curl https://hydra.iohk.io/job/Cardano/haskell-nix/windows-secp256k1/latest/download/1 -o secp256k1.zip
+mkdir secp256k1
+cd secp256k1/
+unzip ../secp256k1.zip
+cp -r * /mingw64
 ```
 
-## Create libsodium.pc file
+If `unzip` complains that `../secp256k1.zip` is not a zip file, unzip the
+file manually in the explorer by choosing the option to unzip from the
+right-click menu.
 
-To find system dependencies like `libsodium`, `cabal` uses `pkg-config`. On Windows, you will need to create the `libsodium.pc` description file in a correct
-directory.
+### Managing package configuration
+Make sure that package configurations are correectly installed. If you are using chocolatey or other tool, it may be possible that more than one `pkg-config` tools are installed. You are encouraged to use `pkg-config` that __MSYS2__ has to minimize the error.  Also make sure that the pkg configuration are discoverable by checking that `pkg-config --list-all` lists all the installed packages.
+
+Note that the pkg-config will only look in the paths pointed to by `pkg-config --variable pc_path pkg-config`. Also check if pkg-config correctly displays library path, by checking `pkg-config --variable libdir <pkg-name>`. In case, the path is incorrectly displayed, modify the file `pkg-config --path <pkg-name>` to correct the path.
 
-In one of the paths reported by:
-```bash
-pkg-config --variable pc_path pkg-config
-```
-create `libsodium.pc` file, which contains:
 
-```
-libdir=VCPKG_PATH/installed/x64-windows/bin
-includedir=VCPKG_PATH/installed/x64-windows/include
 
-Name: libsodium
-VERSION: LIBSODIUM_VERSION
-Description: libsodium library
-Cflags: -I${includedir}/sodium
-Libs: -L${libdir} -llibsodium
+## Global `cabal` configuration
+
+Modify the following entries to your global `cabal` config file in your
+`\path\to\cabal\` directory, such that they look like this:
+
 ```
-> Note that you need to replace `VCPKG_PATH` with the
-absolute path, where you use `vcpkg`, and `LIBSODIUM_VERSION` with the version
-number of `libsodium` which was installed on your system. Please verify that
-the paths above contain
-`libsodium.dll` file and headers.
+-- extra-include-dirs:
+extra-lib-dirs:  C:\ghcup\ghc\8.10.7\mingw\x86_64-w64-mingw32\lib
+extra-prog-path: C:\ghcup\bin,
+                 path\to\cabal\bin
+```
+
+Depending on the installation procedure, the path to your `\path\to\cabal\` directory could be `C:\Users\<username>\AppData\Roaming\cabal`).
 
-> Also, you cannot use `prefix=` in the `libsodium.pc` file. This might be changed for
-some other directory, `pkg-config` provides a switch to use the provided
-`prefix`, but there is no way to instruct `cabal` to do so.
+> Note: The instructions in this section are a workaround for a
+> [GHCup bug](https://github.com/msys2/MINGW-packages/issues/10837#issuecomment-1047105402).
 
-## `cabal` configuration
+## Downloading the source code for cardano-node
 
-Go to the directory, where you cloned the `cardano-node` repository and add the command below
-to your `cabal.project.local` file (if you don't already have it, create one):
+Create a working directory for your builds:
 
+```bash
+mkdir -p ~/src
+cd ~/src
 ```
-max-backjump: 5000
-reorder-goals: True
 
-package cardano-crypt-praos
-  flags: -external-libsodium-vrf
+Download the Cardano node sources:
 
-extra-lib-dirs: "VCPKG_PATH\\installed\\x64-windows\\bin"
-extra-include-dirs: "VCPKG_PATH\\installed\\x64-windows\\include"
+```bash
+git clone https://github.com/input-output-hk/cardano-node.git
 ```
 
-The final step is to add `VCPKG_PATH/installed/x64-windows/bin` to the `PATH`
-variable. Since we are using `git-bash`, you can do this with: 
+Change the working directory to the downloaded source code folder:
+
+```bash
+cd cardano-node
+```
+
+## Local `cabal` configuration
+
+Add the following to the `cabal.project.local` file (if you don't already have it, create one) in the source code folder:
 
 ```
-export PATH="VCPKG_PATH/installed/x64-windows/bin:${PATH}"
+package lmdb
+  extra-lib-dirs:     "<lmdb-library-path> Or standard path E.g. C:\\msys64\\mingw64\\include or C:/tools/ghcup/msys64/mingw64/lib"
+  extra-include-dirs: "<lmdb-library-path> Or standard path E.g. C:\\msys64\\mingw64\\include or C:/tools/ghcup/msys64/mingw64/include"
+
+-- If pkg-config is available, use it.
+package HsOpenSSL
+  flags: +use-pkg-config
+
+package cardano-crypt-praos
+  flags: -external-libsodium-vrf
 ```
-*Remember to substitute `VCPKG_PATH` with the real path.*
+
+## Building and installing the node
 
 You can now build the node with:
 
 ```bash
+cabal update
 cabal build --builddir /c/dist exe:cardano-node
 ```
-> Note: using `--builddir /c/dist` with a succinct directory protects you
-from exceeding the [maximal path
-size](https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation)
-on Windows (which is a limitation of the linker rather than `ghc` itself).
 
-You can now verify whether the node runs: 
+> Note: using `--builddir /c/dist` with a succinct directory protects you from exceeding the [maximal path size](https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation) on Windows (which is a limitation of the linker rather than `ghc` itself).
+
+You can now verify whether the node runs:
 ```bash
 cabal run --builddir /c/dist exe:cardano-node -- run --help
 ```
+
+## Notes
+
+### Speed up compilation
+
+`reorder-goals` can affect the compilation time negatively. You can disable `reorder-goals` by setting it to `False` in `cabal.project.local`.
diff --git a/doc/getting-started/install.md b/doc/getting-started/install.md
@@ -27,6 +27,8 @@ To download the source code and build it, you need the following packages and to
 * developer libraries for `systemd`,
 * developer libraries for `ncurses`,
 * `ncurses` compatibility libraries,
+* developer libraries for `openssl`,
+* developer libraries for `lmdb`,
 * the Haskell build tool `cabal`,
 * the GHC Haskell compiler (version `8.10.7` or above).
 
@@ -35,14 +37,14 @@ In Redhat, Fedora, and Centos:
 ```bash
 sudo yum update -y
 sudo yum install git gcc gcc-c++ tmux gmp-devel make tar xz wget zlib-devel libtool autoconf -y
-sudo yum install systemd-devel ncurses-devel ncurses-compat-libs -y
+sudo yum install systemd-devel ncurses-devel ncurses-compat-libs which jq openssl-devel lmdb-devel -y
 ```
 
 For Debian/Ubuntu, use the following instead:
 
 ```bash
 sudo apt-get update -y
-sudo apt-get install automake build-essential pkg-config libffi-dev libgmp-dev libssl-dev libtinfo-dev libsystemd-dev zlib1g-dev make g++ tmux git jq wget libncursesw5 libtool autoconf -y
+sudo apt-get install automake build-essential pkg-config libffi-dev libgmp-dev libssl-dev libtinfo-dev libsystemd-dev zlib1g-dev make g++ tmux git jq wget libncursesw5 libtool autoconf liblmdb-dev -y
 ```
 
 If you are using a different flavor of Linux, you will need to use the correct package manager for your platform instead of `yum` or `apt-get`, and the names of the packages you need to install might differ.  On MacOSX, use the Homebrew (`brew`) installer.