Merge branch 'develop' into jy14898/evaluatetx-failure-parsing

.github/pull_request_template.md

@@ -2,8 +2,6 @@ Closes # .
 
 ### Pre-review checklist
 
-- [ ] All code has been formatted using our config (`make format` for Purescript, `nixpkgs-fmt` for Nix)
-- [ ] All Purescript imports are explicit, including constructors
-- [ ] **All** existing examples have been run locally against a fully-synced testnet node
+- [ ] All code has been formatted using our config (`make format`)
 - [ ] The integration and unit tests have been run (`npm run test`) locally
-- [ ] The changelog has been updated under the `## Unreleased` header, using the appropriate sub-headings (`### Added`, `### Removed`, `### Fixed`)
+- [ ] The changelog has been updated under the `## Unreleased` header, using the appropriate sub-headings (`### Added`, `### Removed`, `### Fixed`), and the links to the appropriate issues/PRs have been included

.github/workflows/pages.yml

@@ -0,0 +1,25 @@
+name: Build and Deploy docs to Github Pages
+on:
+  push:
+    branches:
+      - develop
+permissions:
+  contents: write
+jobs:
+  build-and-deploy:
+    concurrency: ci-${{ github.ref }} # Recommended if you intend to make multiple deployments in quick succession.
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v3
+
+      - name: Install and Build
+        run: |
+          npm install -g spago
+          npm install -g purescript@0.14.5
+          spago docs
+      - name: Deploy
+        uses: JamesIves/github-pages-deploy-action@v4.3.3
+        with:
+          branch: gh-pages
+          folder: generated-docs/html

.gitignore

@@ -22,3 +22,5 @@ result-*
 output.js
 .DS_Store
 .idea/
+test-data/chrome-user-data/
+tmp

CHANGELOG.md

@@ -25,33 +25,80 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 
 ### Added
 
+- Plutip integration to run `Contract`s in local, private testnets ([#470](https://github.com/Plutonomicon/cardano-transaction-lib/pull/470))
+- Ability to run `Contract`s in Plutip environment in parallel - `Contract.Test.Plutip.withPlutipContractEnv` ([#800](https://github.com/Plutonomicon/cardano-transaction-lib/issues/800))
+- `withKeyWallet` utility that allows to simulate multiple actors in Plutip environment ([#663](https://github.com/Plutonomicon/cardano-transaction-lib/issues/663))
 - `Alt` and `Plus` instances for `Contract`.
 - `Contract.Utxos.getUtxo` call to get a single utxo at a given output reference
-- `purescriptProject`'s `shell` parameter now accepts `packageLockOnly`, which if set to true will stop npm from generating node\_modules. This is enabled for CTL developers
+- `Contract.Monad.withContractEnv` function  that constructs and finalizes a contract environment that is usable inside a bracket callback. **This is the intended way to run multiple contracts**. ([#731](https://github.com/Plutonomicon/cardano-transaction-lib/pull/731))
+- `Contract.Monad.stopContractEnv` function to finalize a contract environment (close the `WebSockets`). It should be used together with `mkContractEnv`, and is not needed with `withContractEnv`. ([#731](https://github.com/Plutonomicon/cardano-transaction-lib/pull/731))
+- `Contract.Config` module that contains everything needed to create and manipulate `ConfigParams`, as well as a number of `ConfigParams` fixtures for common use cases. ([#731](https://github.com/Plutonomicon/cardano-transaction-lib/pull/731))
+- `Contract.Monad.askConfig` and `Contract.Monad.asksConfig` functions to access user-defined configurations. ([#731](https://github.com/Plutonomicon/cardano-transaction-lib/pull/731))
+- `Contract.Config.WalletSpec` type that allows to define wallet parameters declaratively in `ConfigParams`, instead of initializing wallet and setting it to a `ContractConfig` ([#731](https://github.com/Plutonomicon/cardano-transaction-lib/pull/731))
+- Faster initialization of `Contract` runtime due to parallelism. ([#731](https://github.com/Plutonomicon/cardano-transaction-lib/pull/731))
+- `purescriptProject`'s `shell` parameter now accepts `packageLockOnly`, which if set to true will stop npm from generating `node_modules`. This is enabled for CTL developers
 - `Contract.Transaction.awaitTxConfirmed` and `Contract.Transaction.awaitTxConfirmedWithTimeout`
 - `Contract.TextEnvelope.textEnvelopeBytes` and family to decode the `TextEnvelope` format, a common format output by tools like `cardano-cli` to serialize values such as cryptographical keys and on-chain scripts
 - `Contract.Wallet.isNamiAvailable` and `Contract.Wallet.isGeroAvailable` functions ([#558](https://github.com/Plutonomicon/cardano-transaction-lib/issues/558)])
+- `Contract.Transaction.balanceTxWithOwnAddress` and `Contract.Transaction.balanceTxsWithOwnAddress` to override an `Address` used in `balanceTx` internally ([#775](https://github.com/Plutonomicon/cardano-transaction-lib/pull/775))
+- `Contract.Transaction.awaitTxConfirmedWithTimeoutSlots` waits a specified number of slots for a transaction to succeed. ([#790](https://github.com/Plutonomicon/cardano-transaction-lib/pull/790))
+- `Contract.Transaction.submitE` like submit but uses an `Either (Array Aeson) TransactionHash` to handle a SubmitFail response from ogmios
+- `Contract.Chain.waitNSlots`,  `Contract.Chain.currentSlot` and `Contract.Chain.currentTime` a function to wait at least `N` number of slots and functions to get the current time in `Slot` or `POSIXTime`. ([#740](https://github.com/Plutonomicon/cardano-transaction-lib/issues/740))
+- `Contract.Transaction.getTxByHash` to retrieve contents of an on-chain transaction.
+- `project.launchSearchablePursDocs` to create an `apps` output for serving Pursuit documentation locally ([#816](https://github.com/Plutonomicon/cardano-transaction-lib/issues/816))
+- `Contract.PlutusData.IsData` type class (`ToData` + `FromData`) ([#809](https://github.com/Plutonomicon/cardano-transaction-lib/pull/809))
+- A check for port availability before Plutip runtime initialization attempt ([#837](https://github.com/Plutonomicon/cardano-transaction-lib/issues/837))
+- `Contract.Address.addressToBech32` and `Contract.Address.addressWithNetworkTagToBech32` ([#846](https://github.com/Plutonomicon/cardano-transaction-lib/issues/846))
+- `doc/e2e-testing.md` describes the process of E2E testing. ([#814](https://github.com/Plutonomicon/cardano-transaction-lib/pull/814))
+- Added unzip to the `devShell`. New `purescriptProject.shell` flag `withChromium` also optionally adds Chromium to the `devShell` ([#799](https://github.com/Plutonomicon/cardano-transaction-lib/pull/799))
 
 ### Changed
 
+- `runContract` now accepts `ConfigParams` instead of `ContractConfig` ([#731](https://github.com/Plutonomicon/cardano-transaction-lib/pull/731))
+- `mkContractConfig` has been renamed to `mkContractEnv`. Users are advised to use `withContractEnv` instead to ensure proper finalization of WebSocket connections. ([#731](https://github.com/Plutonomicon/cardano-transaction-lib/pull/731))
 - CTL's `overlay` no longer requires an explicitly passed `system`
 - Switched to CSL for utxo min ada value calculation ([#715](https://github.com/Plutonomicon/cardano-transaction-lib/pull/715))
-- ServerConfig accepts a url `path` field
+- `ConfigParams` is now a type synonym instead of a newtype. `ContractConfig` has been renamed to `ContractEnv`.
+- Moved logging functions to `Contract.Log` from `Contract.Monad` ([#727](https://github.com/Plutonomicon/cardano-transaction-lib/issues/727)
+- Renamed `Contract.Wallet.mkKeyWalletFromPrivateKey` to `Contract.Wallet.mkKeyWalletFromPrivateKeys`.
+- ServerConfig accepts a url `path` field ([#728](https://github.com/Plutonomicon/cardano-transaction-lib/issues/728)).
+- Examples now wait for transactions to be confirmed and log success ([#739](https://github.com/Plutonomicon/cardano-transaction-lib/issues/739)).
+- Updated CSL version to v11.0.0 ([#801](https://github.com/Plutonomicon/cardano-transaction-lib/issues/801))
+- Better error message when attempting to initialize a wallet in NodeJS environment ([#778](https://github.com/Plutonomicon/cardano-transaction-lib/issues/778))
+- The [`ctl-scaffold`](https://github.com/mlabs-haskell/ctl-scaffold) repository has been archived and deprecated and its contents moved to `templates.ctl-scaffold` in the CTL flake ([#760](https://github.com/Plutonomicon/cardano-transaction-lib/issues/760)).
+- The CTL `overlay` output has been deprecated and replaced by `overlays.purescript` and `overlays.runtime` ([#796](https://github.com/Plutonomicon/cardano-transaction-lib/issues/796)).
+- `buildCtlRuntime` and `launchCtlRuntime` now take an `extraServices` argument to add `docker-compose` services to the resulting Arion expression ([#769](https://github.com/Plutonomicon/cardano-transaction-lib/issues/769)).
 
-### Fixed 
+### Removed
+
+- `Contract.Monad.traceTestnetContractConfig` - use `Contract.Config.testnetNamiConfig` instead.
+- `runContract_` - use `void <<< runContract`.
+
+### Fixed
 
+- Endless `awaitTxConfirmed` calls ([#804](https://github.com/Plutonomicon/cardano-transaction-lib/issues/804))
 - Bug with collateral selection: only the first UTxO provided by wallet was included as collateral [(#723)](https://github.com/Plutonomicon/cardano-transaction-lib/issues/723)
 - Bug with collateral selection for `KeyWallet` when signing multiple transactions ([#709](https://github.com/Plutonomicon/cardano-transaction-lib/pull/709))
+- Properly implemented CIP-25 V2 metadata. Now there's no need to split arbitrary-length strings manually to fit them in 64 PlutusData bytes (CTL handles that). A new `Cip25String` type has been introduced (a smart constructor ensures that byte representation fits 64 bytes, as required by the spec). Additionally, a new `Metadata.Cip25.Common.Cip25TokenName` wrapper over `TokenName` is added to ensure proper encoding of `asset_name`s. There are still some minor differences from the spec:
+
+-- We do not split strings in pieces when encoding to JSON
+-- We require a `"version": 2` tag.
+-- `policy_id` must be 28 bytes
+-- `asset_name` is up to 32 bytes.
+
+See https://github.com/cardano-foundation/CIPs/issues/303 for motivation
+- `ogmios-datum-cache` now works on `x86_64-darwin`
+- `TypedValidator` interface ([#808](https://github.com/Plutonomicon/cardano-transaction-lib/issues/808))
 
 ## [2.0.0-alpha] - 2022-07-05
 
 This release adds support for running CTL contracts against Babbage-era nodes. **Note**: this release does not support Babbagge-era features and improvements, e.g. inline datums and reference inputs. Those feature will be implemented in v2.0.0 proper.
 
 ### Added
 
+- Support for using a `PrivateKey` as a `Wallet`.
 - `mkKeyWalletFromFile` helper to use `cardano-cli`-style `skey`s
 - Single `Plutus.Conversion` module exposing all `(Type <-> Plutus Type)` conversion functions ([#464](https://github.com/Plutonomicon/cardano-transaction-lib/pull/464))
-- Support for using a `PrivateKey` as a `Wallet`
 - `logAeson` family of functions to be able to log JSON representations
 - `EncodeAeson` instances for most types under `Cardano.Types.*` as well as other useful types (`Value`, `Coin`, etc.)
 - `getProtocolParameters` call to retrieve current protocol parameters from Ogmios ([#541](https://github.com/Plutonomicon/cardano-transaction-lib/issues/541))

Makefile

@@ -1,17 +1,48 @@
 SHELL := bash
 .ONESHELL:
 .PHONY: run-dev run-build check-format format run-datum-cache-postgres-console
-		query-testnet-tip clean check-explicit-exports
+		query-testnet-tip clean check-explicit-exports e2e-test
 .SHELLFLAGS := -eu -o pipefail -c
 
-ps-sources := $(shell fd -epurs)
-nix-sources := $(shell fd -enix --exclude='spago*')
-hs-sources := $(shell fd . './server/src' './server/exe' -ehs)
-js-sources := $(shell fd -ejs)
-ps-entrypoint := Examples.AlwaysSucceeds
-ps-bundle = spago bundle-module -m ${ps-entrypoint} --to output.js
+# find a suitable browser for e2e-tests
+define e2e-browser
+	$(shell which chromium google-chrome | head -n1)
+endef
+
+# find a suitable temp directory for e2e-tests. snaps don't work
+# in $TMPDIR, because of lacking read access!
+define e2e-temp-base
+	$(if $(findstring snap,$(call e2e-browser)),./tmp,$TMPDIR)
+endef
 
+ps-sources := $(shell fd -epurs -Etmp)
+nix-sources := $(shell fd -enix --exclude='spago*' -Etmp)
+hs-sources := $(shell fd . './server/src' './server/exe' -ehs -Etmp)
+js-sources := $(shell fd -ejs -Etmp)
+ps-entrypoint := Examples.ByUrl
+ps-bundle = spago bundle-module -m ${ps-entrypoint} --to output.js
 node-ipc = $(shell docker volume inspect cardano-transaction-lib_node-ipc | jq -r '.[0].Mountpoint')
+e2e-temp-dir := $(strip $(call e2e-temp-base))/$(shell mktemp -du e2e.XXXXXXX)
+e2e-test-chrome-dir := test-data/chrome-user-data
+
+# bump version here
+e2e-test-nami := test-data/chrome-extensions/nami_3.2.5_1.crx
+e2e-test-nami-settings := test-data/nami_settings.tar.gz
+
+# bump version here
+e2e-test-gero := test-data/chrome-extensions/gero_testnet_1.10.0_0.crx
+e2e-test-gero-settings := test-data/gero_settings.tar.gz
+
+# https://stackoverflow.com/questions/2214575/passing-arguments-to-make-run
+# example: make e2e-test -- --no-headless
+#
+# If the first argument is "e2e-test,"...
+ifeq (e2e-test,$(firstword $(MAKECMDGOALS)))
+  # use the rest as arguments for "e2e-test"
+  TEST_ARGS := $(wordlist 2,$(words $(MAKECMDGOALS)),$(MAKECMDGOALS))
+  # ...and turn them into do-nothing targets
+  $(eval $(TEST_ARGS):;@:)
+endif
 
 run-dev:
 	@${ps-bundle} && BROWSER_RUNTIME=1 webpack-dev-server --progress
@@ -24,13 +55,49 @@ check-explicit-exports:
 		(echo "Use explicit exports:" && \
 		grep -rn '(\.\.)' ./src ./test ./examples)
 
-check-format:
+check-format: check-explicit-exports
 	@purs-tidy check ${ps-sources}
 	nixpkgs-fmt --check ${nix-sources}
 	fourmolu -m check -o -XTypeApplications -o -XImportQualifiedPost ${hs-sources}
 	prettier -c ${js-sources}
 	eslint ${js-sources}
 
+e2e-test:
+	@mkdir -p ${e2e-temp-dir}
+	@unzip ${e2e-test-nami} -d ${e2e-temp-dir}/nami > /dev/zero \
+	    || echo "ignore warnings" # or make stops
+	@tar xzf ${e2e-test-nami-settings}
+	@unzip ${e2e-test-gero} -d ${e2e-temp-dir}/gero > /dev/zero \
+	    || echo "ignore warnings" # or make stops
+	@tar xzf ${e2e-test-gero-settings}
+	@rm -f ${e2e-test-chrome-dir}/SingletonLock
+	@spago test --main Test.E2E -a "E2ETest --nami-dir ${e2e-temp-dir}/nami --gero-dir ${e2e-temp-dir}/gero $(TEST_ARGS) --chrome-exe $(call e2e-browser)" || rm -Rf ${e2e-temp-dir}
+
+e2e-run-browser-nami:
+	@mkdir -p ${e2e-temp-dir}
+	@unzip ${e2e-test-nami} -d ${e2e-temp-dir}/nami > /dev/zero \
+	    || echo "ignore warnings" # or make stops
+	@tar xzf ${e2e-test-nami-settings}
+	@$(call e2e-browser) --load-extension=${e2e-temp-dir}/nami --user-data-dir=${e2e-test-chrome-dir} || rm -Rf ${e2e-temp-dir}
+
+e2e-run-browser-gero:
+	@mkdir -p ${e2e-temp-dir}
+	@unzip ${e2e-test-gero} -d ${e2e-temp-dir}/gero > /dev/zero \
+	   || echo "ignore warnings" # or make stops
+	@tar xzf ${e2e-test-gero-settings}
+	echo $(call e2e-browser) --load-extension=${e2e-temp-dir}/gero --user-data-dir=${e2e-test-chrome-dir} || rm -Rf ${e2e-temp-dir}
+	$(call e2e-browser) --load-extension=${e2e-temp-dir}/gero --user-data-dir=${e2e-test-chrome-dir} || rm -Rf ${e2e-temp-dir}
+
+# extract current nami settings from e2e-test-chrome-dir and store them for git
+nami-settings:
+	tar czf ${e2e-test-nami-settings} ${e2e-test-chrome-dir}/Default/Local\ Extension\ Settings/lpfcbjknijpeeillifnkikgncikgfhdo/*
+
+# extract current gero settings from e2e-test-chrome-dir and store them for git
+gero-settings:
+	tar czf ${e2e-test-gero-settings} \
+		${e2e-test-chrome-dir}/Default/IndexedDB/chrome-extension_iifeegfcfhlhhnilhfoeihllenamcfgc_0.indexeddb.leveldb \
+		${e2e-test-chrome-dir}/Default/Extension\ State
+
 format:
 	@purs-tidy format-in-place ${ps-sources}
 	nixpkgs-fmt ${nix-sources}
@@ -52,3 +119,6 @@ clean:
 	@ rm -rf .spago2nix || true
 	@ rm -rf node_modules || true
 	@ rm -rf output || true
+
+tgt:
+	@echo $(call e2e-temp-base,$(call e2e-browser))

README.md

@@ -25,7 +25,7 @@
 - [x] **Stage 3** Once we have a simple working transaction, we will seek to build a Plutus smart contract transaction with datum from scratch
 - [x] **Stage 4** Once we can construct Plutus smart contract transactions, we will seek to build a library/DSL/interface such that transactions can be built using constraints and lookups - as close as possible to a cut-and-paste solution from Plutus' `Contract` monad code in Haskell (but with no guarantee that code changes are not necessary)
   - [ ] **Stage 4.1** Investigate supporting compatibility with the Vasil hardfork and improvements to our initial `Contract` API (**In progress**)
-- [ ] **Stage 5** Once we have a basic `Contract`-style API, we will further refine its public interface, expand wallet support (see [below](#light-wallet-support)), expose a test interface, provide a more ergonomic JS/TS API, support stake validators, and support CIP workflows on the public testnet
+- [ ] **Stage 5** Once we have a basic `Contract`-style API, we will further refine its public interface, expand wallet support (see [below](#light-wallet-support)), expose a test interface (see [here](doc/plutip-testing.md)), provide a more ergonomic JS/TS API, support stake validators, and support CIP workflows on the public testnet
 - [ ] **Stage 6** Once CTL's `Contract` interface has been stabilized, we will add support for even more wallets and attempt to deprecate CTL's currently required Haskell server
 
 ### Light wallet support
@@ -44,13 +44,15 @@ Support is planned for the following light wallets:
 
 Please explore our documentation to discover how to use CTL, how to set up its runtime, and how it compares to Plutus/PAB:
 
+- [Super quick start](./doc/getting-started.md#setting-up-a-new-project)
 - [Migrating from Plutus to CTL](./doc/plutus-comparison.md)
 - [Adding CTL as a dependency](./doc/ctl-as-dependency.md)
 - [Getting started writing CTL contracts](./doc/getting-started.md)
 - [CTL's runtime dependencies](./doc/runtime.md)
 - [Developing on CTL](./doc/development.md)
+- [Testing contracts with Plutip](./doc/plutip-testing.md)
 
-You can also [generate Purescript documentation for CTL and its dependencies](./doc/development.md#generating-ps-documentation).
+You can also access [PureScript documentation for CTL and its dependencies](https://plutonomicon.github.io/cardano-transaction-lib/) for the most recent `develop` version, or [generate it yourself](./doc/development.md#generating-ps-documentation).
 
 ## Architecture
 
@@ -82,5 +84,6 @@ CTL is directly inspired by the Plutus Application Backend (PAB). Unlike PAB, ho
 ## Available support channels info
 
 You can find help, more information and ongoing discusion about the project here:
+
 - Plutonomicon Discord: https://discord.gg/c8kZWxzJ
 - #ctl channel at MLabs' Slack