Use hspec-golden-aeson for automatic golden tests

[Anviking]

Issue Number

#91

Overview

• I have added hspec-golden-aeson as a dependency to our unit tests
• I replaced existing JSON round-trip tests with roundtripAndGoldenSpecsWithSettings
• I ran the tests to generate initial golden test json files

Comments

• Split into two commits

• If the JSON format of WalletBalance fails to match WalletBalance.json, the library will fail and output WalletBalance.faulty.json with the new format. Not sure if/what we should document about this.

The output of roundtrip failures is not the same. 1) is the new and 2) is the old below:

src/Test/Aeson/Internal/RoundtripSpecs.hs:59:5:
  1) Cardano.Wallet.Api.Types, tmp: roundtrip, JSON encoding of A, allows to encode values with aeson and read them back
       Falsifiable (after 1 test):
         "{\"tag\":\"A2\",\"contents\":[{\"age\":\"0\",\"name\":0},\"\"]}"

  To rerun use: --match "/Cardano.Wallet.Api.Types/tmp: roundtrip/JSON encoding of A/allows to encode values with aeson and read them back/"

  test/unit/Cardano/Wallet/Api/TypesSpec.hs:134:18:
  2) Cardano.Wallet.Api.Types, tmp: roundtrip 2, Cardano.Wallet.Api.TypesSpec[127:13]
       Falsifiable (after 1 test):
         A2 (B {b1 = 0, b2 = 0}) ""
       expected: Just (A2 (B {b1 = 0, b2 = 0}) "")
        but got: Nothing

Example output:

stack test cardano-wallet:unit --ta='--match "can perform roundtrip"'
cardano-wallet-2.0.0: unregistering (local file changes: test/unit/Cardano/Wallet/Api/TypesSpec.hs)
cardano-wallet-2.0.0: build (lib + test)
Preprocessing library for cardano-wallet-2.0.0..
Building library for cardano-wallet-2.0.0..
Preprocessing test suite 'unit' for cardano-wallet-2.0.0..
Building test suite 'unit' for cardano-wallet-2.0.0..
[18 of 19] Compiling Cardano.Wallet.Api.TypesSpec ( test/unit/Cardano/Wallet/Api/TypesSpec.hs, .stack-work/dist/x86_64-osx/Cabal-2.4.0.1/build/unit/unit-tmp/Cardano/Wallet/Api/TypesSpec.o )
[19 of 19] Compiling Main             ( test/unit/Main.hs, .stack-work/dist/x86_64-osx/Cabal-2.4.0.1/build/unit/unit-tmp/Main.o ) [Cardano.Wallet.Api.TypesSpec changed]
Linking .stack-work/dist/x86_64-osx/Cabal-2.4.0.1/build/unit/unit ...
cardano-wallet-2.0.0: copy/register
Installing library in /Users/Johannes/IOHK/cardano-wallet/.stack-work/install/x86_64-osx/lts-13.8/8.6.3/lib/x86_64-osx-ghc-8.6.3/cardano-wallet-2.0.0-D27hjXoPYSdCOP2VHI7wg1
Installing executable cardano-wallet-server in /Users/Johannes/IOHK/cardano-wallet/.stack-work/install/x86_64-osx/lts-13.8/8.6.3/bin
Installing executable cardano-generate-mnemonic in /Users/Johannes/IOHK/cardano-wallet/.stack-work/install/x86_64-osx/lts-13.8/8.6.3/bin
Installing executable cardano-wallet-launcher in /Users/Johannes/IOHK/cardano-wallet/.stack-work/install/x86_64-osx/lts-13.8/8.6.3/bin
Registering library for cardano-wallet-2.0.0..
cardano-wallet-2.0.0: test (suite: unit, args: --match "can perform roundtrip")

Progress 1/2: cardano-wallet-2.0.0
Cardano.Wallet.Api.Types
  can perform roundtrip JSON serialization & deserialization, and match existing golden files
    JSON encoding of Wallet
      allows to encode values with aeson and read them back
        +++ OK, passed 100 tests.
    JSON encoding of Wallet
      produces the same JSON as is found in test/data/Cardano/Wallet/Api/Wallet.json
    JSON encoding of (ApiT AddressPoolGap)
      allows to encode values with aeson and read them back
        +++ OK, passed 100 tests.
    JSON encoding of (ApiT AddressPoolGap)
      produces the same JSON as is found in test/data/Cardano/Wallet/Api/ApiT AddressPoolGap.json
    JSON encoding of WalletBalance
      allows to encode values with aeson and read them back
        +++ OK, passed 100 tests.
    JSON encoding of WalletBalance
      produces the same JSON as is found in test/data/Cardano/Wallet/Api/WalletBalance.json
    JSON encoding of WalletDelegation
      allows to encode values with aeson and read them back
        +++ OK, passed 100 tests.
    JSON encoding of WalletDelegation
      produces the same JSON as is found in test/data/Cardano/Wallet/Api/WalletDelegation.json
    JSON encoding of WalletId
      allows to encode values with aeson and read them back
        +++ OK, passed 100 tests.
    JSON encoding of WalletId
      produces the same JSON as is found in test/data/Cardano/Wallet/Api/WalletId.json
    JSON encoding of (ApiT WalletName)
      allows to encode values with aeson and read them back
        +++ OK, passed 100 tests.
    JSON encoding of (ApiT WalletName)
      produces the same JSON as is found in test/data/Cardano/Wallet/Api/ApiT WalletName.json
    JSON encoding of WalletBalance
      allows to encode values with aeson and read them back
        +++ OK, passed 100 tests.
    JSON encoding of WalletBalance
      produces the same JSON as is found in test/data/Cardano/Wallet/Api/WalletBalance.json
    JSON encoding of WalletPassphraseInfo
      allows to encode values with aeson and read them back
        +++ OK, passed 100 tests.
    JSON encoding of WalletPassphraseInfo
      produces the same JSON as is found in test/data/Cardano/Wallet/Api/WalletPassphraseInfo.json
    JSON encoding of WalletState
      allows to encode values with aeson and read them back
        +++ OK, passed 100 tests.
    JSON encoding of WalletState
      produces the same JSON as is found in test/data/Cardano/Wallet/Api/WalletState.json

Finished in 0.0919 seconds
18 examples, 0 failures

cardano-wallet-2.0.0: Test suite unit passed
Completed 2 action(s).

[KtorZ]

@Anviking what do you think of having somewhere (wiki, module documentation comment or README, or an actual executable) the steps required to generate new golden tests files?

[Anviking]

@KtorZ I added a comment to the roundtripAndGolden helper function. We could make it a module level comment too. Though I am not sure if the module, TypesSpec, is intended to be larger than just roundtrip + goldens.

[KtorZ]

Though I am not sure if the module, TypesSpec, is intended to be larger than just roundtrip + goldens.

It will also at least contains the comparison with the swagger spec. I'll push that by the end of the day 👍

[KtorZ]

Ah! It does roundtrip and golden. Perfect, then never mind my previous remark!

[Anviking]

I would have hoped that the number of samples was calculated dynamically to cover all cases of complex ADTs, but that does not seem to be the case from testing in - 1634ff6.

We could adjust the overall sampleSize when needed. The default is 4 per type.

[KtorZ]

Nice work 👍