Decrease coverage requirement in UTxOIndexSpec #2574

Anviking · 2021-03-23T13:06:58Z

Issue Number

#2575

Overview

Decrease all coverage 80 to coverage 70 inside UTxOIndexSpec

Comments

rvl

Do you happen to know why cover 80 was chosen before, and why it's now too much?

Anviking · 2021-03-29T13:01:46Z

I simply saw the #2575 failure, and thought it was an innocent slightly-to-strict cover-limit from when the tests were introduced. cc @jonathanknowles ?

jonathanknowles · 2021-03-30T02:46:42Z

Do you happen to know why cover 80 was chosen before, and why it's now too much?

When we originally set these coverage requirements, the average real coverage was consistently higher. (I recall running these tests hundreds of times to make sure.)

However, if the underlying generators have recently been changed, then the average coverage could have dropped, causing the tests to sporadically fail.

Nevertheless, 70 should still be more than adequate here.

jonathanknowles

Hi @Anviking!

This looks fine to me.

See #2574 (comment)

Anviking · 2021-03-30T07:18:21Z

bors r+

2574: Decrease coverage requirement in UTxOIndexSpec r=Anviking a=Anviking # Issue Number #2575  # Overview  - [x] Decrease all `coverage 80` to `coverage 70` inside UTxOIndexSpec # Comments  Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

iohk-bors · 2021-03-30T07:25:50Z

Build failed:

buildkite/cardano-wallet

Anviking · 2021-03-30T07:29:37Z

bors r+

2574: Decrease coverage requirement in UTxOIndexSpec r=Anviking a=Anviking # Issue Number #2575  # Overview  - [x] Decrease all `coverage 80` to `coverage 70` inside UTxOIndexSpec # Comments  Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

iohk-bors · 2021-03-30T07:37:24Z

Build failed:

buildkite/cardano-wallet

Anviking · 2021-03-30T12:33:26Z

bors r+

2574: Decrease coverage requirement in UTxOIndexSpec r=Anviking a=Anviking # Issue Number #2575  # Overview  - [x] Decrease all `coverage 80` to `coverage 70` inside UTxOIndexSpec # Comments  Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

iohk-bors · 2021-03-30T13:01:57Z

Build failed:

ci/hydra-build:required

Failures:

  test/unit/Cardano/Wallet/Api/Server/TlsSpec.hs:79:5: 
  1) Cardano.Wallet.Api.Server.Tls, TLS Client Authentication, Respond to authenticated client if TLS is enabled
       uncaught exception: HttpException
       HttpExceptionRequest Request {
         host                 = "127.0.0.1"
         port                 = 38153
         secure               = True
         requestHeaders       = []
         path                 = "/"
         queryString          = ""
         method               = "GET"
         proxy                = Nothing
         rawBody              = False
         redirectCount        = 10
         responseTimeout      = ResponseTimeoutDefault
         requestVersion       = HTTP/1.1
       }
        (InternalException (Terminated True "received fatal error: CertificateUnknown" (Error_Protocol ("remote side fatal error",True,CertificateUnknown))))

  To rerun use: --match "/Cardano.Wallet.Api.Server.Tls/TLS Client Authentication/Respond to authenticated client if TLS is enabled/"

Randomized with seed 392927842

Anviking · 2021-03-30T13:19:50Z

bors r+

2574: Decrease coverage requirement in UTxOIndexSpec r=Anviking a=Anviking # Issue Number #2575  # Overview  - [x] Decrease all `coverage 80` to `coverage 70` inside UTxOIndexSpec # Comments  Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

iohk-bors · 2021-03-30T13:38:45Z

Build failed:

ci/hydra-build:required

Failures:

  test/unit/Cardano/Wallet/Api/Server/TlsSpec.hs:79:5: 
  1) Cardano.Wallet.Api.Server.Tls, TLS Client Authentication, Respond to authenticated client if TLS is enabled
       uncaught exception: HttpException
       HttpExceptionRequest Request {
         host                 = "127.0.0.1"
         port                 = 42167
         secure               = True
         requestHeaders       = []
         path                 = "/"
         queryString          = ""
         method               = "GET"
         proxy                = Nothing
         rawBody              = False
         redirectCount        = 10
         responseTimeout      = ResponseTimeoutDefault
         requestVersion       = HTTP/1.1
       }
        (InternalException (Terminated True "received fatal error: CertificateUnknown" (Error_Protocol ("remote side fatal error",True,CertificateUnknown))))

  To rerun use: --match "/Cardano.Wallet.Api.Server.Tls/TLS Client Authentication/Respond to authenticated client if TLS is enabled/"

Randomized with seed 1225658022

Anviking · 2021-03-30T13:43:14Z

bors r+

2574: Decrease coverage requirement in UTxOIndexSpec r=Anviking a=Anviking # Issue Number #2575  # Overview  - [x] Decrease all `coverage 80` to `coverage 70` inside UTxOIndexSpec # Comments  Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

iohk-bors · 2021-03-30T14:01:31Z

Build failed:

ci/hydra-build:required

Anviking · 2021-04-01T11:31:06Z

bors r+

2574: Decrease coverage requirement in UTxOIndexSpec r=Anviking a=Anviking # Issue Number #2575  # Overview  - [x] Decrease all `coverage 80` to `coverage 70` inside UTxOIndexSpec # Comments  Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

iohk-bors · 2021-04-01T12:31:07Z

Build failed:

buildkite/cardano-wallet

cardano-wallet                 >
--
  | cardano-wallet                 >   src/Test/Integration/Framework/DSL.hs:2017:7:
  | cardano-wallet                 >   1) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_JOIN_01rewards - Can join a pool, earn rewards and collect them
  | cardano-wallet                 >        expected a successful response but got an error: DecodeFailure "{\"code\":\"created_invalid_transaction\",\"message\":\"That's embarrassing. It looks like I've created an invalid transaction that could not be parsed by the node. Here's an error message that may help with debugging: HardForkApplyTxErrFromEra S (S (S (Z (WrapApplyTxErr {unwrapApplyTxErr = ApplyTxError [LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \\\"8fb209fce870f9ffe39ebe853fc838f8e289ace1e7eaa230378827e6\\\")},Coin 1534631)])))]}))))\"}"
  | cardano-wallet                 >        While verifying (Status {statusCode = 500, statusMessage = "Internal Server Error"},Left (DecodeFailure "{\"code\":\"created_invalid_transaction\",\"message\":\"That's embarrassing. It looks like I've created an invalid transaction that could not be parsed by the node. Here's an error message that may help with debugging: HardForkApplyTxErrFromEra S (S (S (Z (WrapApplyTxErr {unwrapApplyTxErr = ApplyTxError [LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \\\"8fb209fce870f9ffe39ebe853fc838f8e289ace1e7eaa230378827e6\\\")},Coin 1534631)])))]}))))\"}"))
  | cardano-wallet                 >
  | cardano-wallet                 >   To rerun use: --match "/API Specifications/SHELLEY_STAKE_POOLS/STAKE_POOLS_JOIN_01rewards - Can join a pool, earn rewards and collect them/"
  | cardano-wallet                 >

#2467

Anviking · 2021-04-01T12:32:56Z

bors r+

2574: Decrease coverage requirement in UTxOIndexSpec r=Anviking a=Anviking # Issue Number #2575  # Overview  - [x] Decrease all `coverage 80` to `coverage 70` inside UTxOIndexSpec # Comments  Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

iohk-bors · 2021-04-01T14:59:35Z

Build failed:

ci/hydra-build:required

Has been failing twice recently. Latest failure was Failures: test/unit/Cardano/Wallet/Primitive/Types/UTxOIndexSpec.hs:146:9: 1) Cardano.Wallet.Primitive.Types.UTxOIndex, Indexed UTxO set properties, Index Selection, prop_selectRandom_one_withAsset Insufficient coverage (after 800 tests): 89.5% index has more than one asset 71.1% index has the specified asset 71.1% selected an entry Only 71.1% index has the specified asset, but expected 80.0% Only 71.1% selected an entry, but expected 80.0% To rerun use: --match "/Cardano.Wallet.Primitive.Types.UTxOIndex/Indexed UTxO set properties/Index Selection/prop_selectRandom_one_withAsset/" test/unit/Cardano/Wallet/Primitive/Types/UTxOIndexSpec.hs:148:9: 2) Cardano.Wallet.Primitive.Types.UTxOIndex, Indexed UTxO set properties, Index Selection, prop_selectRandom_one_withAssetOnly Insufficient coverage (after 800 tests): 89.5% index has more than one asset 71.1% index has the specified asset 22.5% selected an entry Only 71.1% index has the specified asset, but expected 80.0% To rerun use: --match "/Cardano.Wallet.Primitive.Types.UTxOIndex/Indexed UTxO set properties/Index Selection/prop_selectRandom_one_withAssetOnly/" test/unit/Cardano/Wallet/Primitive/Types/UTxOIndexSpec.hs:154:9: 3) Cardano.Wallet.Primitive.Types.UTxOIndex, Indexed UTxO set properties, Index Selection, prop_selectRandom_all_withAsset Insufficient coverage (after 800 tests): 89.5% index has more than one asset 71.1% index has the specified asset 71.1% selected at least one entry Only 71.1% index has the specified asset, but expected 80.0% Only 71.1% selected at least one entry, but expected 80.0% To rerun use: --match "/Cardano.Wallet.Primitive.Types.UTxOIndex/Indexed UTxO set properties/Index Selection/prop_selectRandom_all_withAsset/" test/unit/Cardano/Wallet/Primitive/Types/UTxOIndexSpec.hs:156:9: 4) Cardano.Wallet.Primitive.Types.UTxOIndex, Indexed UTxO set properties, Index Selection, prop_selectRandom_all_withAssetOnly Insufficient coverage (after 800 tests): 89.5% index has more than one asset 71.1% index has the specified asset 22.5% selected at least one entry Only 71.1% index has the specified asset, but expected 80.0% To rerun use: --match "/Cardano.Wallet.Primitive.Types.UTxOIndex/Indexed UTxO set properties/Index Selection/prop_selectRandom_all_withAssetOnly/"

Anviking · 2021-04-01T15:01:20Z

bors r+

2574: Decrease coverage requirement in UTxOIndexSpec r=Anviking a=Anviking # Issue Number #2575  # Overview  - [x] Decrease all `coverage 80` to `coverage 70` inside UTxOIndexSpec # Comments  Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

iohk-bors · 2021-04-01T15:19:23Z

Build failed:

ci/hydra-build:required

Anviking · 2021-04-01T19:29:58Z

bors r+
pls

2574: Decrease coverage requirement in UTxOIndexSpec r=Anviking a=Anviking # Issue Number #2575  # Overview  - [x] Decrease all `coverage 80` to `coverage 70` inside UTxOIndexSpec # Comments  Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

iohk-bors · 2021-04-01T19:43:28Z

Build failed:

ci/hydra-build:required

#duplicate (cached failure)

Anviking · 2021-04-02T18:39:31Z

bors r+

2574: Decrease coverage requirement in UTxOIndexSpec r=Anviking a=Anviking # Issue Number #2575  # Overview  - [x] Decrease all `coverage 80` to `coverage 70` inside UTxOIndexSpec # Comments  Co-authored-by: Johannes Lund <johannes.lund@iohk.io>

iohk-bors · 2021-04-02T19:31:48Z

Build failed:

buildkite/cardano-wallet

cardano-wallet                 >   src/Test/Integration/Scenario/CLI/Shelley/HWWallets.hs:152:26:
--
  | cardano-wallet                 >   1) CLI Specifications, SHELLEY_CLI_HW_WALLETS, HW_WALLETS_01x - Restoration from account public key preserves funds
  | cardano-wallet                 >        While verifying ApiWallet {id = ApiT {getApiT = WalletId {getWalletId = d7218c30c579c9e96e4a7ce311e96f299474d782}}, addressPoolGap = ApiT {getApiT = AddressPoolGap {getAddressPoolGap = 20}}, balance = ApiWalletBalance {available = Quantity {getQuantity = 0}, total = Quantity {getQuantity = 0}, reward = Quantity {getQuantity = 0}}, assets = ApiWalletAssetsBalance {available = ApiT {getApiT = TokenMap (fromList [])}, total = ApiT {getApiT = TokenMap (fromList [])}}, delegation = ApiWalletDelegation {active = ApiWalletDelegationNext {status = NotDelegating, target = Nothing, changesAt = Nothing}, next = []}, name = ApiT {getApiT = WalletName {getWalletName = "Wallet from pub key"}}, passphrase = Nothing, state = ApiT {getApiT = Syncing (Quantity {getQuantity = Percentage {getPercentage = 3777 % 5000}})}, tip = ApiBlockReference {absoluteSlotNumber = ApiT {getApiT = SlotNo 5942}, slotId = ApiSlotId {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 118}}, slotNumber = ApiT {getApiT = SlotInEpoch {unSlotInEpoch = 42}}}, time = 2021-04-02 19:21:03.4 UTC, block = ApiBlockInfo {height = Quantity {getQuantity = 3000}}}}
  | cardano-wallet                 >        Waited longer than 90s to resolve action: "Wallet balance is as expected on wallet from pubKey".

#2428

Anviking · 2021-04-02T19:45:45Z

bors r+

iohk-bors · 2021-04-02T20:40:33Z

Build succeeded:

Anviking requested a review from jonathanknowles March 23, 2021 13:07

Anviking mentioned this pull request Mar 23, 2021

Better error handling in restore benchmark #2568

Merged

Anviking added Test failure A flaky test or nightly CI failure and removed Test failure A flaky test or nightly CI failure labels Mar 23, 2021

Anviking self-assigned this Mar 24, 2021

rvl reviewed Mar 29, 2021

View reviewed changes

jonathanknowles approved these changes Mar 30, 2021

View reviewed changes

Anviking force-pushed the anviking/lower-coverage branch from 4b32e03 to 6b4a6ee Compare April 1, 2021 11:30

Anviking force-pushed the anviking/lower-coverage branch from 6b4a6ee to d4c29e2 Compare April 1, 2021 15:00

iohk-bors bot merged commit 7acc8ac into master Apr 2, 2021

iohk-bors bot deleted the anviking/lower-coverage branch April 2, 2021 20:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Decrease coverage requirement in UTxOIndexSpec #2574

Decrease coverage requirement in UTxOIndexSpec #2574

Anviking commented Mar 23, 2021 •

edited

rvl left a comment

Anviking commented Mar 29, 2021 •

edited

jonathanknowles commented Mar 30, 2021

jonathanknowles left a comment

Anviking commented Mar 30, 2021

iohk-bors bot commented Mar 30, 2021

Anviking commented Mar 30, 2021

iohk-bors bot commented Mar 30, 2021

Anviking commented Mar 30, 2021

iohk-bors bot commented Mar 30, 2021 •

edited by Anviking

Anviking commented Mar 30, 2021

iohk-bors bot commented Mar 30, 2021 •

edited by Anviking

Anviking commented Mar 30, 2021

iohk-bors bot commented Mar 30, 2021

Anviking commented Apr 1, 2021

iohk-bors bot commented Apr 1, 2021 •

edited by Anviking

Anviking commented Apr 1, 2021

iohk-bors bot commented Apr 1, 2021

Anviking commented Apr 1, 2021

iohk-bors bot commented Apr 1, 2021

Anviking commented Apr 1, 2021

iohk-bors bot commented Apr 1, 2021 •

edited by Anviking

Anviking commented Apr 2, 2021

iohk-bors bot commented Apr 2, 2021 •

edited by Anviking

Anviking commented Apr 2, 2021

iohk-bors bot commented Apr 2, 2021

Decrease coverage requirement in UTxOIndexSpec #2574

Decrease coverage requirement in UTxOIndexSpec #2574

Conversation

Anviking commented Mar 23, 2021 • edited

Issue Number

Overview

Comments

rvl left a comment

Choose a reason for hiding this comment

Anviking commented Mar 29, 2021 • edited

jonathanknowles commented Mar 30, 2021

jonathanknowles left a comment

Choose a reason for hiding this comment

Anviking commented Mar 30, 2021

iohk-bors bot commented Mar 30, 2021

Anviking commented Mar 30, 2021

iohk-bors bot commented Mar 30, 2021

Anviking commented Mar 30, 2021

iohk-bors bot commented Mar 30, 2021 • edited by Anviking

Anviking commented Mar 30, 2021

iohk-bors bot commented Mar 30, 2021 • edited by Anviking

Anviking commented Mar 30, 2021

iohk-bors bot commented Mar 30, 2021

Anviking commented Apr 1, 2021

iohk-bors bot commented Apr 1, 2021 • edited by Anviking

Anviking commented Apr 1, 2021

iohk-bors bot commented Apr 1, 2021

Anviking commented Apr 1, 2021

iohk-bors bot commented Apr 1, 2021

Anviking commented Apr 1, 2021

iohk-bors bot commented Apr 1, 2021 • edited by Anviking

Anviking commented Apr 2, 2021

iohk-bors bot commented Apr 2, 2021 • edited by Anviking

Anviking commented Apr 2, 2021

iohk-bors bot commented Apr 2, 2021

Anviking commented Mar 23, 2021 •

edited

Anviking commented Mar 29, 2021 •

edited

iohk-bors bot commented Mar 30, 2021 •

edited by Anviking

iohk-bors bot commented Mar 30, 2021 •

edited by Anviking

iohk-bors bot commented Apr 1, 2021 •

edited by Anviking

iohk-bors bot commented Apr 1, 2021 •

edited by Anviking

iohk-bors bot commented Apr 2, 2021 •

edited by Anviking