Further prevent flakiness of LoggingSpec properties #2580

Anviking · 2021-03-25T12:13:50Z

Issue Number

Overview

Use withMaxSuccess 1 to ensure each ctx is only used for one test.

Comments

Quick and I believe fine workaround for Create completely new TVars in LoggingSpec #2573 (review)

We previously ensured each test had its own ctx to prevent interference. But in these property tests it doesn't matter. Let's use withMaxSuccess 1 as a quick workaround - which should still be acceptable.

This reverts commit 321678f321fceca46960b59ebe5b96efcba27224 and 337f742.

rvl

Instead of limiting the property to 1 example (fine, but not nice from a flakiness perspective), we could (a) remove the use of QuickCheck here (see reverted commit), or (b) run a new server for each property example (see latest commit).

Anviking · 2021-03-26T09:18:04Z

Fine, that works too 😸 — thanks
bors r+

2580: Use withMaxSuccess 1 in LoggingSpec properties r=Anviking a=Anviking # Issue Number #2368 # Overview  - [x] Use `withMaxSuccess 1` to ensure each `ctx` is only used for one test. # Comments - Quick and I believe fine workaround for #2573 (review)  Co-authored-by: Johannes Lund <johannes.lund@iohk.io> Co-authored-by: Rodney Lorrimar <rodney.lorrimar@iohk.io>

iohk-bors · 2021-03-26T10:10:42Z

Build failed:

buildkite/cardano-wallet

src/Test/Integration/Scenario/API/Byron/Wallets.hs:187:59:
--
  | cardano-wallet             >   1) API Specifications, BYRON_WALLETS, BYRON_RESTORE_01, GET_01, LIST_01 - Restore a wallet, icarus 18 words
  | cardano-wallet             >        While verifying (Status {statusCode = 200, statusMessage = "OK"},Right (ApiByronWallet {id = ApiT {getApiT = WalletId {getWalletId = 558c149bb91bd95cbd6b53a2359817698d8fcb69}}, balance = ApiByronWalletBalance {available = Quantity {getQuantity = 0}, total = Quantity {getQuantity = 0}}, assets = ApiWalletAssetsBalance {available = ApiT {getApiT = TokenMap (fromList [])}, total = ApiT {getApiT = TokenMap (fromList [])}}, discovery = DiscoverySequential, name = ApiT {getApiT = WalletName {getWalletName = "Empty Byron Wallet"}}, passphrase = Just (ApiWalletPassphraseInfo {lastUpdatedAt = 2021-03-26 09:42:04.068596646 UTC}), state = ApiT {getApiT = Syncing (Quantity {getQuantity = Percentage {getPercentage = 7453 % 10000}})}, tip = ApiBlockReference {absoluteSlotNumber = ApiT {getApiT = SlotNo 1998}, slotId = ApiSlotId {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 39}}, slotNumber = ApiT {getApiT = SlotInEpoch {unSlotInEpoch = 48}}}, time = 2021-03-26 09:39:48.6 UTC, block = ApiBlockInfo {height = Quantity {getQuantity = 1000}}}}))
  | cardano-wallet             >        Waited longer than 90s to resolve action: "wallet is available and ready".
  | cardano-wallet             > [cardano-wallet.network:Warning:1862] [2021-03-26 10:08:34.04 UTC] Connection lost with the node. writev: resource vanished (Broken pipe)
  | cardano-wallet             >        expected: Ready
  | cardano-wallet             >         but got: Syncing (Quantity {getQuantity = Percentage {getPercentage = 7453 % 10000}})
  | cardano-wallet             >
  | cardano-wallet             >   To rerun use: --match "/API Specifications/BYRON_WALLETS/BYRON_RESTORE_01, GET_01, LIST_01 - Restore a wallet/icarus 18 words/"
  | cardano-wallet             >
  | cardano-wallet             >   src/Test/Integration/Framework/DSL.hs:2017:7:
  | cardano-wallet             >   2) API Specifications, SHELLEY_STAKE_POOLS, STAKE_POOLS_JOIN_01rewards - Can join a pool, earn rewards and collect them
  | cardano-wallet             >        expected a successful response but got an error: DecodeFailure "{\"code\":\"created_invalid_transaction\",\"message\":\"That's embarrassing. It looks like I've created an invalid transaction that could not be parsed by the node. Here's an error message that may help with debugging: HardForkApplyTxErrFromEra S (S (S (Z (WrapApplyTxErr {unwrapApplyTxErr = ApplyTxError [LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \\\"8fb209fce870f9ffe39ebe853fc838f8e289ace1e7eaa230378827e6\\\")},Coin 804520)])))]}))))\"}"
  | cardano-wallet             >        While verifying (Status {statusCode = 500, statusMessage = "Internal Server Error"},Left (DecodeFailure "{\"code\":\"created_invalid_transaction\",\"message\":\"That's embarrassing. It looks like I've created an invalid transaction that could not be parsed by the node. Here's an error message that may help with debugging: HardForkApplyTxErrFromEra S (S (S (Z (WrapApplyTxErr {unwrapApplyTxErr = ApplyTxError [LedgerFailure (DelegsFailure (WithdrawalsNotInRewardsDELEGS (fromList [(RewardAcnt {getRwdNetwork = Mainnet, getRwdCred = KeyHashObj (KeyHash \\\"8fb209fce870f9ffe39ebe853fc838f8e289ace1e7eaa230378827e6\\\")},Coin 804520)])))]}))))\"}"))
  | cardano-wallet             >
  | cardano-wallet             >   To rerun use: --match "/API Specifications/SHELLEY_STAKE_POOLS/STAKE_POOLS_JOIN_01rewards - Can join a pool, earn rewards and collect them/"
  | cardano-wallet             >
  | cardano-wallet             >   src/Test/Integration/Scenario/CLI/Shelley/HWWallets.hs:152:26:
  | cardano-wallet             >   3) CLI Specifications, SHELLEY_CLI_HW_WALLETS, HW_WALLETS_01x - Restoration from account public key preserves funds
  | cardano-wallet             >        While verifying ApiWallet {id = ApiT {getApiT = WalletId {getWalletId = 654ef7afd3b6c1cffa736ba7884351668ceff7a0}}, addressPoolGap = ApiT {getApiT = AddressPoolGap {getAddressPoolGap = 20}}, balance = ApiWalletBalance {available = Quantity {getQuantity = 0}, total = Quantity {getQuantity = 0}, reward = Quantity {getQuantity = 0}}, assets = ApiWalletAssetsBalance {available = ApiT {getApiT = TokenMap (fromList [])}, total = ApiT {getApiT = TokenMap (fromList [])}}, delegation = ApiWalletDelegation {active = ApiWalletDelegationNext {status = NotDelegating, target = Nothing, changesAt = Nothing}, next = []}, name = ApiT {getApiT = WalletName {getWalletName = "Wallet from pub key"}}, passphrase = Nothing, state = ApiT {getApiT = Syncing (Quantity {getQuantity = Percentage {getPercentage = 4021 % 10000}})}, tip = ApiBlockReference {absoluteSlotNumber = ApiT {getApiT = SlotNo 4000}, slotId = ApiSlotId {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 80}}, slotNumber = ApiT {getApiT = SlotInEpoch {unSlotInEpoch = 0}}}, time = 2021-03-26 09:46:29 UTC, block = ApiBlockInfo {height = Quantity {getQuantity = 2000}}}}
  | cardano-wallet             >        Waited longer than 90s to resolve action: "Wallet balance is as expected on wallet from pubKey".
  | cardano-wallet             >        expected: Quantity {getQuantity = 1000000}
  | cardano-wallet             >         but got: Quantity {getQuantity = 0}
  | cardano-wallet             >

#2565
#2428
#2467

Anviking · 2021-03-26T11:18:01Z

bors r+

2580: Further prevent flakiness of LoggingSpec properties r=Anviking a=Anviking # Issue Number #2368 # Overview  - [x] Use `withMaxSuccess 1` to ensure each `ctx` is only used for one test. # Comments - Quick and I believe fine workaround for #2573 (review)  Co-authored-by: Johannes Lund <johannes.lund@iohk.io> Co-authored-by: Rodney Lorrimar <rodney.lorrimar@iohk.io>

iohk-bors · 2021-03-26T12:17:57Z

Build failed:

ci/hydra-build:required

  src/Test/Integration/Scenario/CLI/Shelley/Wallets.hs:416:50:
  1) CLI Specifications, SHELLEY_CLI_WALLETS, WALLETS_CREATE_08 - --address-pool-gap values, -1000 -> fail
       uncaught exception: IOException of type ResourceVanished
       fd:209: hFlush: resource vanished (Broken pipe)

  To rerun use: --match "/CLI Specifications/SHELLEY_CLI_WALLETS/WALLETS_CREATE_08 - --address-pool-gap values/-1000 -> fail/"

Anviking · 2021-03-26T12:36:46Z

bors r+

2580: Further prevent flakiness of LoggingSpec properties r=Anviking a=Anviking # Issue Number #2368 # Overview  - [x] Use `withMaxSuccess 1` to ensure each `ctx` is only used for one test. # Comments - Quick and I believe fine workaround for #2573 (review)  Co-authored-by: Johannes Lund <johannes.lund@iohk.io> Co-authored-by: Rodney Lorrimar <rodney.lorrimar@iohk.io>

iohk-bors · 2021-03-26T13:32:19Z

Build failed:

buildkite/cardano-wallet


/tmp/stack-4b9f43a24bed67cc/test-ghc-env: openBinaryFile: resource busy (file is locked)
--
  |  
  | /tmp/stack-4b9f43a24bed67cc/test-ghc-env: openBinaryFile: resource busy (file is locked)
  |  
  | /tmp/stack-4b9f43a24bed67cc/test-ghc-env: openBinaryFile: resource busy (file is locked)

Anviking · 2021-03-26T17:00:53Z

bors r+

2580: Further prevent flakiness of LoggingSpec properties r=Anviking a=Anviking # Issue Number #2368 # Overview  - [x] Use `withMaxSuccess 1` to ensure each `ctx` is only used for one test. # Comments - Quick and I believe fine workaround for #2573 (review)  Co-authored-by: Johannes Lund <johannes.lund@iohk.io> Co-authored-by: Rodney Lorrimar <rodney.lorrimar@iohk.io>

iohk-bors · 2021-03-26T17:51:54Z

Build failed:

buildkite/cardano-wallet

cardano-wallet             > Failures:
--
  | cardano-wallet             >
  | cardano-wallet             >   src/Test/Integration/Scenario/CLI/Shelley/HWWallets.hs:152:26:
  | cardano-wallet             >   1) CLI Specifications, SHELLEY_CLI_HW_WALLETS, HW_WALLETS_01x - Restoration from account public key preserves funds
  | cardano-wallet             >        While verifying ApiWallet {id = ApiT {getApiT = WalletId {getWalletId = 858ba5482bcb7abad14db29bcdb057c27995dd30}}, addressPoolGap = ApiT {getApiT = AddressPoolGap {getAddressPoolGap = 20}}, balance = ApiWalletBalance {available = Quantity {getQuantity = 0}, total = Quantity {getQuantity = 0}, reward = Quantity {getQuantity = 0}}, assets = ApiWalletAssetsBalance {available = ApiT {getApiT = TokenMap (fromList [])}, total = ApiT {getApiT = TokenMap (fromList [])}}, delegation = ApiWalletDelegation {active = ApiWalletDelegationNext {status = NotDelegating, target = Nothing, changesAt = Nothing}, next = []}, name = ApiT {getApiT = WalletName {getWalletName = "Wallet from pub key"}}, passphrase = Nothing, state = ApiT {getApiT = Syncing (Quantity {getQuantity = Percentage {getPercentage = 441 % 1000}})}, tip = ApiBlockReference {absoluteSlotNumber = ApiT {getApiT = SlotNo 4055}, slotId = ApiSlotId {epochNumber = ApiT {getApiT = EpochNo {unEpochNo = 81}}, slotNumber = ApiT {getApiT = SlotInEpoch {unSlotInEpoch = 5}}}, time = 2021-03-26 17:30:02 UTC, block = ApiBlockInfo {height = Quantity {getQuantity = 2000}}}}
  | cardano-wallet             >        Waited longer than 90s to resolve action: "Wallet balance is as expected on wallet from pubKey".
  | cardano-wallet             >        expected: Quantity {getQuantity = 1000000}
  | cardano-wallet             >         but got: Quantity {getQuantity = 0}
  | cardano-wallet             >
  | cardano-wallet             >   To rerun use: --match "/CLI Specifications/SHELLEY_CLI_HW_WALLETS/HW_WALLETS_01x - Restoration from account public key preserves funds/"
  | cardano-wallet             >
  | cardano-wallet             > Randomized with seed 410200852

Anviking · 2021-03-27T11:32:11Z

bors r+

2580: Further prevent flakiness of LoggingSpec properties r=Anviking a=Anviking # Issue Number #2368 # Overview  - [x] Use `withMaxSuccess 1` to ensure each `ctx` is only used for one test. # Comments - Quick and I believe fine workaround for #2573 (review)  Co-authored-by: Johannes Lund <johannes.lund@iohk.io> Co-authored-by: Rodney Lorrimar <rodney.lorrimar@iohk.io>

iohk-bors · 2021-03-27T12:10:40Z

Build failed:

ci/hydra-build:required

Anviking · 2021-03-27T17:54:08Z

bors r+

2580: Further prevent flakiness of LoggingSpec properties r=Anviking a=Anviking # Issue Number #2368 # Overview  - [x] Use `withMaxSuccess 1` to ensure each `ctx` is only used for one test. # Comments - Quick and I believe fine workaround for #2573 (review)  Co-authored-by: Johannes Lund <johannes.lund@iohk.io> Co-authored-by: Rodney Lorrimar <rodney.lorrimar@iohk.io>

iohk-bors · 2021-03-27T18:46:08Z

Build failed:

ci/hydra-build:required

Anviking · 2021-03-28T10:40:03Z

bors r+

2580: Further prevent flakiness of LoggingSpec properties r=Anviking a=Anviking # Issue Number #2368 # Overview  - [x] Use `withMaxSuccess 1` to ensure each `ctx` is only used for one test. # Comments - Quick and I believe fine workaround for #2573 (review)  Co-authored-by: Johannes Lund <johannes.lund@iohk.io> Co-authored-by: Rodney Lorrimar <rodney.lorrimar@iohk.io>

iohk-bors · 2021-03-28T11:28:38Z

Build failed:

buildkite/cardano-wallet

Anviking · 2021-03-28T12:14:24Z

bors r+

Pls

iohk-bors · 2021-03-28T13:04:31Z

Build succeeded:

Use withMaxSuccess 1

337f742

We previously ensured each test had its own ctx to prevent interference. But in these property tests it doesn't matter. Let's use withMaxSuccess 1 as a quick workaround - which should still be acceptable.

Anviking requested a review from rvl March 25, 2021 12:14

Anviking mentioned this pull request Mar 25, 2021

Create completely new TVars in LoggingSpec #2573

Merged

Anviking self-assigned this Mar 25, 2021

rvl added 3 commits March 26, 2021 14:43

Don't use QuickCheck properties for request logging tests

6b91f04

Revert previous

74fe89f

This reverts commit 321678f321fceca46960b59ebe5b96efcba27224 and 337f742.

LoggingSpec: start a server for each QuickCheck example

062c182

rvl approved these changes Mar 26, 2021

View reviewed changes

Anviking changed the title ~~Use withMaxSuccess 1 in LoggingSpec properties~~ Further prevent flakiness of LoggingSpec properties Mar 26, 2021

iohk-bors bot merged commit 933139c into master Mar 28, 2021

iohk-bors bot deleted the anviking/2368/maxSuccess branch March 28, 2021 13:04

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Further prevent flakiness of LoggingSpec properties #2580

Further prevent flakiness of LoggingSpec properties #2580

Anviking commented Mar 25, 2021

rvl left a comment

Anviking commented Mar 26, 2021 •

edited

iohk-bors bot commented Mar 26, 2021 •

edited by Anviking

Anviking commented Mar 26, 2021

iohk-bors bot commented Mar 26, 2021 •

edited by Anviking

Anviking commented Mar 26, 2021

iohk-bors bot commented Mar 26, 2021 •

edited by Anviking

Anviking commented Mar 26, 2021

iohk-bors bot commented Mar 26, 2021 •

edited by Anviking

Anviking commented Mar 27, 2021

iohk-bors bot commented Mar 27, 2021

Anviking commented Mar 27, 2021

iohk-bors bot commented Mar 27, 2021

Anviking commented Mar 28, 2021

iohk-bors bot commented Mar 28, 2021

Anviking commented Mar 28, 2021

iohk-bors bot commented Mar 28, 2021

Further prevent flakiness of LoggingSpec properties #2580

Further prevent flakiness of LoggingSpec properties #2580

Conversation

Anviking commented Mar 25, 2021

Issue Number

Overview

Comments

rvl left a comment

Choose a reason for hiding this comment

Anviking commented Mar 26, 2021 • edited

iohk-bors bot commented Mar 26, 2021 • edited by Anviking

Anviking commented Mar 26, 2021

iohk-bors bot commented Mar 26, 2021 • edited by Anviking

Anviking commented Mar 26, 2021

iohk-bors bot commented Mar 26, 2021 • edited by Anviking

Anviking commented Mar 26, 2021

iohk-bors bot commented Mar 26, 2021 • edited by Anviking

Anviking commented Mar 27, 2021

iohk-bors bot commented Mar 27, 2021

Anviking commented Mar 27, 2021

iohk-bors bot commented Mar 27, 2021

Anviking commented Mar 28, 2021

iohk-bors bot commented Mar 28, 2021

Anviking commented Mar 28, 2021

iohk-bors bot commented Mar 28, 2021

Anviking commented Mar 26, 2021 •

edited

iohk-bors bot commented Mar 26, 2021 •

edited by Anviking

iohk-bors bot commented Mar 26, 2021 •

edited by Anviking

iohk-bors bot commented Mar 26, 2021 •

edited by Anviking

iohk-bors bot commented Mar 26, 2021 •

edited by Anviking