Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Review the .travis.yaml file from cardano-wallet (+ connect travis to new repository) #1

Closed
3 of 6 tasks
piotr-iohk opened this issue Mar 1, 2019 · 4 comments
Closed
3 of 6 tasks
Assignees

Comments

@piotr-iohk
Copy link
Contributor

piotr-iohk commented Mar 1, 2019

Context

We want to have CI set up for the new cardano-wallet in order to have a means to automatically build the code and run all our tests against the build. At this moment we decided to go with Travis, so we want to get the .travis.yaml from the previous cardano-wallet review it and remove all unnecessary dependencies to cardano-sl from it. This task is also to make sure that the new repo is connected to Travis CI.

Decision

Connect new cardano-wallet repo with Travis CI. Get the .travis.yaml from the previous cardano-wallet review it and remove all unnecessary dependencies to cardano-sl from it. This task is also to make sure that the new repo is connected to Travis CI.

AC

  • Travis is running builds on PRs.
  • Travis is running builds on the master branch.
  • Tests are being executed by Travis.

Development Plan

  • Initial Travis build which runs stack test.
  • Connect repo to travis.
  • Also run Coveralls and stylish-haskell.

PR

Number Base
#15 master
#19 master

QA

  • Check that the list of AC is complete and done.
@rvl
Copy link
Contributor

rvl commented Mar 4, 2019

rvl added a commit that referenced this issue Mar 4, 2019
@rvl rvl self-assigned this Mar 4, 2019
rvl added a commit that referenced this issue Mar 4, 2019
rvl added a commit that referenced this issue Mar 4, 2019
@KtorZ
Copy link
Member

KtorZ commented Mar 4, 2019

I'll also add support for stylish-haskell & coveralls 👍

Anviking added a commit that referenced this issue Mar 4, 2019
…h-haskell

Add Coveralls & Stylish Haskell to CI
@piotr-iohk
Copy link
Contributor Author

lgtm 👍

I think it would be nice to have stylish-haskell, weeder, hlint, tests as separate checks in Travis for better readibility (as it was on previous repo). I think also the checks are then capable of runnig in parallel.
Perhaps it is not needed now but I will add that as a thought to the wiki for discussion.

@KtorZ
Copy link
Member

KtorZ commented Mar 5, 2019

Agreed @piotr-iohk , and ideally, we would even push it further and have the status reported inside github directly as separate CI checks. @Anviking had some ideas on the matter :)

KtorZ added a commit that referenced this issue Mar 11, 2019
paweljakubas added a commit that referenced this issue Mar 11, 2019
[3] Adding ticking function test and downloading block logic

[3] Block syncer working ok plus additional tests

[3] Cleaning the code

[3] cabal fix

[3] Complete refactoring to the review

[3] Final refactoring

[3] cabal fix

[3] hlint and weeding

[3] add killing thread at the end of test

[3] Replace IORef with MVar

refactoring #1 | inline intermediate functions to see more clearly + remove debug console prints

refactoring #2 | review indentation of 'where' clause

refactoring #3 | inline loop and remove test intrumentation from test logic

refactoring #4 | define generator for ticking args in a declarative manner

refactoring #5 | purify tickingFunctionTest and make it a monadic property

refactoring #6 | review naming in Arbitrary TickingArgs

refactoring #7 | use guards in mkConsecutiveTestBlocks

refactoring #8 | define single block generator from previous block

refactoring #9 | use 'fromPreviousBlock' and start loop with an already initialize list

refactoring #10 | purify mkConsecutiveTestBlocks by defining a test hash function

refactoring #11 | switch argument positions in mkConsecutiveTestBlocks

refactoring #12 | replace loop with built-in list 'iterate'

refactoring #13 | define Arbitrary instance for creating consecutive blocks

refactoring #14 | replace mkConsecutiveBlocks with a property parameter

refactoring #15 | use 'newMVar' instead or 'newEmptyMVar' + 'putMVar'

refactoring #16 | remove unecessary IO in 'writeToIORefAction'

refactoring #17 | replace takeMVar + putMVar with modifyMVar

refactoring #18 | review naming for 'writeToIORefAction' --> 'reader'

refactoring #19 | use a 'Map.lookup' instead of 'List.filter' + pattern-match

refactoring #20 | remove 'BlocksConsumed' wrapper

refactoring #21 | generalize reader with polymorphic parametrism

refactoring #22 | review pushNextBlocks indentation

refactoring #23 | group case pattern matches using tuple

refactoring #24 | remove 'Hash BlockHeader' from the block to inject

refactoring #25 | use synchronization lock instead of computed times

refactoring #26 | Move generation of duplicated blocks onto 'Arbitrary Blocks'

refactoring #27 | remove 'chunkSizes' in a favor of inline random selection

refactoring #28 | remove 'DeliveryMode' in favor of the most general case

refactoring #29 | cleanup wrapper types

refactoring #30 | generalize pushNextBlocks with parametric polymorphism

refactoring #31 | rename pushNextBlocks to 'writer'

refactoring #32 | define reader on Block instead of BlockHeaderHash

refactoring #33 | replace old reader with reader'

refactoring #34 | move creation of writer MVar inside writer action

refactoring #35 | remove header hash from 'Blocks'

refactoring #36 | rename 'consecutiveBlocks' into 'blocks'

refactoring #37 | re-organize module to separate effectful logic from declarations

refactoring #38 | move waiting logic into dedicated function

refactoring #39 | move 'done' and 'readerChan' initialization into reader and writer

refactoring #40 | use Millisecond instead of Second for shorter tests

refactoring #41 | Move creation of blocks from writer to 'Arbitrary Blocks'

[3] fix line width

[3] aligning the code with other code changes
paweljakubas added a commit that referenced this issue Mar 11, 2019
[3] Adding ticking function test and downloading block logic

[3] Block syncer working ok plus additional tests

[3] Cleaning the code

[3] cabal fix

[3] Complete refactoring to the review

[3] Final refactoring

[3] cabal fix

[3] hlint and weeding

[3] add killing thread at the end of test

[3] Replace IORef with MVar

refactoring #1 | inline intermediate functions to see more clearly + remove debug console prints

refactoring #2 | review indentation of 'where' clause

refactoring #3 | inline loop and remove test intrumentation from test logic

refactoring #4 | define generator for ticking args in a declarative manner

refactoring #5 | purify tickingFunctionTest and make it a monadic property

refactoring #6 | review naming in Arbitrary TickingArgs

refactoring #7 | use guards in mkConsecutiveTestBlocks

refactoring #8 | define single block generator from previous block

refactoring #9 | use 'fromPreviousBlock' and start loop with an already initialize list

refactoring #10 | purify mkConsecutiveTestBlocks by defining a test hash function

refactoring #11 | switch argument positions in mkConsecutiveTestBlocks

refactoring #12 | replace loop with built-in list 'iterate'

refactoring #13 | define Arbitrary instance for creating consecutive blocks

refactoring #14 | replace mkConsecutiveBlocks with a property parameter

refactoring #15 | use 'newMVar' instead or 'newEmptyMVar' + 'putMVar'

refactoring #16 | remove unecessary IO in 'writeToIORefAction'

refactoring #17 | replace takeMVar + putMVar with modifyMVar

refactoring #18 | review naming for 'writeToIORefAction' --> 'reader'

refactoring #19 | use a 'Map.lookup' instead of 'List.filter' + pattern-match

refactoring #20 | remove 'BlocksConsumed' wrapper

refactoring #21 | generalize reader with polymorphic parametrism

refactoring #22 | review pushNextBlocks indentation

refactoring #23 | group case pattern matches using tuple

refactoring #24 | remove 'Hash BlockHeader' from the block to inject

refactoring #25 | use synchronization lock instead of computed times

refactoring #26 | Move generation of duplicated blocks onto 'Arbitrary Blocks'

refactoring #27 | remove 'chunkSizes' in a favor of inline random selection

refactoring #28 | remove 'DeliveryMode' in favor of the most general case

refactoring #29 | cleanup wrapper types

refactoring #30 | generalize pushNextBlocks with parametric polymorphism

refactoring #31 | rename pushNextBlocks to 'writer'

refactoring #32 | define reader on Block instead of BlockHeaderHash

refactoring #33 | replace old reader with reader'

refactoring #34 | move creation of writer MVar inside writer action

refactoring #35 | remove header hash from 'Blocks'

refactoring #36 | rename 'consecutiveBlocks' into 'blocks'

refactoring #37 | re-organize module to separate effectful logic from declarations

refactoring #38 | move waiting logic into dedicated function

refactoring #39 | move 'done' and 'readerChan' initialization into reader and writer

refactoring #40 | use Millisecond instead of Second for shorter tests

refactoring #41 | Move creation of blocks from writer to 'Arbitrary Blocks'

[3] fix line width

[3] aligning the code with other code changes

[3] hlint suggestion
paweljakubas added a commit that referenced this issue Mar 11, 2019
[3] Adding ticking function test and downloading block logic

[3] Block syncer working ok plus additional tests

[3] Cleaning the code

[3] cabal fix

[3] Complete refactoring to the review

[3] Final refactoring

[3] cabal fix

[3] hlint and weeding

[3] add killing thread at the end of test

[3] Replace IORef with MVar

refactoring #1 | inline intermediate functions to see more clearly + remove debug console prints

refactoring #2 | review indentation of 'where' clause

refactoring #3 | inline loop and remove test intrumentation from test logic

refactoring #4 | define generator for ticking args in a declarative manner

refactoring #5 | purify tickingFunctionTest and make it a monadic property

refactoring #6 | review naming in Arbitrary TickingArgs

refactoring #7 | use guards in mkConsecutiveTestBlocks

refactoring #8 | define single block generator from previous block

refactoring #9 | use 'fromPreviousBlock' and start loop with an already initialize list

refactoring #10 | purify mkConsecutiveTestBlocks by defining a test hash function

refactoring #11 | switch argument positions in mkConsecutiveTestBlocks

refactoring #12 | replace loop with built-in list 'iterate'

refactoring #13 | define Arbitrary instance for creating consecutive blocks

refactoring #14 | replace mkConsecutiveBlocks with a property parameter

refactoring #15 | use 'newMVar' instead or 'newEmptyMVar' + 'putMVar'

refactoring #16 | remove unecessary IO in 'writeToIORefAction'

refactoring #17 | replace takeMVar + putMVar with modifyMVar

refactoring #18 | review naming for 'writeToIORefAction' --> 'reader'

refactoring #19 | use a 'Map.lookup' instead of 'List.filter' + pattern-match

refactoring #20 | remove 'BlocksConsumed' wrapper

refactoring #21 | generalize reader with polymorphic parametrism

refactoring #22 | review pushNextBlocks indentation

refactoring #23 | group case pattern matches using tuple

refactoring #24 | remove 'Hash BlockHeader' from the block to inject

refactoring #25 | use synchronization lock instead of computed times

refactoring #26 | Move generation of duplicated blocks onto 'Arbitrary Blocks'

refactoring #27 | remove 'chunkSizes' in a favor of inline random selection

refactoring #28 | remove 'DeliveryMode' in favor of the most general case

refactoring #29 | cleanup wrapper types

refactoring #30 | generalize pushNextBlocks with parametric polymorphism

refactoring #31 | rename pushNextBlocks to 'writer'

refactoring #32 | define reader on Block instead of BlockHeaderHash

refactoring #33 | replace old reader with reader'

refactoring #34 | move creation of writer MVar inside writer action

refactoring #35 | remove header hash from 'Blocks'

refactoring #36 | rename 'consecutiveBlocks' into 'blocks'

refactoring #37 | re-organize module to separate effectful logic from declarations

refactoring #38 | move waiting logic into dedicated function

refactoring #39 | move 'done' and 'readerChan' initialization into reader and writer

refactoring #40 | use Millisecond instead of Second for shorter tests

refactoring #41 | Move creation of blocks from writer to 'Arbitrary Blocks'

[3] fix line width

[3] aligning the code with other code changes

[3] hlint suggestion

[3] add .weeder.yaml to omit duplicateMaybes and groups to be announce during weeder execution

[3] undo weeder ignore file plus remove pragma and unnecessary exports
paweljakubas added a commit that referenced this issue Mar 11, 2019
[3] Adding ticking function test and downloading block logic

[3] Block syncer working ok plus additional tests

[3] Cleaning the code

[3] cabal fix

[3] Complete refactoring to the review

[3] Final refactoring

[3] cabal fix

[3] hlint and weeding

[3] add killing thread at the end of test

[3] Replace IORef with MVar

refactoring #1 | inline intermediate functions to see more clearly + remove debug console prints

refactoring #2 | review indentation of 'where' clause

refactoring #3 | inline loop and remove test intrumentation from test logic

refactoring #4 | define generator for ticking args in a declarative manner

refactoring #5 | purify tickingFunctionTest and make it a monadic property

refactoring #6 | review naming in Arbitrary TickingArgs

refactoring #7 | use guards in mkConsecutiveTestBlocks

refactoring #8 | define single block generator from previous block

refactoring #9 | use 'fromPreviousBlock' and start loop with an already initialize list

refactoring #10 | purify mkConsecutiveTestBlocks by defining a test hash function

refactoring #11 | switch argument positions in mkConsecutiveTestBlocks

refactoring #12 | replace loop with built-in list 'iterate'

refactoring #13 | define Arbitrary instance for creating consecutive blocks

refactoring #14 | replace mkConsecutiveBlocks with a property parameter

refactoring #15 | use 'newMVar' instead or 'newEmptyMVar' + 'putMVar'

refactoring #16 | remove unecessary IO in 'writeToIORefAction'

refactoring #17 | replace takeMVar + putMVar with modifyMVar

refactoring #18 | review naming for 'writeToIORefAction' --> 'reader'

refactoring #19 | use a 'Map.lookup' instead of 'List.filter' + pattern-match

refactoring #20 | remove 'BlocksConsumed' wrapper

refactoring #21 | generalize reader with polymorphic parametrism

refactoring #22 | review pushNextBlocks indentation

refactoring #23 | group case pattern matches using tuple

refactoring #24 | remove 'Hash BlockHeader' from the block to inject

refactoring #25 | use synchronization lock instead of computed times

refactoring #26 | Move generation of duplicated blocks onto 'Arbitrary Blocks'

refactoring #27 | remove 'chunkSizes' in a favor of inline random selection

refactoring #28 | remove 'DeliveryMode' in favor of the most general case

refactoring #29 | cleanup wrapper types

refactoring #30 | generalize pushNextBlocks with parametric polymorphism

refactoring #31 | rename pushNextBlocks to 'writer'

refactoring #32 | define reader on Block instead of BlockHeaderHash

refactoring #33 | replace old reader with reader'

refactoring #34 | move creation of writer MVar inside writer action

refactoring #35 | remove header hash from 'Blocks'

refactoring #36 | rename 'consecutiveBlocks' into 'blocks'

refactoring #37 | re-organize module to separate effectful logic from declarations

refactoring #38 | move waiting logic into dedicated function

refactoring #39 | move 'done' and 'readerChan' initialization into reader and writer

refactoring #40 | use Millisecond instead of Second for shorter tests

refactoring #41 | Move creation of blocks from writer to 'Arbitrary Blocks'

[3] fix line width

[3] aligning the code with other code changes

[3] hlint suggestion

[3] add .weeder.yaml to omit duplicateMaybes and groups to be announce during weeder execution

[3] undo weeder ignore file plus remove pragma and unnecessary exports

[3] remove not needed imports
iohk-bors bot added a commit that referenced this issue Jan 21, 2020
1290: Force rollback of tables with foreign keys / cascading deletes on automatic db migrations r=KtorZ a=KtorZ

# Issue Number

<!-- Put here a reference to the issue this PR relates to and which requirements it tackles -->

#1279 

# Overview

<!-- Detail in a few bullet points the work accomplished in this PR -->

- 74c232e
  force rollback to genesis after automated migrations
    `persistent` clear foreign tables on automatic migration due to
  cascading delete. One possible approach to solve this: forcing a
  manual rollback to genesis whenever there are migrations applied
  to the database, regardless of the migration. Only the foreign
  tables are affected, so static metadata and wallet credentials
  are preserved. This mean that we would force resync of wallets
  on every updates that have database changes, but that's a fair
  price to pay for now.

- de3825d
  remove foreign keys to 'Wallet' table and cascade deletes manually
  This to prevent 'persistent' from doing it during automated migration! We don't do
this for checkpoints because the cascading updates are truly helpful there to manage
the database state and, can be more easily rolled back at any time (since the data
regarding checkpoints entirely comes from the chain).


# Comments

<!-- Additional comments or screenshots to attach if any -->

~~:warning: Currently manually testing db migration from various previous versions.~~

Automated migration passes! And on old version, we have a nice log showing what's going on, including manual migration, and handling of automated ones!

```
 *** Migrating from v2019-12-13 ***

[cardano-wallet.application:Info:22] [2020-01-20 17:58:12.47 UTC] Found existing wallet: 99ef09bdab5f62579f2d6db02ba7df5e88b7e865
[cardano-wallet.wallet-db:Notice:28] [2020-01-20 17:58:12.47 UTC] checkpoint table does not contain required field 'active_slot_coeff'. Adding this field with a default value of 1.0.
[cardano-wallet.wallet-db:Notice:28] [2020-01-20 17:58:12.52 UTC] 48 migrations were applied to the database.
[cardano-wallet.wallet-db:Notice:28] [2020-01-20 17:58:12.52 UTC] Some automated database migrations happened. Forced to rollback to genesis.
[cardano-wallet.main:Info:22] [2020-01-20 17:58:12.53 UTC] Wallet backend server listening on 127.0.0.1:8090
[cardano-wallet.wallet-engine:Info:28] [2020-01-20 17:58:12.53 UTC] 99ef09bd: Applying blocks [1167051.7 ... 1167051.7]
[cardano-wallet.wallet-engine:Info:28] [2020-01-20 17:58:12.53 UTC] 99ef09bd: Creating checkpoint at 3b2894d8-[1167051.7#1]
```

So, to summarize what's happening:

1. An existing wallet database is found. 
2. Before running any automatic migration, the database is inspected and an extra column is found. So we manually alter the table to add a column with a default value.
3. Then, persistent runs its own set of automated migrations on top of the database, running 48 of them.
4. This is then detected and induces immediately a rollback to genesis of all the on-chain data.
5. The wallet starts applying blocks and creating checkpoints, starting with a checkpoint at height `#1` (right after genesis)

:tada: 


<!-- 
Don't forget to:

 ✓ Self-review your changes to make sure nothing unexpected slipped through
 ✓ Assign yourself to the PR
 ✓ Assign one or several reviewer(s)
 ✓ Once created, link this PR to its corresponding ticket
 ✓ Assign the PR to a corresponding milestone
 ✓ Acknowledge any changes required to the Wiki
-->


Co-authored-by: KtorZ <matthias.benkort@gmail.com>
rvl added a commit that referenced this issue Mar 20, 2020
It was failing with this message printed repeatedly:

    Mar 19 23:15:23.900 ERRO trusted-peers cannot be empty. to avoid bootstrap use 'skip_bootstrap: true', task: bootstrap
    Mar 19 23:15:23.900 INFO bootstrap attempt #1 failed, trying again in 5 seconds..., task: bootstrap

Then jormungandr would never exit.
rvl added a commit that referenced this issue Mar 20, 2020
It was failing with this message printed repeatedly:

    Mar 19 23:15:23.900 ERRO trusted-peers cannot be empty. to avoid bootstrap use 'skip_bootstrap: true', task: bootstrap
    Mar 19 23:15:23.900 INFO bootstrap attempt #1 failed, trying again in 5 seconds..., task: bootstrap

Then jormungandr would never exit.
rvl added a commit that referenced this issue Mar 31, 2020
It was failing with this message printed repeatedly:

    Mar 19 23:15:23.900 ERRO trusted-peers cannot be empty. to avoid bootstrap use 'skip_bootstrap: true', task: bootstrap
    Mar 19 23:15:23.900 INFO bootstrap attempt #1 failed, trying again in 5 seconds..., task: bootstrap

Then jormungandr would never exit.
Anviking added a commit that referenced this issue Aug 27, 2021
This gives us a fairly easy way to close specific tickets in favor of a
more general one, without having to edit the corresponding bors-comments.

This is done by writing e.g. "(Instance of #2)" for issue #1, if you
want re-annotate failures from #1 as really being from #2.
Anviking added a commit that referenced this issue Aug 27, 2021
This gives us a fairly easy way to close specific tickets in favor of a
more general one, without having to edit the corresponding bors-comments.

This is done by writing e.g. "(Instance of #2)" for issue #1, if you
want re-annotate failures from #1 as really being from #2.
iohk-bors bot added a commit that referenced this issue Aug 29, 2021
2857: Fix leaks of ≈21 more wallets in integration tests r=rvl a=Anviking

- [x] Replace 3 manual calls to `Link.postWallets` with the `ResourceT` `postByronWallet`
- [x] Also make `createWalletFromPublicKeyViaCLI` run in `ResourceT`

### Comments

- 21 wallets is a pretty significant number, so will be good to fix. Although the leaks were at the end of the tests, which lessens the impact.
- There seem to be _some_ concurrency related leaks.

#### Before second fix

- --match "WALLET" -j 1 ->18 leaks, all at the end

#### After fixes

- --match "WALLET" -j 1, no leaks
- --match "WALLET" -j 8, 10 leaks (3 BUSY, 6 others)

```
SQLITE_BUSY in logs:
ica.14828b804a1ebe804caddd1abb5d314d2be0a62a.sqlite
ica.25341683360a040721f587d069b95762b302f608.sqlite
rnd.5fc0960622bc4d3a5d366a0920dfd9b5ef578ced.sqlite

rnd.81273c47add77e98b3993e01cab8b2a836cd48b7.sqlite
she.95cff5e68f07b759c72d4da4f1aa96e0b101bba2.sqlite
she.a99218ce4bbc77f05617d642c0595246621b67b6.sqlite

Remaining DBs:
ica.14828b804a1ebe804caddd1abb5d314d2be0a62a.sqlite
ica.25341683360a040721f587d069b95762b302f608.sqlite
rnd.5fc0960622bc4d3a5d366a0920dfd9b5ef578ced.sqlite

ica.ed49f6e7723fa2c15575c147566981beab6aba3e.sqlite
rnd.693940686b48985ab6ec8035a56575e0473e96de.sqlite
rnd.81273c47add77e98b3993e01cab8b2a836cd48b7.sqlite
sha.9316fc445533f9e58becf0d367d3a9404585ea86.sqlite
she.3464b427c15341dab8aaf6d89c8f79e5f24f9eae.sqlite
she.a0bd7c6accac92d54fb667bf32bcba4db4cd552f.sqlite
stake-pools.sqlite
```


<!-- Additional comments, links, or screenshots to attach, if any. -->

### Issue Number

ADP-1090


2858: Add (Instance of #ISSUE) re-write rule for bors comments r=rvl a=Anviking

- [x] Add simple re-write rules for bors-failure tags using annotations in Github issue titles

### Comments

This gives us a fairly easy way to close specific tickets in favor of a
more general one, without having to edit the corresponding bors-comments.

This is done by writing e.g. "(Instance of #2)" for issue #1, if you
want re-annotate failures from #1 as really being from #2.

**Example:** #2855 

<!-- Additional comments, links, or screenshots to attach, if any. -->

### Issue Number

<!-- Reference the Jira/GitHub issue that this PR relates to, and which requirements it tackles.
  Note: Jira issues of the form ADP- will be auto-linked. -->


Co-authored-by: Johannes Lund <johannes.lund@iohk.io>
rvl pushed a commit that referenced this issue Nov 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants