Sqlite: enable db property tests and fix failures #300

rvl · 2019-05-22T06:48:36Z

Relates to issue #154

Overview

Runs the dbPropertyTests in SqliteSpec.
Adds fixes for the failures.

paweljakubas · 2019-05-22T19:33:08Z

@rvl These are my conclusions (hope will help you hammer out the solution) :

They are deterministic and strange. For example, when I am running :

$ stack test --ta '-m "put . read yields a result"' cardano-wallet-core:test:unit

the test fails for "put . read yields a result" -> Tx History always at 63/100 test and this is irrespective of what WalletId was chosen.
2. The tests fail because of ErrWalletAlreadyExists. I investigated what happens. So in passing test we have something like that :

cleanDB wal BEFORE : [PrimaryKey (WalletId {getWalletId = 854fed8387aabb4ed709147b98b3f0c4e4804691})]
cleanDB wal AFTER : []
wal=[]
key=PrimaryKey (WalletId {getWalletId = ad54f8e826744f5fa76a1730bdf9b65771a6e976})

so we have cleanDB doing its job, as a result when the key is chosen for the test case (by createWallet) the wal=[] (this is listWallet). I would say this is as expected. In case of failure we have,

wal=[PrimaryKey (WalletId {getWalletId = ad54f8e826744f5fa76a1730bdf9b65771a6e976})]
      Tx History FAILED [1]
cleanDB wal BEFORE : [PrimaryKey (WalletId {getWalletId = ad54f8e826744f5fa76a1730bdf9b65771a6e976})]
cleanDB wal AFTER : []

ie., wal is not empty, test fails. cleanDB is missing. It comes only when the next test case runs.
This is very strange. as we use beforeAll in a predictable way we have situation that in 63/100 test the call to cleanDB is missing. If we run the same command as above we have different WalletIds.
3. Why I get to the conclusion that cleanDB is missing. It comes only when the next test case runs. So I introduced MVar into the code. In cleanDB I putMVar and in the tests (in the beginning) I takeMVar. And I got stucked :

cleanDB wal BEFORE : [PrimaryKey (WalletId {getWalletId =6645b53521f8eaf97e3d6404385b1835da0bb135})]
cleanDB wal AFTER : []
wal=[]
key=PrimaryKey (WalletId {getWalletId = ad54f8e826744f5fa76a1730bdf9b65771a6e976})
cleanDB wal BEFORE : [PrimaryKey (WalletId {getWalletId = ad54f8e826744f5fa76a1730bdf9b65771a6e976})]
cleanDB wal AFTER : []
wal=[]
key=PrimaryKey (WalletId {getWalletId = 8adfe5c94630e8ae4b47d6e4c9363ef31f5fa36b})
cleanDB wal BEFORE : [PrimaryKey (WalletId {getWalletId = 8adfe5c94630e8ae4b47d6e4c9363ef31f5fa36b})]
cleanDB wal AFTER : []
wal=[]
key=PrimaryKey (WalletId {getWalletId = 1d05f4517d2decf9b118b7a48e46a534b42f67a4})
^C

rvl · 2019-05-24T06:59:45Z

@paweljakubas Thanks for the debugging and fixes.

KtorZ · 2019-05-24T08:26:27Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

+    , Wallet (..)
+    , migrateAll
+    , unWalletKey
+    )


Okay ^.^ , I had kept this one separate as an "exception", but I admit this is more consistent and aligned with our style.

KtorZ · 2019-05-24T08:41:36Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

-                insertCheckpoint wid cp
+        { createWallet = \(PrimaryKey wid) cp meta -> withWriteLock $
+            ExceptT $ runQuery conn $ do
+                res <- handleConstraint (ErrWalletAlreadyExists wid) $


Are we sure that any constraint errors are actually because of already existing wallet id?

KtorZ · 2019-05-24T08:48:46Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

+                res <- handleConstraint (ErrWalletAlreadyExists wid) $
+                    insert_ (mkWalletEntity wid meta)
+                when (isRight res) $
+                    insertCheckpoint wid cp


~~So, in case a wallet already exists (isLeft res), this returns a success 🤔 ?~~

Okay actually not. I was confused initially because we aren't in an 'Either' or 'ExceptT' monad here.

KtorZ · 2019-05-24T09:00:27Z

lib/core/test/unit/Cardano/Wallet/DBSpec.hs

@@ -182,7 +179,7 @@ instance Arbitrary TxMeta where
    arbitrary = TxMeta
        <$> elements [Pending, InLedger, Invalidated]
        <*> elements [Incoming, Outgoing]
-        <*> (SlotId <$> arbitrary <*> choose (0, 21600))
+        <*> (SlotId <$> choose (0, 1000) <*> choose (0, 21599))


KtorZ · 2019-05-24T09:03:01Z

lib/core/test/unit/Cardano/Wallet/DBSpec.hs

+    arbitrary = initWallet <$> arbitrary
+
+instance Eq (SeqState DummyTarget) where
+    _ == _ = True


What's the rationale for this instance? And what is the justification for it to be always true ?

I believe (cannot say in the name of Rodney) it is because it is transparent in the context of tests we deliver here

It feels somewhat wrong though. Why would any SeqState be equal? I wonder where this enters into play.

yes, we need to add :

191 instance Eq (SeqState DummyTarget) where 192 (SeqState int1 ext1 pen1) == (SeqState int2 ext2 pen2) = 193 (int1 == int2) && (ext1 == ext2) && (pen1 == pen2)

or just :

deriving instance Eq (SeqState DummyTarget)

KtorZ · 2019-05-24T09:06:54Z

lib/core/test/unit/Cardano/Wallet/Primitive/AddressDiscoverySpec.hs

@@ -13,6 +13,7 @@

 module Cardano.Wallet.Primitive.AddressDiscoverySpec
    ( spec
+    , DummyTarget


Hmmm... Not really sure about this. The test and instance from AddressDiscovery are quite specific and tailored for the properties of that module. They only use a single account key for instance. I guess it's okay for the DBSpec, but I'd rather not do that yet, and stick with our current approach of not sharing instances between spec files. This is an open-gate for messiness later.

We may discuss how we can better share generators between spec files without making spec files depend on each others. It becomes quite hard then to follow which arbitrary instance comes from where....

moved to DBspec needed bits, and here export DummyTarget to MVarSpec and SliteSpec.

KtorZ · 2019-05-24T09:09:21Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

-- | Delete transactions that belong to a wallet and aren't referred to by
-- either Pending or TxMeta.
-deleteLooseTransactions
+-- | Delete unused TxMeta values for a wallet.


Comment is a bit of a lie here 😬 ... the function doesn't care much about whether the meta is used or unused, it just cleans them all.

fixed. also take care of indenting in this file. Some functions had 2-character indents in this file

paweljakubas

A lot of work and effort was put into this PR. LGTM!

KtorZ

LGTM 👍
Nice investigation work.

- putMany was keeping the old TxMeta entries, rather than replacing them. - Clean up imports.

rvl added this to the SQLite implementation for the DB Layer milestone May 22, 2019

rvl self-assigned this May 22, 2019

akegalj mentioned this pull request May 22, 2019

Extend the current DB layer with an SQLite interface #154

Closed

12 tasks

rvl force-pushed the rvl/154/dblayer-properties branch 2 times, most recently from d08f5f5 to ee6ad17 Compare May 24, 2019 06:55

rvl marked this pull request as ready for review May 24, 2019 06:55

rvl force-pushed the rvl/154/dblayer-properties branch from ee6ad17 to 689cf6e Compare May 24, 2019 07:20

KtorZ reviewed May 24, 2019

View reviewed changes

paweljakubas force-pushed the rvl/154/dblayer-properties branch from 689cf6e to 5b66760 Compare May 24, 2019 11:48

paweljakubas approved these changes May 24, 2019

View reviewed changes

paweljakubas force-pushed the rvl/154/dblayer-properties branch from 5b66760 to ce69389 Compare May 24, 2019 12:55

KtorZ approved these changes May 24, 2019

View reviewed changes

rvl and others added 10 commits May 24, 2019 14:58

tests: DBLayer properties to use SeqState, also run with Sqlite

94cef8b

tests: Add a simple checkpoint tests to SqliteSpec

7095e2e

Sqlite: fix test failures

84ce3de

add regression test illustrating problem with TxHistory

4acd287

Correct deleteLooseTransactions

2e24c31

DBSpec: Correct the TxMeta SlotId generator

ef33737

Sqlite: Fix updating of TxMeta in putTxHistory

8f8ca64

- putMany was keeping the old TxMeta entries, rather than replacing them. - Clean up imports.

Sqlite: add application-level locking around DBLayer writes

096c281

Reintroduce DummyTarget to DBSpec and comply with review remarks

1a44b2b

correct Eq (SeqState DummyTarget) instance

ce69389

KtorZ merged commit e73b111 into master May 24, 2019

iohk-bors bot deleted the rvl/154/dblayer-properties branch May 24, 2019 13:33

KtorZ removed this from the SQLite implementation for the DB Layer milestone Jun 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sqlite: enable db property tests and fix failures #300

Sqlite: enable db property tests and fix failures #300

rvl commented May 22, 2019 •

edited

Loading

paweljakubas commented May 22, 2019

rvl commented May 24, 2019

KtorZ May 24, 2019

KtorZ May 24, 2019

KtorZ May 24, 2019

KtorZ May 24, 2019

KtorZ May 24, 2019

paweljakubas May 24, 2019

KtorZ May 24, 2019

paweljakubas May 24, 2019 •

edited

Loading

KtorZ May 24, 2019

paweljakubas May 24, 2019

KtorZ May 24, 2019

paweljakubas May 24, 2019

paweljakubas left a comment

KtorZ left a comment

Sqlite: enable db property tests and fix failures #300

Sqlite: enable db property tests and fix failures #300

Conversation

rvl commented May 22, 2019 • edited Loading

Overview

paweljakubas commented May 22, 2019

rvl commented May 24, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paweljakubas May 24, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paweljakubas left a comment

Choose a reason for hiding this comment

KtorZ left a comment

Choose a reason for hiding this comment

rvl commented May 22, 2019 •

edited

Loading

paweljakubas May 24, 2019 •

edited

Loading