SQL Schema #227

rvl · 2019-05-07T07:18:10Z

Relates to #147

Overview

I have added schema for Wallet, TxInput, TxOutput, TxMeta, Utxo, Slot

Comments

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

lib/core/cardano-wallet-core.cabal

lib/core/src/Cardano/Wallet/DB/SqliteTypes.hs

lib/core/cardano-wallet-core.cabal

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

lib/core/src/Cardano/Wallet/DB/SqliteTypes.hs

stack.yaml

paweljakubas · 2019-05-08T06:50:44Z

just made compilation fix and rebased

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

All comments ahve been addressed

jonathanknowles

So far so good! 👍

I've added a few comments to the review.

Also, there are a few parts that should be fixed to adhere to the coding standards. I've highlighted these in the comments, but I also created a PR (#246) with the problems fixed, which you can use directly if desired.

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

lib/core/test/unit/Cardano/Wallet/DB/SqliteSpec.hs

lib/text-class/test/unit/Data/Text/ClassSpec.hs

lib/core/src/Cardano/Wallet/DB/SqliteTypes.hs

lib/text-class/test/unit/Data/Text/ClassSpec.hs

jonathanknowles · 2019-05-09T07:07:58Z

lib/core/src/Cardano/Wallet/DB/SqliteTypes.hs

+instance PersistField AddressScheme where
+    toPersistValue = toPersistValue . schemeToBool
+    fromPersistValue pv = do
+        let err = T.pack $ "not a valid value: " <> show pv


You could probably write this as:

"not a valid value: " <> T.pack (show pv)

jonathanknowles · 2019-05-09T07:10:02Z

lib/core/src/Cardano/Wallet/DB/SqliteTypes.hs

+
+schemeToBool :: AddressScheme -> Bool
+schemeToBool Sequential = True
+schemeToBool Random = False


Perhaps this can be renamed to isSchemeSequential. (Provide an equivalent isSchemeRandom if desired.)

Hmmm... Just seeing this bit from the PR, but I don't think a Bool is a wise choice for this one... We will likely introduce more schemes in the future (special ones for exchanges), and serializing this to Bool might bite us later. Also, we already have a 3rd scheme that we use in the benchmarks Any, and this will need to work with the DB as well...

It's not used yet. We can turn it into an enum later.

changed to data AddressScheme = Sequential | Random | Any

jonathanknowles · 2019-05-09T07:11:11Z

lib/core/src/Cardano/Wallet/DB/SqliteTypes.hs

+schemeToBool Sequential = True
+schemeToBool Random = False
+
+schemeFromBool :: Bool -> AddressScheme


Perhaps this deserves a comment, to explain the exact mapping that will take place. (Important for someone reading the haddock documentation.)

@paweljakubas

changed to data AddressScheme = Sequential | Random | Any

Just to clarify. If I'm a casual user of this function, it currently isn't clear how a given Bool value will map onto an AddressScheme. There are two possible mappings:

{True, False} -> {Sequential, Random}

{True, False} -> {Random, Sequential}

To understand precisely which mapping is being used, I'd have to dig into the source code, which kind of defeats the point.

I would recommend either:

Removing this function (as this mapping is somewhat arbitrary).

Renaming it (so that the exact mapping performed is clear from the name).

Adding a haddock documentation comment to make the mapping explicit.

Introduced data AddressScheme = Sequential | Random | Any now relying on FromText, ToText - so basically opted for point 1 and 2 ;-)

paweljakubas · 2019-05-09T09:33:06Z

@jonathanknowles @KtorZ if we are ok with the current state, I can rebase the branch and merge it

KtorZ · 2019-05-09T09:36:20Z

I haven't reviewed thoroughly any of this, just quick looks. I trust you guys on this :)

akegalj · 2019-05-10T14:36:49Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

+-- blocks).
+--
+-- TxMeta is specific to a wallet because multiple wallets may have the same
+-- transaction with different metdata values. The associated inputs and outputs


typo s/metdata/metadata/

thanks! corrected

akegalj · 2019-05-10T14:41:34Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

+    txMetaTableSlotId     W.SlotId     sql=slot_id
+    txMetaTableAmount     W.Coin       sql=amount
+
+    Primary txMetaTableTxId txMetaTableWalletId


I believe this can be shortend to only TxId as every TxId should be unique by itself and WalletId shouldn't be needed to persist uniqness.

Also I don't think we need to have WalletId in this table

addressed here #227 (comment)

akegalj · 2019-05-10T14:52:41Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

+-- of the transaction are in the TxIn and TxOut tables.
+TxMeta
+    txMetaTableTxId       TxId         sql=tx_id
+    txMetaTableWalletId   W.WalletId   sql=wallet_id


I think we don't need reference to Wallet (ie, WalletId) in this table.

conceptually every TxMeta is part of some transaction Tx and every Tx is part of some or many Wallets. If someone needs to query TxMeta, like with he will most likely already have TxId of the transaction that he is working with.

Also conceptually every Tx might be part of multiple wallets: tx will show up in a wallet from which we sent the tx, and it might show up in a wallet to which we sent some tx. In a similar fashion - coresponding TxMeta (as it is part of Tx) might be part of multiple Wallets (sender and receiver wallet), not necessary one Wallet as specified with WalletId foreign key.

I might be on a wrong track but I would rather leave WalletId out for now as at least conceptually it doesn't make sense to have it here.

Because I thought Wallet and Tx are in many-to-many relationship. Every transaction can be in multiple wallets, and every wallet can have multiple transactions

But instead of having WalletId in all these tables related to Tx (like TxMeta, TxOut, TxIn, ...) why not create WalletTx table that will keep all pairs of WalletId and TxId:

WalletTx walletId WalletId txId TxId Primary walletId txId

this way we can have this many-to-many relationship in one place without having to manage WalletId in every table that we have.

Maybe I am wrong :/ ? (I am still getting warmed up with sql)

akegalj · 2019-05-10T14:53:22Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

+    txMetaTableAmount     W.Coin       sql=amount
+
+    Primary txMetaTableTxId txMetaTableWalletId
+    Foreign Wallet fk_wallet_tx_meta txMetaTableWalletId


I think we should remove WalletId from TxMeta - see comment above.

#227 (comment)

akegalj · 2019-05-10T15:01:35Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

+-- not the wallet. txInputTableTxId is referred to by TxMeta and PendingTx
+TxIn
+    txInputTableTxId         TxId        sql=tx_id
+    txInputTableWalletId     W.WalletId  sql=wallet_id


Similar answer as I said above - I think we should have WalletId reference here

akegalj · 2019-05-10T15:02:05Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

+-- not the wallet. txOutputTableTxId is referred to by TxMeta and PendingTx
+TxOut
+    txOutputTableTxId     TxId        sql=tx_id
+    txOutputTableWalletId W.WalletId  sql=wallet_id


Similar answer as I said above - I think we should have WalletId reference here

akegalj · 2019-05-10T15:21:06Z

lib/core/src/Cardano/Wallet/DB/Sqlite.hs

+-- not the wallet. txInputTableTxId is referred to by TxMeta and PendingTx
+TxIn
+    txInputTableTxId         TxId        sql=tx_id
+    txInputTableWalletId     W.WalletId  sql=wallet_id


also I am not sure I understand why we have two TxIds here ?

rvl · 2019-05-10T23:04:17Z

@akegalj Hi Ante, the schema will become clearer when we implement the insert/query methods.

TxIn and TxOut are exactly the same regardless of which wallet they "belong" to. That's why they have no wallet_id.
The inputs and outputs of a transaction must be associated together. That's why the TxIn and TxOut tables have tx_ids.
The TxIn database also has a TxId for its source transaction. That's why the TxIn table has two tx_id columns.
TxMeta will be different depending on which wallet it is associated with. That's why TxMeta has a wallet_id. Suppose you have a database with two wallets. Funds are sent from one wallet to the other. The TxMeta associated with this transaction should say direction=outgoing for one wallet and direction=incoming for the other wallet.

Thanks for reviewing the substance of this PR rather than just its formatting.

akegalj · 2019-05-13T08:09:50Z

@rvl

TxMeta will be different depending on which wallet it is associated with. That's why TxMeta has a wallet_id. Suppose you have a database with two wallets. Funds are sent from one wallet to the other. The TxMeta associated with this transaction should say direction=outgoing for one wallet and direction=incoming for the other wallet.

ah yes - didn't thought about this case

our coding standards.

improve on percentage improve error msg comply with review remarks

The fromPersistentValue instance had an infinite loop.

Concluding import fixes Typo fix

akegalj

Mostly looks good to me.

This implementation will sattle when we start implementing queries on the DB and with that collect more experience.

What confuses me currently is last point raised by @jonathanknowles . There are bits in this implementation that look like:

instance Read TxId where
    readsPrec _ = error "readsPrec stub needed for persistent"

and while I was going through persistent and trying to understnad I could find the explanation why and where Read will be used. Seems like this is a constrained from PersistentEntity class https://hackage.haskell.org/package/persistent-2.5/docs/Database-Persist-Class.html#t:PersistEntity .

I would need further to dig the library to understand what for this is used as I can't clearly understand where it is used.

We are going to issue a PR in future to fix this

paweljakubas · 2019-05-13T09:55:13Z

@jonathanknowles your comments in the review are addressed. As you do not feel ok to check it and we want to have it merged finally I click on Dismiss review - just to proceed further

comments addressed and we do not want to wait until next day

jonathanknowles reviewed May 7, 2019

View reviewed changes

lib/core/src/Cardano/Wallet/DB/Sqlite.hs Outdated Show resolved Hide resolved

jonathanknowles reviewed May 7, 2019

View reviewed changes

lib/core/src/Cardano/Wallet/DB/Sqlite.hs Outdated Show resolved Hide resolved

rvl commented May 7, 2019

View reviewed changes

lib/core/src/Cardano/Wallet/DB/Sqlite.hs Outdated Show resolved Hide resolved

jonathanknowles reviewed May 7, 2019

View reviewed changes

lib/core/src/Cardano/Wallet/DB/Sqlite.hs Outdated Show resolved Hide resolved

paweljakubas mentioned this pull request May 7, 2019

Define SQL Schema representing relevant Haskell types #147

Closed

5 tasks

paweljakubas force-pushed the rvl/147/sql-schema branch from d9840e9 to d05b30c Compare May 7, 2019 11:29

paweljakubas assigned rvl and paweljakubas May 7, 2019

paweljakubas changed the title ~~WIP SQL Schema~~ SQL Schema May 7, 2019

paweljakubas force-pushed the rvl/147/sql-schema branch from d05b30c to a8b4100 Compare May 7, 2019 13:25

paweljakubas marked this pull request as ready for review May 7, 2019 13:25

akegalj reviewed May 7, 2019

View reviewed changes

lib/core/cardano-wallet-core.cabal Outdated Show resolved Hide resolved

akegalj reviewed May 7, 2019

View reviewed changes

lib/core/src/Cardano/Wallet/DB/SqliteTypes.hs Show resolved Hide resolved

akegalj reviewed May 7, 2019

View reviewed changes

lib/core/src/Cardano/Wallet/DB/SqliteTypes.hs Outdated Show resolved Hide resolved

KtorZ previously requested changes May 7, 2019

View reviewed changes

paweljakubas force-pushed the rvl/147/sql-schema branch from 79d789e to 20a80f6 Compare May 8, 2019 06:50

paweljakubas requested a review from KtorZ May 8, 2019 08:24

paweljakubas force-pushed the rvl/147/sql-schema branch from e3981de to 55e7bbc Compare May 8, 2019 11:19

paweljakubas reviewed May 8, 2019

View reviewed changes

lib/core/src/Cardano/Wallet/DB/Sqlite.hs Outdated Show resolved Hide resolved

paweljakubas reviewed May 8, 2019

View reviewed changes

lib/core/src/Cardano/Wallet/DB/Sqlite.hs Outdated Show resolved Hide resolved

rvl force-pushed the rvl/147/sql-schema branch from da54f0b to 07fd333 Compare May 9, 2019 02:48

rvl added this to the SQLite implementation for the DB Layer milestone May 9, 2019

jonathanknowles previously requested changes May 9, 2019

View reviewed changes

paweljakubas force-pushed the rvl/147/sql-schema branch from 98a325b to 6bc357d Compare May 9, 2019 09:29

paweljakubas requested a review from jonathanknowles May 9, 2019 09:31

paweljakubas force-pushed the rvl/147/sql-schema branch from 6bc357d to 58834a6 Compare May 9, 2019 11:48

paweljakubas force-pushed the rvl/147/sql-schema branch 2 times, most recently from d219095 to f2c65ad Compare May 10, 2019 13:33

akegalj reviewed May 10, 2019

View reviewed changes

paweljakubas force-pushed the rvl/147/sql-schema branch from f2c65ad to a1efedb Compare May 10, 2019 14:41

akegalj reviewed May 10, 2019

View reviewed changes

paweljakubas force-pushed the rvl/147/sql-schema branch from a1efedb to ca6be91 Compare May 11, 2019 18:01

paweljakubas mentioned this pull request May 11, 2019

Set up database connection and create tables #247

Merged

3 tasks

paweljakubas force-pushed the rvl/147/sql-schema branch from ca6be91 to a3ec3a6 Compare May 13, 2019 08:54

rvl and others added 12 commits May 13, 2019 10:56

Add flatSlot/fromFlatSlot to Cardano.Wallet.Primitive.Types

24e5426

Add Data.Text.Class.fromTextMaybe

522b470

Add TxStatus support and improve on schema

0e6d475

First refactoring of review comments

8556f53

Initial persistent SQL schema

a7c47ad

Add a test for flatSlot/fromFlatSlot

82ee336

Add another test for fromText/fromTextMaybe

6df5cd4

Tidy up some of the formatting within Cardano.Wallet.DB.Sqlite to meet

a161b14

our coding standards.

Wrap code in Cardano.Wallet.DB.SqliteSpec to meet our coding standard.

69ffbbd

Extend AddressSchema

85fca9d

improve on percentage improve error msg comply with review remarks

Fix AddressScheme PersistField instance

25781fb

The fromPersistentValue instance had an infinite loop.

Last estetic bits

a3ec3a6

Concluding import fixes Typo fix

akegalj approved these changes May 13, 2019

View reviewed changes

paweljakubas requested a review from jonathanknowles May 13, 2019 09:55

paweljakubas merged commit 102e37d into master May 13, 2019

KtorZ deleted the rvl/147/sql-schema branch May 13, 2019 10:35

akegalj mentioned this pull request May 21, 2019

Extend the current DB layer with an SQLite interface #154

Closed

12 tasks

KtorZ removed this from the SQLite implementation for the DB Layer milestone Jun 5, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SQL Schema #227

SQL Schema #227

rvl commented May 7, 2019 •

edited by paweljakubas

Loading

paweljakubas commented May 8, 2019

jonathanknowles left a comment

jonathanknowles May 9, 2019

paweljakubas May 9, 2019

jonathanknowles May 9, 2019

KtorZ May 9, 2019

rvl May 9, 2019

paweljakubas May 9, 2019

jonathanknowles May 9, 2019

rvl May 9, 2019

paweljakubas May 9, 2019 •

edited

Loading

jonathanknowles May 9, 2019 •

edited

Loading

paweljakubas May 9, 2019

paweljakubas commented May 9, 2019

KtorZ commented May 9, 2019

akegalj May 10, 2019

paweljakubas May 10, 2019

akegalj May 10, 2019

akegalj May 13, 2019

akegalj May 10, 2019

akegalj May 10, 2019

akegalj May 13, 2019

akegalj May 10, 2019

akegalj May 10, 2019

akegalj May 10, 2019

rvl commented May 10, 2019

akegalj commented May 13, 2019

akegalj left a comment

paweljakubas commented May 13, 2019

SQL Schema #227

SQL Schema #227

Conversation

rvl commented May 7, 2019 • edited by paweljakubas Loading

Overview

Comments

paweljakubas commented May 8, 2019

jonathanknowles left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paweljakubas May 9, 2019 • edited Loading

Choose a reason for hiding this comment

jonathanknowles May 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paweljakubas commented May 9, 2019

KtorZ commented May 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rvl commented May 10, 2019

akegalj commented May 13, 2019

akegalj left a comment

Choose a reason for hiding this comment

paweljakubas commented May 13, 2019

rvl commented May 7, 2019 •

edited by paweljakubas

Loading

paweljakubas May 9, 2019 •

edited

Loading

jonathanknowles May 9, 2019 •

edited

Loading