ledger: remove "rowid" from the public interface #5177

icorderi · 2023-03-02T15:42:46Z

Summary

Removed rowid from the public interface
Rowid is now abstracted by a different interface on. each type that required identifier.
This work should allow us to have different identifier content on the kv implementation of the store

Test Plan

Existing tests.

AlgoAxel

Looking at your failing test I think it's just a missing assert update:

        	            	expected: int64(2)
        	            	actual  : sqlitedriver.sqlRowRef(sqlitedriver.sqlRowRef{rowid:2})

Also thinking out loud about how this is implemented -- outside of unit testing, are there compilation errors if production code doesn't use the new Ref structure? Just thinking that since the Ref just an opaque interface, it might be possible to miss updates? Maybe not, hence why I'm asking out loud :)

ledger/store/trackerdb/data.go

AlgoAxel · 2023-03-02T20:52:38Z

ledger/store/trackerdb/sqlitedriver/sql.go

 			if len(buf) > 0 && rowid.Valid {
-				data.Addrid = rowid.Int64
+				data.AcctRef = sqlRowRef{rowid.Int64}


Just leaving the SQL NullInt64 implementation here, since this made me curious about what it looked like:

// NullInt64 represents an int64 that may be null. // NullInt64 implements the Scanner interface so // it can be used as a scan destination, similar to NullString. type NullInt64 struct { Int64 int64 Valid bool // Valid is true if Int64 is not NULL }

codecov · 2023-03-02T22:07:56Z

Codecov Report

Merging #5177 (ca26aac) into master (25e8e94) will decrease coverage by 0.08%.
The diff coverage is 41.98%.

@@            Coverage Diff             @@
##           master    #5177      +/-   ##
==========================================
- Coverage   53.59%   53.52%   -0.08%     
==========================================
  Files         439      439              
  Lines       54950    54985      +35     
==========================================
- Hits        29452    29432      -20     
- Misses      23218    23263      +45     
- Partials     2280     2290      +10

Impacted Files	Coverage Δ
ledger/catchpointtracker.go	`56.66% <0.00%> (-0.77%)`	⬇️
ledger/store/trackerdb/data.go	`81.86% <ø> (ø)`
ledger/store/trackerdb/sqlitedriver/accountsV2.go	`13.54% <0.00%> (-0.38%)`	⬇️
...tore/trackerdb/sqlitedriver/orderedAccountsIter.go	`0.00% <0.00%> (ø)`
ledger/store/trackerdb/sqlitedriver/sql.go	`2.88% <0.00%> (-0.21%)`	⬇️
ledger/acctdeltas.go	`79.08% <77.41%> (-0.16%)`	⬇️
ledger/acctonline.go	`77.66% <100.00%> (ø)`
ledger/acctupdates.go	`69.36% <100.00%> (+0.23%)`	⬆️
crypto/merkletrie/trie.go	`66.42% <0.00%> (-2.19%)`	⬇️
crypto/merkletrie/node.go	`91.62% <0.00%> (-1.87%)`	⬇️
... and 7 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

icorderi · 2023-03-03T13:56:48Z

Also thinking out loud about how this is implemented -- outside of unit testing, are there compilation errors if production code doesn't use the new Ref structure? Just thinking that since the Ref just an opaque interface, it might be possible to miss updates? Maybe not, hence why I'm asking out loud :)

It's a valid point @AlgoAxel, I wasn't too happy with the auto impl of the interface for any type either.
I added some "marker" methods on each to force an explicit impl for any type that can be used as a ref.

AlgoAxel · 2023-03-03T15:54:01Z

ledger/store/trackerdb/interface.go

+// AccountRef is an opaque ref to an account in the db.
+type AccountRef interface {
+	AccountRefMarker()
+}
+
+// OnlineAccountRef is an opaque ref to an "online" account in the db.
+type OnlineAccountRef interface {
+	OnlineAccountRefMarker()
+}
+
+// ResourceRef is an opaque ref to a resource in the db.
+type ResourceRef interface {
+	ResourceRefMarker()
+}


cool idea! this will certainly help keep us from using incompatible ref types.

Tagging @cce specifically because this could be a valuable pattern in other places and I'd like to highlight it. Specifically, our Peer interface is also an opaque ref

ledger/store/trackerdb/interface.go

algorandskiy · 2023-03-03T16:17:39Z

ledger/acctdeltas.go

 		switch err {
 		case nil:
 			if len(acctDataBuf) > 0 {
-				persistedAcctData := &trackerdb.PersistedAccountData{Addr: addr, Rowid: rowid}
+				persistedAcctData := &trackerdb.PersistedAccountData{Addr: addr, Ref: ref}


it looks like more like account id, not reference

Its intended name and use is as an opaque reference to the database entry.
You can't do much with except to give it back to the database engine.
Having an opaque identifier is a bit odd. This has no meaning outside of the engine it came out of.

In the case of the account, which is the identifier, the address? or the sqlite specific rowid?

ledger/acctdeltas_test.go

ledger/store/trackerdb/sqlitedriver/accountsV2.go

ledger/store/trackerdb/sqlitedriver/sql.go

algorandskiy · 2023-03-07T16:14:39Z

ledger/acctdeltas.go

+		if delta.oldResource.AcctRef != nil {
+			acctRef = delta.oldResource.AcctRef
+		} else if acctRef, ok = knownAddresses[addr]; !ok {
+			acctRef, err = arw.LookupAccountRowID(addr)


nit: LookupAccountRowID -> LookupAccountByRef
maybe a separate PR

algorandskiy · 2023-03-07T16:15:34Z

ledger/acctdeltas.go

 				err = protocol.Decode(resDataBuf, &persistedResData.Data)
 				if err != nil {
 					return err
 				}
 				a.updateOld(missIdx, persistedResData)
 			} else {
-				err = fmt.Errorf("empty resource record: addrid=%d, aidx=%d", addrid, aidx)
+				err = fmt.Errorf("empty resource record: addrid=%d, aidx=%d", acctRef, aidx)


what it will print out? acctRef is an interface, %d expects a number

func main() { var acctRef AccountRef acctRef = ar{123} fmt.Println("ln", acctRef) fmt.Printf("as num %d", acctRef) }

gives

ln {123} as num {123}

So the output changes but not critical looks like

algorandskiy · 2023-03-07T19:45:46Z

Please followup with with a PR addrid=%d -> addrid=%s + String() method in the interface

icorderi added Team Carbon-11 Skip-Release-Notes labels Mar 2, 2023

icorderi self-assigned this Mar 2, 2023

icorderi force-pushed the refactor/rowid branch 2 times, most recently from f923d8e to cfaa941 Compare March 2, 2023 19:18

algorandskiy requested review from jannotti, cce and algorandskiy March 2, 2023 20:49

AlgoAxel reviewed Mar 2, 2023

View reviewed changes

icorderi added 3 commits March 2, 2023 21:36

refactor: remove rowid/addrid from the interface

255f0cd

fixes

fe70066

fixed test

2de9076

icorderi force-pushed the refactor/rowid branch from 87aad0a to 2de9076 Compare March 3, 2023 00:36

icorderi added 2 commits March 3, 2023 10:36

rebase fix

26b5dd4

changed comments around rowid

a880026

icorderi removed request for jannotti and cce March 3, 2023 13:39

make the ref interfaces excplicit

b6cd3fe

fix tests

c2b2ce6

AlgoAxel reviewed Mar 3, 2023

View reviewed changes

refactor: add marker to CreatableRef interface

cad5181

AlgoAxel previously approved these changes Mar 3, 2023

View reviewed changes

algorandskiy reviewed Mar 3, 2023

View reviewed changes

icorderi added 3 commits March 6, 2023 11:58

remove commented out asserts

8082e86

nicer errors

a465b5b

crosscheck online account ids

f7b2428

icorderi dismissed AlgoAxel’s stale review via f7b2428 March 6, 2023 15:34

better xcheck

845431a

errors instead of panic on nil ref

ca26aac

algorandskiy approved these changes Mar 7, 2023

View reviewed changes

AlgoAxel approved these changes Mar 7, 2023

View reviewed changes

algorandskiy changed the title ~~refactor: Remove "rowid" from the public interface~~ ledger: remove "rowid" from the public interface Mar 7, 2023

algorandskiy merged commit 7fb8ebd into algorand:master Mar 7, 2023

algorandskiy mentioned this pull request Mar 9, 2023

ledger: fix error shadowing in onlineAccountsNewRoundImpl #5188

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ledger: remove "rowid" from the public interface #5177

ledger: remove "rowid" from the public interface #5177

icorderi commented Mar 2, 2023

AlgoAxel left a comment

AlgoAxel Mar 2, 2023

codecov bot commented Mar 2, 2023 •

edited

Loading

icorderi commented Mar 3, 2023

AlgoAxel Mar 3, 2023

algorandskiy Mar 3, 2023

icorderi Mar 6, 2023

algorandskiy Mar 7, 2023

algorandskiy Mar 7, 2023

algorandskiy Mar 7, 2023

algorandskiy commented Mar 7, 2023

ledger: remove "rowid" from the public interface #5177

ledger: remove "rowid" from the public interface #5177

Conversation

icorderi commented Mar 2, 2023

Summary

Test Plan

AlgoAxel left a comment

Choose a reason for hiding this comment

AlgoAxel Mar 2, 2023

Choose a reason for hiding this comment

codecov bot commented Mar 2, 2023 • edited Loading

Codecov Report

icorderi commented Mar 3, 2023

AlgoAxel Mar 3, 2023

Choose a reason for hiding this comment

algorandskiy Mar 3, 2023

Choose a reason for hiding this comment

icorderi Mar 6, 2023

Choose a reason for hiding this comment

algorandskiy Mar 7, 2023

Choose a reason for hiding this comment

algorandskiy Mar 7, 2023

Choose a reason for hiding this comment

algorandskiy Mar 7, 2023

Choose a reason for hiding this comment

algorandskiy commented Mar 7, 2023

codecov bot commented Mar 2, 2023 •

edited

Loading