Ledger init status #1058

shiqizng · 2022-06-23T20:40:29Z

Summary

This PR adds status logging for ledger init and stops node if it's running when indexer receives SIGTERM or SIGINT.

Test Plan

regression testing, run daemon.

codecov · 2022-06-23T20:59:29Z

Codecov Report

❗ No coverage uploaded for pull request base (localledger/integration@5584c6d). Click here to learn what that means.
The diff coverage is n/a.

@@                    Coverage Diff                     @@
##             localledger/integration    #1058   +/-   ##
==========================================================
  Coverage                           ?   58.74%           
==========================================================
  Files                              ?       49           
  Lines                              ?     8780           
  Branches                           ?        0           
==========================================================
  Hits                               ?     5158           
  Misses                             ?     3139           
  Partials                           ?      483

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5584c6d...762a17e. Read the comment docs.

processor/blockprocessor/initialize.go

shiqizng · 2022-06-24T14:15:13Z

processor/blockprocessor/block_processor.go

 		if err != nil {
 			return &blockProcessor{}, fmt.Errorf("MakeProcessorWithCatchup() slow catchup err: %w", err)
 		}
 	}
-	return MakeProcessor(logger, genesis, nextDBRound, opts.AlgodDataDir, handler)
+	return MakeProcessor(logger, genesis, nextDBRound, opts.IndexerDatadir, handler)


I think it should be the indexer directory?

processor/blockprocessor/initialize.go

winder · 2022-06-27T18:23:38Z

processor/blockprocessor/initialize.go

-			os.Exit(1)
-		}()
-	}
+func InitializeLedgerSimple(ctx context.Context, cf context.CancelFunc, logger *log.Logger, round uint64, opts *idb.IndexerDbOptions) error {


Is something missing here? How does this exit when it's all caught up?

cancel() is called here

indexer/processor/blockprocessor/initialize.go

Lines 162 to 165 in eb2ac87

if uint64(block.Block.Round()) == dbRound {

// migration completes

cancel()

} else {

Thanks, I missed that.

I think this would cause problems because the context is going to be cancelled when we're finished initializing. You can wrap the context with another WithCancel. I had to try it, but this seems to work as expected: https://go.dev/play/p/l1S-LqG0MkR

processor/blockprocessor/initialize_test.go

winder · 2022-06-28T13:27:03Z

processor/blockprocessor/initialize.go

@@ -26,21 +24,9 @@ import (
 )

 // InitializeLedgerSimple executes the migration core functionality.
-func InitializeLedgerSimple(logger *log.Logger, round uint64, opts *idb.IndexerDbOptions) error {
-	ctx, cf := context.WithCancel(context.Background())
+func InitializeLedgerSimple(ctx context.Context, logger *log.Logger, round uint64, opts *idb.IndexerDbOptions) error {


Is this function supposed to block until initialization is complete? It looks like it starts fetcher and then runs in the background.

bot.Run is blocking, nevermind!

* Local Ledger (#1011) * integrate block processor * Local Ledger Deployment (#1013) * add simple local ledger migration * add deleted opts * fast catchup (#1023) * add fast catchup * Localledger merge (#1036) * return empty lists from fetchApplications and fetchAppLocalStates (#1010) * Update model to converge with algod (#1005) * New Feature: Adds Data Directory Support (#1012) - Updates the go-algorand submodule hash to point to rel/beta - Moves the cpu profiling file, pid file and indexer configuration file to be options of only the daemon sub-command - Changes os.Exit() to be a panic with a special handler. This is so that defer's are handled instead of being ignored. - Detects auto-loading configuration files in the data directory and issues errors if equivalent command line arguments are supplied. - Updates the README with instructions on how to use the auto-loading configuration files and the data directory. * Update mockery version Co-authored-by: erer1243 <1377477+erer1243@users.noreply.github.com> Co-authored-by: AlgoStephenAkiki <85183435+AlgoStephenAkiki@users.noreply.github.com> * recovery scenario (#1024) * handle ledger recovery scenario * refactor create genesis block (#1026) * refactor create genesis block * Adds Local Ledger Readme (#1035) * Adds Local Ledger Readme Resolves #4109 Starts Readme docs * Update docs/LocalLedger.md Co-authored-by: Will Winder <wwinder.unh@gmail.com> * Update docs/LocalLedger.md Co-authored-by: Will Winder <wwinder.unh@gmail.com> * Update docs/LocalLedger.md Co-authored-by: Will Winder <wwinder.unh@gmail.com> * Removed troubleshooting section Co-authored-by: Will Winder <wwinder.unh@gmail.com> * update ledger file path and migration (#1042) * LocalLedger Refactoring + Catchpoint Service (#1049) Part 1 cleanup genesis file access. put node catchup into a function that can be swapped out with the catchup service. pass the indexer logger into the block processor. move open ledger into a util function, and move the initial state util function into a new ledger util file. add initial catchupservice implementation. move ledger init from daemon.go to constructor. Merge multiple read genesis functions. Part 2 Merge local_ledger migration package into blockprocessor. Rename Migration to Initialize Use logger in catchup service catchup Part 3 Update submodule and use NewWrappedLogger. Make util.CreateInitState private * build: merge develop into localledger/integration (#1062) * Ledger init status (#1058) * Generate an error if the catchpoint is not valid for initialization. (#1075) * Use main logger in handler and fetcher. (#1077) * Switch from fullNode catchup to catchpoint catchup service. (#1076) * Refactor daemon, add more tests (#1039) Refactors daemon cmd into separate, testable pieces. * Merge develop into localledger/integration (#1083) * Misc Local Ledger cleanup (#1086) * Update processor/blockprocessor/initialize.go Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com> * commit * fix function call args * RFC-0001: Rfc 0001 impl (#1069) Adds an Exporter interface and a noop exporter implementation with factory methods for construction * Fix test errors * Add/fix tests * Add postgresql_exporter tests * Update config loading * Change BlockExportData to pointers * Move and rename ExportData * Add Empty func to BlockData * Add comment Co-authored-by: shiqizng <80276844+shiqizng@users.noreply.github.com> Co-authored-by: shiqi.zheng@algorand.com <shiqi.zheng@algorand.com> Co-authored-by: erer1243 <1377477+erer1243@users.noreply.github.com> Co-authored-by: AlgoStephenAkiki <85183435+AlgoStephenAkiki@users.noreply.github.com> Co-authored-by: Will Winder <wwinder.unh@gmail.com> Co-authored-by: Zeph Grunschlag <tzaffi@users.noreply.github.com>

log init status and stop node when init is aborted

ecdc289

shiqizng self-assigned this Jun 23, 2022

shiqizng added the Team Lamprey label Jun 23, 2022

shiqizng requested review from winder, AlgoStephenAkiki and Eric-Warehime June 23, 2022 20:40

fix lint

c12d84f

winder reviewed Jun 24, 2022

View reviewed changes

processor/blockprocessor/initialize.go Outdated Show resolved Hide resolved

AlgoStephenAkiki approved these changes Jun 24, 2022

View reviewed changes

pass parent ctx

2dad7da

shiqizng commented Jun 24, 2022

View reviewed changes

shiqizng requested a review from winder June 24, 2022 14:15

winder reviewed Jun 25, 2022

View reviewed changes

processor/blockprocessor/initialize.go Outdated Show resolved Hide resolved

winder reviewed Jun 25, 2022

View reviewed changes

processor/blockprocessor/initialize.go Outdated Show resolved Hide resolved

chaihoang removed the Team Lamprey label Jun 27, 2022

shiqizng added 2 commits June 27, 2022 12:17

contexts and bug fix

a465041

fix failing test

eb2ac87

shiqizng requested a review from winder June 27, 2022 17:15

winder reviewed Jun 27, 2022

View reviewed changes

processor/blockprocessor/initialize_test.go Outdated Show resolved Hide resolved

shiqizng added 3 commits June 27, 2022 17:39

context fix

8f6e0bf

more fix

15fc48c

more context fix

762a17e

winder reviewed Jun 28, 2022

View reviewed changes

winder approved these changes Jun 28, 2022

View reviewed changes

winder merged commit a1a08b2 into localledger/integration Jun 28, 2022

winder deleted the shiqi/logging branch June 28, 2022 18:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ledger init status #1058

Ledger init status #1058

shiqizng commented Jun 23, 2022

codecov bot commented Jun 23, 2022 •

edited

shiqizng Jun 24, 2022

winder Jun 27, 2022

shiqizng Jun 27, 2022

winder Jun 27, 2022

shiqizng Jun 27, 2022

winder Jun 28, 2022

winder Jun 28, 2022

	if uint64(block.Block.Round()) == dbRound {
	// migration completes
	cancel()
	} else {

Ledger init status #1058

Ledger init status #1058

Conversation

shiqizng commented Jun 23, 2022

Summary

Test Plan

codecov bot commented Jun 23, 2022 • edited

Codecov Report

shiqizng Jun 24, 2022

Choose a reason for hiding this comment

winder Jun 27, 2022

Choose a reason for hiding this comment

shiqizng Jun 27, 2022

Choose a reason for hiding this comment

winder Jun 27, 2022

Choose a reason for hiding this comment

shiqizng Jun 27, 2022

Choose a reason for hiding this comment

winder Jun 28, 2022

Choose a reason for hiding this comment

winder Jun 28, 2022

Choose a reason for hiding this comment

codecov bot commented Jun 23, 2022 •

edited