Add new getTransactions endpoint and database backend for transacti…

…ons (#174) * Increase `getTransaction` window via ingestion into a local database (#141) * Add common interface between in-memory and on-disk transaction storage * Revert "Add common interface between in-memory and on-disk transaction storage" This reverts commit 3196ade. * Add initial implementation of DB-backed tx store * Drop all references to the in-memory transaction store * Actually filter for the tx hash you want 🤦 * Drop remaining reference to transaction store * Update stellar/go to latest master * Add a bunch of logging/verbose errors, another column to DB * Propogate contexts, fix all bugs; e2e flow works! * Add metric for tx ingestion * Split TransactionHandler into separate readers and writers * Move tx reads into their own isolated, read-only db tx * Add in-memory cache for ledger ranges * Remove cache: read/write are separate so it doesn't work * Add trimming of transactions table based on retention window * Tests work (but dont pass)! Plus some fixes based on test cases 👍 * Tidy up the go.mod file * Fixup the way ledger ranges are handled * Add prometheus metrics back * Drop columns and logging that aren't useful * Incorporate errors into ledger range retrieval * Return instead of log errors in getTransaction * Fixup tests: an empty db isn't a fetch error * Make 24hr the new default transaction retention window, fixup errors * Expect the correct retention window in tests * Use raw byte hash over hex for 1/2 the column space * Rename interfaces to avoid overloading 'Transaction' term * PR feedback: variable rename * Separate tx parsing into its own function * Modify queries to not use transactions * Pass around the logger rather than using the global * Fixup all of the tests to pass a logger * Use subqueries to get around () sqlite limitations * Use .Select() to simplify querying * Add db unit test but it's failing :( * Prefer custom error over EOF or DB nonsense * Prepare randoms in advance, writer for the ledger as well * Pass daemon to db to hook in metrics * Check another error in e2e test * Feedback: use no-op interface, drop metric * Refactor method tests to use 'fake' tx/ledger backend * Simplify code a bit * Fixup mocking interface * Move test function back, fixup metrics and tests * Add `getTransactions` endpoint (#136) * Add cursor for getTransactions * Add validation of pagination and range * implement getTransactions handler - 1 * Add getLedgers * Read a single ledger instead of getting all ledgers * Add ParseCursor method for tx cursor * Cursor changes - 1 * Cursor changes - 2 * Cursor changes - 3 * revert go-mod changes * Use reader.Seek() * Use reader.Seek() - 2 * Add config-options for tx pagination limits * Fix failing cursor test * Go mod changes * Go mod changes - 2 * Add unittests - 1 * Update go.mod and go.sum * Add integration tests * Update go.mod and go.sum * Add ledgerSeq to error string * Add docstrings * Change transactions limits * add defensive check for error other than EOF * add defensive check for error other than EOF - 2 * Change ledger sequence to uint32 * Add comments/docstrings * Include only cursor in response * Use toid instead of new cursor * Revert cursor changes * Return cursor as string in result * Refactor reader.Seek error handling * Small refactoring * Remove startLedger check * Remove endLedger * Import fix * Fix failing tests * Refactor to use new transaction db * Refactor mocks * Refactor unittests for using the new db changes * Refactor integration test * Add config vars for max requests and request duration * Fix failing test * Use transactionInfo struct instead of db.Transactions * Start indexing from 1 instead of 0 for toid * Operation index from 1 * Add lines to make sure structs implement interfaces * Remove omitempty * rename test func * make txInfo struct public and convert string cursor to int * Use sendSuccessfulTransaction helper func * Convert cursor to string in request * Change jrpc response codes * Change ledger close meta code to invalid-params * Revert back to InvalidParams error for reader.Read() * Refactor if-else statement * Refactor if-else statement - 2 * Add documentation on opting into the new db backend (#182) * Refactor getFeeStats changes to add db changes * Remove code making `GetLedgerRange` conform to interface (#186) It's unnecessary because we aren't using that interface anymore. * Add changelog entry for `getTransactions` (#183) * Add more changelog details on new endpoint * Fixup spacing --------- Co-authored-by: George <Shaptic@users.noreply.github.com> Co-authored-by: George Kudrayvtsev <george@stellar.org>
stellar · May 23, 2024 · c818bb8 · c818bb8
1 parent f15ad4a
commit c818bb8
Show file tree

Hide file tree

Showing 33 changed files with 1,837 additions and 965 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,7 +1,44 @@
 # Changelog
 
 ## Unreleased
-n/a
+
+### Added
+* Transactions will now be stored in a database rather than in memory ([#174](https://github.com/stellar/soroban-rpc/pull/174)).
+
+You can opt-in to longer transaction retention by setting `--transaction-retention-window` / `TRANSACTION_RETENTION_WINDOW` to a higher number of ledgers. This will also retain corresponding number of ledgers in the database. Keep in mind, of course, that this will cause an increase in disk usage for the growing database.
+
+* There is a new `getTransactions` endpoint with the following API ([#136](https://github.com/stellar/soroban-rpc/pull/136)):
+
+```typescript
+interface Request {
+  startLedger: number; // uint32
+  pagination?: {
+    cursor?: string;
+    limit?:  number; // uint
+  }
+}
+
+interface Response {
+  transactions: Transaction[];        // see below
+  latestLedger: number;               // uint32
+  latestLedgerCloseTimestamp: number; // int64
+  oldestLedger: number;               // uint32
+  oldestLedgerCloseTimestamp: number; // int64
+  cursor: string;
+}
+
+interface Transaction {
+  status: boolean;          // whether or not the transaction succeeded
+  applicationOrder: number; // int32, index of the transaction in the ledger
+  feeBump: boolean;         // if it's a fee-bump transaction
+  envelopeXdr: string;      // TransactionEnvelope XDR
+  resultXdr: string;        // TransactionResult XDR
+  resultMetaXdr: string;    // TransactionMeta XDR
+  ledger: number;           // uint32, ledger sequence with this transaction
+  createdAt: int64;         // int64, UNIX timestamp the transaction's inclusion
+  diagnosticEventsXdr?: string[]; // if failed, DiagnosticEvent XDRs
+}
+```
 
 
 ## [v21.2.0](https://github.com/stellar/soroban-rpc/compare/v21.1.0...v21.2.0)

diff --git a/cmd/soroban-rpc/internal/config/config.go b/cmd/soroban-rpc/internal/config/config.go
@@ -25,6 +25,7 @@ type Config struct {
 	CheckpointFrequency                            uint32
 	CoreRequestTimeout                             time.Duration
 	DefaultEventsLimit                             uint
+	DefaultTransactionsLimit                       uint
 	EventLedgerRetentionWindow                     uint32
 	FriendbotURL                                   string
 	HistoryArchiveURLs                             []string
@@ -33,6 +34,7 @@ type Config struct {
 	LogFormat                                      LogFormat
 	LogLevel                                       logrus.Level
 	MaxEventsLimit                                 uint
+	MaxTransactionsLimit                           uint
 	MaxHealthyLedgerLatency                        time.Duration
 	NetworkPassphrase                              string
 	PreflightWorkerCount                           uint
@@ -50,6 +52,7 @@ type Config struct {
 	RequestBacklogGetLatestLedgerQueueLimit        uint
 	RequestBacklogGetLedgerEntriesQueueLimit       uint
 	RequestBacklogGetTransactionQueueLimit         uint
+	RequestBacklogGetTransactionsQueueLimit        uint
 	RequestBacklogSendTransactionQueueLimit        uint
 	RequestBacklogSimulateTransactionQueueLimit    uint
 	RequestBacklogGetFeeStatsTransactionQueueLimit uint
@@ -62,6 +65,7 @@ type Config struct {
 	MaxGetLatestLedgerExecutionDuration            time.Duration
 	MaxGetLedgerEntriesExecutionDuration           time.Duration
 	MaxGetTransactionExecutionDuration             time.Duration
+	MaxGetTransactionsExecutionDuration            time.Duration
 	MaxSendTransactionExecutionDuration            time.Duration
 	MaxSimulateTransactionExecutionDuration        time.Duration
 	MaxGetFeeStatsExecutionDuration                time.Duration

diff --git a/cmd/soroban-rpc/internal/config/options.go b/cmd/soroban-rpc/internal/config/options.go
@@ -211,18 +211,22 @@ func (cfg *Config) options() ConfigOptions {
 		},
 		{
 			Name: "event-retention-window",
-			Usage: fmt.Sprintf("configures the event retention window expressed in number of ledgers,"+
-				" the default value is %d which corresponds to about 24 hours of history", ledgerbucketwindow.DefaultEventLedgerRetentionWindow),
+			Usage: fmt.Sprintf(
+				"configures the event retention window expressed in number of ledgers,"+
+					" the default value is %d which corresponds to about 24 hours of history",
+				ledgerbucketwindow.DefaultEventLedgerRetentionWindow),
 			ConfigKey:    &cfg.EventLedgerRetentionWindow,
 			DefaultValue: uint32(ledgerbucketwindow.DefaultEventLedgerRetentionWindow),
 			Validate:     positive,
 		},
 		{
 			Name: "transaction-retention-window",
-			Usage: "configures the transaction retention window expressed in number of ledgers," +
-				" the default value is 1440 which corresponds to about 2 hours of history",
+			Usage: fmt.Sprintf(
+				"configures the transaction retention window expressed in number of ledgers,"+
+					" the default value is %d which corresponds to about 24 hours of history",
+				ledgerbucketwindow.OneDayOfLedgers),
 			ConfigKey:    &cfg.TransactionLedgerRetentionWindow,
-			DefaultValue: uint32(1440),
+			DefaultValue: uint32(ledgerbucketwindow.OneDayOfLedgers),
 			Validate:     positive,
 		},
 		{
@@ -261,6 +265,28 @@ func (cfg *Config) options() ConfigOptions {
 				return nil
 			},
 		},
+		{
+			Name:         "max-transactions-limit",
+			Usage:        "Maximum amount of transactions allowed in a single getTransactions response",
+			ConfigKey:    &cfg.MaxTransactionsLimit,
+			DefaultValue: uint(200),
+		},
+		{
+			Name:         "default-transactions-limit",
+			Usage:        "Default cap on the amount of transactions included in a single getTransactions response",
+			ConfigKey:    &cfg.DefaultTransactionsLimit,
+			DefaultValue: uint(50),
+			Validate: func(co *ConfigOption) error {
+				if cfg.DefaultTransactionsLimit > cfg.MaxTransactionsLimit {
+					return fmt.Errorf(
+						"default-transactions-limit (%v) cannot exceed max-transactions-limit (%v)",
+						cfg.DefaultTransactionsLimit,
+						cfg.MaxTransactionsLimit,
+					)
+				}
+				return nil
+			},
+		},
 		{
 			Name: "max-healthy-ledger-latency",
 			Usage: "maximum ledger latency (i.e. time elapsed since the last known ledger closing time) considered to be healthy" +
@@ -344,6 +370,13 @@ func (cfg *Config) options() ConfigOptions {
 			DefaultValue: uint(1000),
 			Validate:     positive,
 		},
+		{
+			TomlKey:      strutils.KebabToConstantCase("request-backlog-get-transactions-queue-limit"),
+			Usage:        "Maximum number of outstanding GetTransactions requests",
+			ConfigKey:    &cfg.RequestBacklogGetTransactionsQueueLimit,
+			DefaultValue: uint(1000),
+			Validate:     positive,
+		},
 		{
 			TomlKey:      strutils.KebabToConstantCase("request-backlog-send-transaction-queue-limit"),
 			Usage:        "Maximum number of outstanding SendTransaction requests",
@@ -419,6 +452,12 @@ func (cfg *Config) options() ConfigOptions {
 			ConfigKey:    &cfg.MaxGetTransactionExecutionDuration,
 			DefaultValue: 5 * time.Second,
 		},
+		{
+			TomlKey:      strutils.KebabToConstantCase("max-get-transactions-execution-duration"),
+			Usage:        "The maximum duration of time allowed for processing a getTransactions request. When that time elapses, the rpc server would return -32001 and abort the request's execution",
+			ConfigKey:    &cfg.MaxGetTransactionsExecutionDuration,
+			DefaultValue: 5 * time.Second,
+		},
 		{
 			TomlKey:      strutils.KebabToConstantCase("max-send-transaction-execution-duration"),
 			Usage:        "The maximum duration of time allowed for processing a sendTransaction request. When that time elapses, the rpc server would return -32001 and abort the request's execution",

diff --git a/cmd/soroban-rpc/internal/daemon/daemon.go b/cmd/soroban-rpc/internal/daemon/daemon.go
@@ -19,6 +19,7 @@ import (
 	"github.com/stellar/go/ingest/ledgerbackend"
 	supporthttp "github.com/stellar/go/support/http"
 	supportlog "github.com/stellar/go/support/log"
+	"github.com/stellar/go/support/ordered"
 	"github.com/stellar/go/support/storage"
 	"github.com/stellar/go/xdr"
 
@@ -28,8 +29,8 @@ import (
 	"github.com/stellar/soroban-rpc/cmd/soroban-rpc/internal/events"
 	"github.com/stellar/soroban-rpc/cmd/soroban-rpc/internal/feewindow"
 	"github.com/stellar/soroban-rpc/cmd/soroban-rpc/internal/ingest"
+	"github.com/stellar/soroban-rpc/cmd/soroban-rpc/internal/ledgerbucketwindow"
 	"github.com/stellar/soroban-rpc/cmd/soroban-rpc/internal/preflight"
-	"github.com/stellar/soroban-rpc/cmd/soroban-rpc/internal/transactions"
 	"github.com/stellar/soroban-rpc/cmd/soroban-rpc/internal/util"
 )
 
@@ -135,7 +136,6 @@ func newCaptiveCore(cfg *config.Config, logger *supportlog.Entry) (*ledgerbacken
 func MustNew(cfg *config.Config) *Daemon {
 	logger := supportlog.New()
 	logger.SetLevel(cfg.LogLevel)
-
 	if cfg.LogFormat == config.LogFormatJSON {
 		logger.UseJSONFormatter()
 	}
@@ -151,7 +151,7 @@ func MustNew(cfg *config.Config) *Daemon {
 	}
 
 	if len(cfg.HistoryArchiveURLs) == 0 {
-		logger.Fatal("no history archives url were provided")
+		logger.Fatal("no history archives URLs were provided")
 	}
 
 	historyArchive, err := historyarchive.NewArchivePool(
@@ -192,11 +192,6 @@ func MustNew(cfg *config.Config) *Daemon {
 		cfg.NetworkPassphrase,
 		cfg.EventLedgerRetentionWindow,
 	)
-	transactionStore := transactions.NewMemoryStore(
-		daemon,
-		cfg.NetworkPassphrase,
-		cfg.TransactionLedgerRetentionWindow,
-	)
 	feewindows := feewindow.NewFeeWindows(cfg.ClassicFeeStatsLedgerRetentionWindow, cfg.SorobanFeeStatsLedgerRetentionWindow, cfg.NetworkPassphrase)
 
 	// initialize the stores using what was on the DB
@@ -222,9 +217,6 @@ func MustNew(cfg *config.Config) *Daemon {
 		if err := eventStore.IngestEvents(txmeta); err != nil {
 			logger.WithError(err).Fatal("could not initialize event memory store")
 		}
-		if err := transactionStore.IngestTransactions(txmeta); err != nil {
-			logger.WithError(err).Fatal("could not initialize transaction memory store")
-		}
 		if err := feewindows.IngestFees(txmeta); err != nil {
 			logger.WithError(err).Fatal("could not initialize fee stats")
 		}
@@ -242,12 +234,27 @@ func MustNew(cfg *config.Config) *Daemon {
 	onIngestionRetry := func(err error, dur time.Duration) {
 		logger.WithError(err).Error("could not run ingestion. Retrying")
 	}
-	maxRetentionWindow := max(cfg.EventLedgerRetentionWindow, cfg.TransactionLedgerRetentionWindow, cfg.ClassicFeeStatsLedgerRetentionWindow, cfg.SorobanFeeStatsLedgerRetentionWindow)
+
+	// Take the larger of (event retention, tx retention) and then the smaller
+	// of (tx retention, default event retention) if event retention wasn't
+	// specified, for some reason...?
+	maxRetentionWindow := ordered.Max(cfg.EventLedgerRetentionWindow, cfg.TransactionLedgerRetentionWindow)
+	if cfg.EventLedgerRetentionWindow <= 0 {
+		maxRetentionWindow = ordered.Min(
+			maxRetentionWindow,
+			ledgerbucketwindow.DefaultEventLedgerRetentionWindow)
+	}
 	ingestService := ingest.NewService(ingest.Config{
-		Logger:            logger,
-		DB:                db.NewReadWriter(dbConn, maxLedgerEntryWriteBatchSize, maxRetentionWindow),
+		Logger: logger,
+		DB: db.NewReadWriter(
+			logger,
+			dbConn,
+			daemon,
+			maxLedgerEntryWriteBatchSize,
+			maxRetentionWindow,
+			cfg.NetworkPassphrase,
+		),
 		EventStore:        eventStore,
-		TransactionStore:  transactionStore,
 		NetworkPassPhrase: cfg.NetworkPassphrase,
 		Archive:           historyArchive,
 		LedgerBackend:     core,
@@ -271,11 +278,11 @@ func MustNew(cfg *config.Config) *Daemon {
 	jsonRPCHandler := internal.NewJSONRPCHandler(cfg, internal.HandlerParams{
 		Daemon:            daemon,
 		EventStore:        eventStore,
-		TransactionStore:  transactionStore,
 		FeeStatWindows:    feewindows,
 		Logger:            logger,
 		LedgerReader:      db.NewLedgerReader(dbConn),
 		LedgerEntryReader: db.NewLedgerEntryReader(dbConn),
+		TransactionReader: db.NewTransactionReader(logger, dbConn, cfg.NetworkPassphrase),
 		PreflightGetter:   preflightWorkerPool,
 	})
 

diff --git a/cmd/soroban-rpc/internal/daemon/interfaces/noOpDaemon.go b/cmd/soroban-rpc/internal/daemon/interfaces/noOpDaemon.go
@@ -24,7 +24,7 @@ func MakeNoOpDeamon() *noOpDaemon {
 }
 
 func (d *noOpDaemon) MetricsRegistry() *prometheus.Registry {
-	return d.metricsRegistry
+	return prometheus.NewRegistry() // so that you can register metrics many times
 }
 
 func (d *noOpDaemon) MetricsNamespace() string {