Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

services/horizon: Allow captive core to run with sqlite database #4092

Merged
merged 31 commits into from
Feb 17, 2022
Merged

services/horizon: Allow captive core to run with sqlite database #4092

merged 31 commits into from
Feb 17, 2022

Conversation

paulbellamy
Copy link
Contributor

@paulbellamy paulbellamy commented Nov 22, 2021 •
edited by sreuland

PR Checklist

PR Structure

  • This PR has reasonably narrow scope (if not, break it down into smaller PRs).
  • This PR avoids mixing refactoring changes with feature changes (split into two PRs
    otherwise).
  • This PR's title starts with name of package that is most changed in the PR, ex.
    services/friendbot, or all or doc if the changes are broad or impact many
    packages.

Thoroughness

  • This PR adds tests for the most critical parts of the new functionality or fixes.
  • I've updated any docs (developer docs, .md
    files, etc... affected by this change). Take a look in the docs folder for a given service,
    like this one.

Release planning

  • I've updated the relevant CHANGELOG (here for Horizon) if
    needed with deprecations, added features, breaking changes, and DB schema changes.
  • I've decided if this PR requires a new major/minor version according to
    semver, or if it's mainly a patch change. The PR is targeted at the next
    release branch if it's not a patch change.

What

Set default captive core invocations to use the sqlite on-disk database rather than in-memory.

This will change the existing Horizon run command with params for ingestion enabled and captive core enabled to use the on-disk database in core for ledger ingestion, such as this:
horizon --db-url <db_url> --ingest --captive-core-config-path <path_to_core_conf_file> --stellar-core-binary-path /usr/local/bin/stellar-core --history-archive-urls <archive_urls> --network-passphrase "<passphrase>"

Why

Help core use less memory, and be more efficient for large ledger handling.

Fixes #4038

Known limitations

--in-memory is required when we launch core with --start-at-ledger, so it can only use sqlite sometimes.

TODO

  • Run on staging for a bit, and check the impact.
  • Decide if we want to add a horizon --in-memory flag to force the old behaviour.

2opremio
2opremio reviewed Nov 22, 2021
View reviewed changes
ingest/ledgerbackend/stellar_core_runner.go Outdated
"--start-at-ledger", fmt.Sprintf("%d", from),
"--start-at-hash", hash,
"--metadata-output-stream", r.getPipeName(),
"--in-memory",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't createCmd() responsible to adding --in-memory?

@paulbellamy paulbellamy force-pushed the 4038/captive-core-sqlite branch 3 times, most recently from 823eb1b to 197f388 Compare December 7, 2021 17:25
@paulbellamy
Copy link
Contributor Author

paulbellamy commented Jan 10, 2022 •
edited

stellar/stellar-core#3306 should help with disk usage leak when reingesting, in the meantime we'll have to pick a shorter ledger ranges and periodically restart captive-core (in catchup mode), to clean up the leaked files. So, we need a heuristic to make reingestion ranges shorter. Maybe every x blocks? We could check available disk and base it on that 😂 seems over-engineered, tho.

@paulbellamy paulbellamy force-pushed the 4038/captive-core-sqlite branch 3 times, most recently from fcd3df5 to d2c2cc5 Compare January 14, 2022 15:15
@paulbellamy paulbellamy force-pushed the 4038/captive-core-sqlite branch 3 times, most recently from 14b94cb to e532513 Compare January 19, 2022 17:04
@sreuland
Copy link
Contributor

@tamirms

Will captive core work without any issues if a postgres DB is configured in the toml file?

it should be supported also, the DATABASE cfg/toml parameter is doc'd to accept both db driver urls, and I checked on core team chat, they confirmed it should work, but, this configuration to postgres hasn't been tested live, will have to try running that still.

so, horizon validation effort on DATABASE should probably keep it simple best effort, such as if present it either needs to start with sqlite:// or postgres:// as the leading value.

@sreuland
Copy link
Contributor

we confirmed on further discussion in slack thread, to limit the DATABASE value for captive core config to just sqlite:// db connection string, will add validation for that.

@sreuland
#4038: use --in-memory for cc ingest by default
a0ce745 
only use on disk with new captive-core-use-external-storage=true cli param
@sreuland sreuland requested review from tamirms and a team February 16, 2022 19:43
tamirms
tamirms approved these changes Feb 17, 2022
View reviewed changes
Copy link
Contributor

@tamirms tamirms left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good!

@sreuland sreuland merged commit b7fde58 into master Feb 17, 2022
@sreuland sreuland mentioned this pull request Feb 25, 2022
Open
Shaptic added a commit that referenced this pull request May 24, 2022
@Shaptic @stfung77
@leighmcculloch @2opremio @bartekn @tamirms @graydon @satyamz @erika-sdf @iateadonut @sreuland @MishraShivendra @jacekn
lighthorizon: Sync with upstream master branch (#4404)
4eeba9d 
* services/ticker: ingest assets optimizations (#4218)
* Add CHANGELOG entry for Horizon 2.14.0 release (#4208) (#4220)
* Make sure we test reingestion for all possible operations (#4231)
* services/horizon: Allow captive core to run with sqlite database (#4092)
* services/horizon: Release DB connection in /paths when no longer needed (#4228)
* services/horizon: Exclude trades with >10% rounding slippage from trade aggregations (#4178)
* all: staticcheck fixes (#4239)
* Migrate Horizon integration tests to GitHub Actions (#4242)
* Fix StreamAllLiquidityPools and StreamAllOffers (#4236)
* all: run builds and tests with go1.18rc1 (#4143)
* all: cache go module downloads and other build and test artifacts (#3727)
* services/horizon: Add LedgerHashStore to Captive-Core config (#4251)
* all: migrate the rest of the CircleCI jobs to GitHub Actions (#4250)
* horizon: Fix GitHub action problem with verify-range push in master (#4253)
* all: fix ci ref_protected check for caching (#4254)
* Switch over from CircleCI to GitHub A tions (#4256)
* all: [GitHub actions] Reset the module and build cache in master/protected (#4266)
* Forgot to add sudo in #4266 (#4270)
* all: More go-setup github action fixes (#4274)
* xdr: add instructions for generating xdr (#4280)
* services/ticker: cache tomls during scraping (#4286)
* services/ticker: use log fields during asset ingestion (#4288)
* services/ticker: reduce size of toml cache in memory (#4289)
* historyarchive: add --skip-optional flag (#3906)
* all: Add Protocol 19 XDR and update StrKey to support Signed Payloads (#4279)
* Replace keybase with publicnode in the stellar core config (#4291)
* Fix captive core tests to write to /tmp, instead of polluting the repo (#4296)
* all: remove go1.16 add go1.18 (#4284)
* Rename methods and functions in submission system (#4298)
* PR feedback (#4300)
* Support new account fields for protocol-19. (#4294)
* xdr, keypair: Add helpers to create CAP-40 decorated signatures (#4302)
* services/horizon: Update txsub queue to account for new CAP-21 preconditions (#4301)
* Uncomment StateVerifier test that generates account v3 extensions now that they are implemented. (#4304)
* txnbuild: Add support for new CAP-21 preconditions. (#4303)
* services/horizon: Support new CAP-21 transaction conditions (#4297)
* txnbuild: Complete rename, avoid using XDR types in `TransactionParams`. (#4307)
* all: Update Protocol 19 XDR to the latest (#4308)
* services/horizon: Add a rate limit for path finding requests. (#4310)
* clients/horizonclient: fix multi-parameter url for claimable balance query (#4248)
* all: Fix Horizon integration tests (#4292)
* horizon: Fix integration tests (#4314)
* horizon: Set up protocol 19 integration tests infrastructure (#4312)
* all: Change outdated CircleCI build badge (#4324)
* horizon: Test new protocol 19 account fields (#4322)
* all: update staticcheck to 2022.1 (#4326)
* all: remove go.list and related docs (#4328)
* horizon: Add transaction submission test for Protocol 19 (#4327)
* Horizon v2.16.1 CHANGELOG (#4333)
* Revert "Pin go versions temporarily" (#4338)
* services/horizon: Use `bigint` over `timestamp` to accommodate large years (#4337)
* xdr: Update xdrgen (#4341)
* services/horizon: Change `min_account_sequence_age` column from `bigint` to string (#4339)
* services/horizon: Bump stellar-core to v19.0.0rc1 for Horizon tests (#4345)
* services/horizon: expose supported protocol version on root endpoint (#4347)
* horizon: Small transaction submission refactoring (#4344)
* services/horizon: Pass through nil ExtraSigners to avoid nil pointer deref (#4349)
* doc: rename license file (#4350)
* all: upgrade dep github.com/valyala/fasthttp (#4351)
* services/horizon: Promote Stellar Core to v19.0.0 stable. (#4353)
* services/horizon/integration: Precondition edge cases and V18->19 upgrade boundary. (#4354)
* xdr: Synchronizes monorepo XDR with Stellar Core (#4355)
* services/horizon: Properly allow nullable Protocol 19 account fields (#4357)
* services/friendbot: include txhash in logs (#4359)
* services/horizon: Improve transaction precondition `omitempty` behavior (#4360)
* tools/horizon-cmp: Improve panic error message (#4365)
* services/horizon: Merge stable v2.17.0 back into master: (#4363)
* Use UNIX timestamps instead of RFC3339 strings for timebounds. (#4361)
* xdrgen: remove gemfile and rakefile to just use docker for the xdrgen (#4366)
* Conservatively limit the number of DB connections of integration tests (#4368)
* internal/integrations: db_test should drop test db instances when finished (#4185)
* GHA: Bump Core version to v19.0.1 in Horizon workflows. (#4378)
* services/horizon, clients/horizonclient: Allow filtering ingested transactions by account or asset. (#4277)
* Push stellar/ledger-state-diff images from Github actions (#4380)
* services/horizon: Fixes copy-paste typo in `--help` text (#4383)
* tools/alb-replay: Add new features to alb-replay (#4384)
* services/horizon: Optimize claimable balances query to limit records earlier (#4385)
* support/db, services/horizon/internal: Configure postgres client connection timeouts for read only db (#4390)
* Refactor trade aggregation query. (#4389)
* services/horizon/internal/db2/history: Implement StreamAllOffers using batches (#4397)
* Add flag to disable path finding endpoints (#4399)

Co-authored-by: stfung77 <stfung77@gmail.com>
Co-authored-by: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com>
Co-authored-by: Alfonso Acosta <alfonso@stellar.org>
Co-authored-by: Paul Bellamy <paul@stellar.org>
Co-authored-by: Bartek Nowotarski <bartek@nowotarski.info>
Co-authored-by: tamirms <tamirms@gmail.com>
Co-authored-by: Alfonso Acosta <fons@syntacticsugar.consulting>
Co-authored-by: Graydon Hoare <graydon@pobox.com>
Co-authored-by: Satyam Zode <5508956+satyamz@users.noreply.github.com>
Co-authored-by: Satyam Zode <satyamz@users.noreply.github.com>
Co-authored-by: erika-sdf <92893480+erika-sdf@users.noreply.github.com>
Co-authored-by: iateadonut <orienta.dan@gmail.com>
Co-authored-by: Shawn Reuland <shawn@stellar.org>
Co-authored-by: shawn <sreuland@users.noreply.github.com>
Co-authored-by: Shivendra Mishra <shivendra_mishra@live.com>
Co-authored-by: Jacek Nykis <jacek@stellar.org>
Co-authored-by: jacekn <jacekn@users.noreply.github.com>
@mollykarcher mollykarcher mentioned this pull request Feb 14, 2023
Closed
@sreuland sreuland mentioned this pull request Sep 11, 2023
Open
Shaptic added a commit that referenced this pull request Jan 9, 2024
@paulbellamy @bartekn
@tamirms @Shaptic @stfung77 @leighmcculloch @2opremio @graydon @satyamz @erika-sdf @iateadonut @sreuland @MishraShivendra @jacekn
exp: Add the Zenith (Light Horizon) prototype (#4352)
423fa1f 
* exp/lighthorizon: Add initial support for XDR serialization (#4369)
* exp/lighthorizon: Improve trie tests to avoid raw comparisons/outputs. (#4373)
* exp/lighthorizon: Add XDR marshalling for the `TrieNode` structure. (#4375)
* Add encoding stdlib interfaces
* lighthorizon: Sync with upstream master branch (#4404)
* services/ticker: ingest assets optimizations (#4218)
* Add CHANGELOG entry for Horizon 2.14.0 release (#4208) (#4220)
* Make sure we test reingestion for all possible operations (#4231)
* services/horizon: Allow captive core to run with sqlite database (#4092)
* services/horizon: Release DB connection in /paths when no longer needed (#4228)
* services/horizon: Exclude trades with >10% rounding slippage from trade aggregations (#4178)
* all: staticcheck fixes (#4239)
* Migrate Horizon integration tests to GitHub Actions (#4242)
* Fix StreamAllLiquidityPools and StreamAllOffers (#4236)
* all: run builds and tests with go1.18rc1 (#4143)
* all: cache go module downloads and other build and test artifacts (#3727)
* services/horizon: Add LedgerHashStore to Captive-Core config (#4251)
* all: migrate the rest of the CircleCI jobs to GitHub Actions (#4250)
* horizon: Fix GitHub action problem with verify-range push in master (#4253)
* all: fix ci ref_protected check for caching (#4254)
* Switch over from CircleCI to GitHub A tions (#4256)
* all: [GitHub actions] Reset the module and build cache in master/protected (#4266)
* Forgot to add sudo in #4266 (#4270)
* all: More go-setup github action fixes (#4274)
* xdr: add instructions for generating xdr (#4280)
* services/ticker: cache tomls during scraping (#4286)
* services/ticker: use log fields during asset ingestion (#4288)
* services/ticker: reduce size of toml cache in memory (#4289)
* historyarchive: add --skip-optional flag (#3906)
* all: Add Protocol 19 XDR and update StrKey to support Signed Payloads (#4279)
* Replace keybase with publicnode in the stellar core config (#4291)
* Fix captive core tests to write to /tmp, instead of polluting the repo (#4296)
* all: remove go1.16 add go1.18 (#4284)
* Rename methods and functions in submission system (#4298)
* PR feedback (#4300)
* Support new account fields for protocol-19. (#4294)
* xdr, keypair: Add helpers to create CAP-40 decorated signatures (#4302)
* services/horizon: Update txsub queue to account for new CAP-21 preconditions (#4301)
* Uncomment StateVerifier test that generates account v3 extensions now that they are implemented. (#4304)
* txnbuild: Add support for new CAP-21 preconditions. (#4303)
* services/horizon: Support new CAP-21 transaction conditions (#4297)
* txnbuild: Complete rename, avoid using XDR types in `TransactionParams`. (#4307)
* all: Update Protocol 19 XDR to the latest (#4308)
* services/horizon: Add a rate limit for path finding requests. (#4310)
* clients/horizonclient: fix multi-parameter url for claimable balance query (#4248)
* all: Fix Horizon integration tests (#4292)
* horizon: Fix integration tests (#4314)
* horizon: Set up protocol 19 integration tests infrastructure (#4312)
* all: Change outdated CircleCI build badge (#4324)
* horizon: Test new protocol 19 account fields (#4322)
* all: update staticcheck to 2022.1 (#4326)
* all: remove go.list and related docs (#4328)
* horizon: Add transaction submission test for Protocol 19 (#4327)
* Horizon v2.16.1 CHANGELOG (#4333)
* Revert "Pin go versions temporarily" (#4338)
* services/horizon: Use `bigint` over `timestamp` to accommodate large years (#4337)
* xdr: Update xdrgen (#4341)
* services/horizon: Change `min_account_sequence_age` column from `bigint` to string (#4339)
* services/horizon: Bump stellar-core to v19.0.0rc1 for Horizon tests (#4345)
* services/horizon: expose supported protocol version on root endpoint (#4347)
* horizon: Small transaction submission refactoring (#4344)
* services/horizon: Pass through nil ExtraSigners to avoid nil pointer deref (#4349)
* doc: rename license file (#4350)
* all: upgrade dep github.com/valyala/fasthttp (#4351)
* services/horizon: Promote Stellar Core to v19.0.0 stable. (#4353)
* services/horizon/integration: Precondition edge cases and V18->19 upgrade boundary. (#4354)
* xdr: Synchronizes monorepo XDR with Stellar Core (#4355)
* services/horizon: Properly allow nullable Protocol 19 account fields (#4357)
* services/friendbot: include txhash in logs (#4359)
* services/horizon: Improve transaction precondition `omitempty` behavior (#4360)
* tools/horizon-cmp: Improve panic error message (#4365)
* services/horizon: Merge stable v2.17.0 back into master: (#4363)
* Use UNIX timestamps instead of RFC3339 strings for timebounds. (#4361)
* xdrgen: remove gemfile and rakefile to just use docker for the xdrgen (#4366)
* Conservatively limit the number of DB connections of integration tests (#4368)
* internal/integrations: db_test should drop test db instances when finished (#4185)
* GHA: Bump Core version to v19.0.1 in Horizon workflows. (#4378)
* services/horizon, clients/horizonclient: Allow filtering ingested transactions by account or asset. (#4277)
* Push stellar/ledger-state-diff images from Github actions (#4380)
* services/horizon: Fixes copy-paste typo in `--help` text (#4383)
* tools/alb-replay: Add new features to alb-replay (#4384)
* services/horizon: Optimize claimable balances query to limit records earlier (#4385)
* support/db, services/horizon/internal: Configure postgres client connection timeouts for read only db (#4390)
* Refactor trade aggregation query. (#4389)
* services/horizon/internal/db2/history: Implement StreamAllOffers using batches (#4397)
* Add flag to disable path finding endpoints (#4399)

Co-authored-by: stfung77 <stfung77@gmail.com>
Co-authored-by: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com>
Co-authored-by: Alfonso Acosta <alfonso@stellar.org>
Co-authored-by: Paul Bellamy <paul@stellar.org>
Co-authored-by: Bartek Nowotarski <bartek@nowotarski.info>
Co-authored-by: tamirms <tamirms@gmail.com>
Co-authored-by: Alfonso Acosta <fons@syntacticsugar.consulting>
Co-authored-by: Graydon Hoare <graydon@pobox.com>
Co-authored-by: Satyam Zode <5508956+satyamz@users.noreply.github.com>
Co-authored-by: Satyam Zode <satyamz@users.noreply.github.com>
Co-authored-by: erika-sdf <92893480+erika-sdf@users.noreply.github.com>
Co-authored-by: iateadonut <orienta.dan@gmail.com>
Co-authored-by: Shawn Reuland <shawn@stellar.org>
Co-authored-by: shawn <sreuland@users.noreply.github.com>
Co-authored-by: Shivendra Mishra <shivendra_mishra@live.com>
Co-authored-by: Jacek Nykis <jacek@stellar.org>
Co-authored-by: jacekn <jacekn@users.noreply.github.com>

* Explain map and reduce commands

* exp/lighthorizon: Refactor single-process index builder. (#4410)

* Refactor index builder:
 - allow worker count to be a command line parameter
 - split work by checkpoints rather than ledgers
 - move actual index insertion work to helpers
 - move progress bar into helpers
 - simplify participants code, payments vs. all
* Properly work on a checkpoint range at a time:
 - previously, it was just arbitrary 64-ledger chunks which is not as helpful
* Define a generic module processing function
* Move index building into a separate object
* Fix off-by-one error in checkpoint index builder:
  - Keeping this as-is would mean that the first chunk of ledgers
    will be "Checkpoint 0" which doesn't make sense in the bitmap
  - Calling index.setActive(0) is essentially a no-op, because no
    bit will ever be set.
  - In the case of an empty index in which the only active account
    checkpoint is the first one, this is indistinguishable from an
    index with no activity.

* exp/services/ledgerexporter: Extend tool to support lower ledger bound. (#4405)

* exp/lighthorizon: Refactor and repair the reduce job (#4424)

* Use envvars for every configurable thing, incl. index sources and final merged
  index target:

    This removes any hard dependency on S3 and lets you use any supported
    backend for the map-reduce operation. It was done specifically with local
    filesystem-based testing in mind, but naturally opens up other backends as
    well.

* Add lots of helper functions:

    Specifically, helpers now exist for both merging two sets of named indices
    together and partitioning work based on the account/transaction hashes into
    separate jobs/routines.

* Lots more logging! For progress tracking, debugging, etc.

* Create a thread-safe string set abstraction for tracking completed work.

* Better error handling: 

    `os.IsNotExist(err)` is much more reliable over a direct equality check to
    `ErrNotExist`. This also ties in to backend-independence. 

    We can also log and return an error rather than immediately panicking on its
    occurrence.

* Transaction flushes need to be thread-safe if they're going to be done from
  different goroutines during reduction.

    Otherwise, you get panics from concurrent writes to its maps.

* The "account list" (aka the file containing a list of all accounts in the
  partitioned index) needs to be flushed at the same time as the index itself:

    If this isn't done, then `FlushAccounts()` will do absolutely nothing after
    a `Flush()`, because the previous `Flush()` will clear the map of indices
    out of memory. Since the account list comes from memory, it becomes a no-op.

* Split work across multiple channels rather than just one

    If the work comes from a single channel, accounts can get skipped overall
    because they aren't put back on the queue if they're skipped by a single
    worker.

    It makes more sense to make each worker have its own channel, partitioning
    the work *before* it gets to the worker rather than after.

* exp/lighthorizon: Unify map-reduce and single-process index builders (#4423)

* Main thing: `./index/cmd/single` and `./index/cmd/batch/map` now leverage the
  same index building code (i.e. `BuildIndices`)

* This also extends the map-reduce builder to take the txmeta source / index
  destination URLs from envvars rather:

    This eliminates a hard dependency on S3, and it's done here because
    splitting that out from the giga-PR was difficult.

* We can infer checkpoints from `ledger.LedgerSequence()` rather than passing
  them in as a parameter, which cleans up modules.

* This finally adds a new `ProcessAccountsWithoutBackend` module for the Map job

* exp/lighthorizon: Thread-safe support for reading account list via FileBackend (#4422)

Three key changes:

    - actually read the account list when using a filesystem backend
    - using `O_APPEND` on the file to support concurrent writes
    - ensure that the read list is a unique set of accounts

* exp/lighthorizon: Restructure index package into sensible sub-packages (#4427)

* exp/lighthorizon: Merge on-disk index with in-memory one on load. (#4435)

* Add test for single-process index builder
* Merge in-memory index with on-disk one when loading
* Add fixture of unpacked ledgers for fast local testing
* Isolate the index we need to merge
* Use a ByteReader so that multiple indices in one file work 🤦
* Add to/from XDR support to bitmap index
* Fix and extend gzip tests to handle the bytereader bug
* Simplify participant processing code

* exp/lighthorizon: Allow indexer to continually update as new txmeta appears (#4432)

* exp/lighthorizon: enforce the limit from request on the response size  (#4431)

* Dockerize ledgerexport to run in AWS Batch

This Change:

1. Creates docker image (stellar/horizon-ledgerexporter) which works in a similar fashion to stellar/horizon-verify-rage
   and is tested and pushed as part of the Horizon GitHub workflow.
2. Adds two more parameters to ledgerexporter
   * --end-ledger: which indicates at what ledger to stop the export
   * --write-latest-path: which indicates whether to udpate the /latest path of the target

Latest path writing is disabled in the container by default in order to avoid race-conditions between parallel jobs

* exp/lighthorizon: Add test for batch index building map job (#4440)

* Modify single-process test to generalize to whatever fixture data exists
This also adds a test to check that single-process works on a non-checkpoint
starting point which is important.

* Fix map program to properly build sub-paths depending on its job index
Previously, this only happened for explicitly S3 backends.

* Make map job default to using all CPUs
* Stop clearing indices from memory if using unbacked module
* Use historyarchive.CheckpointManager for all checkpoint math
* Update lastBuiltLedger w/ safely concurrent writes

* Refactor bound preparation and add --continue flag

* Address review feedback and rework env variable names

* Run gofmt -w (I don't know why those files were changed)

* Add proper logging to indicate what range is being exported

* Add clarification about end ledger

* Fix boolean argument passing

* Address review feedback

* Address feedback

* Use sqlite for captive core

* exp/lighthorizon: Add basic scaffolding for metrics. (#4456)

* Use correct network passphrase when populating transaction
* Add scaffolding for Prom/log metrics and some example ones
* Misc. clarifications and fixes to the index builder

* lighthorizon: Prepend version to ledger files (#4450)

* Prepend version to ledger files

* Encode versioning in XDR

* Regenerate fixtures

* Fix ledger fixtures

* Appease govet

* Move all lighthorizon types to /xdr

* exp/lighthorizon/index: More testing for batch indexing and off-by-one bugfix. (#4442)

* Add reduce test to ensure combining map jobs works
* Actually test that TOIDs are correct
* Bugfix: Transaction prefix loop should be inclusive
* Isolate loggers to individual processing "sections"

* Minor ledgerexporter infrastructure improvements (#4461)

* Push the stellar/horizon-ledgerexporter docker image when pushing to the lighthorizon branch
* Fix the ledger exporter aws batch jobs when running on the first batch

* Forgot to add login step to ledgerexporter workflow

* exp/lighthorizon: Set a default number of workers. (#4465)

* Default to the number of CPUs if worker count isn't specified
* Set a timeout on the reduce job to avoid test suite hanging indefinitely

* exp/lighthorizon: Fix the single-process index builder data race. (#4470)

* Add synchronization for the work submission routine. Thank you @sreuland!

Co-authored-by: shawn <sreuland@users.noreply.github.com>

* /exp/lighthorizon: new endpoints for tx and ops paged listing by account id (#4453)

* exp/lighthorizon: Add an on-disk cache for frequently accessed ledgers. (#4457)

* Replace custom LRU solution with an off-the-shelf data structure.
* Add a filesystem cache in front of the ledger backend to lower latency
* Add cache size parameter; only setup cache if not file://
* Extract S3 region from the archive URL if it's applicable.

* exp/lighthorizon/index: Drop building indices for successful transactions. (#4482)

* Add metrics middleware to collect request duration metrics (#4486)

* exp/lighthorizon: Isolate cursor advancement code to its own interface (#4484)

* Move cursor manipulation code to a separate interface
* Small test refactor to improve readability and long-running lines
* Combine tx and op tests into subtests
* Fix how IndexStore is mocked out

* exp/lighthorizon/index: Parse network passphrase from the env. (#4491)

* Refactor access to meta archive (#4488)

Refactor `historyarchive` and `ledgerbackend` to allow better access to the new meta archives:
* Created `metaarchive` package that connects to the new meta archives (and
  allows accessing `xdr.SerializedLedgerCloseMeta`).
* Extracted `ArchiveBackend` to the new `support/storage` package as it contains
  only storage related methods. New package is used in both `historyarchive` and
  `metaarchive`.

* exp/lighthorizon: Add response age prometheus metrics (#4492)

* exp/lighthorizon/index: Allow accounts to be indexed by ledger. (#4495)

* Add builders to make account indices by ledger
* Add `MODULE` parameter to map job in batch builder
* Don't build transaction indices by default

* services/horizon/docker/ledgerexporter: deploy ledgerexporter image as service (#4490)

* Make indexing s3 bucket configurable (#4507)

* exp/lighthorizon: Add duration metrics for on-the-fly ingestion elements. (#4476)

Add basic aggregate metrics for request fulfillment:
 - how long did ledger downloads take, on average?
 - how long did ledger processing take, on average?
 - how long did index lookups take, on average?
 - how many ledgers were needed?
 - how long did the entire request take, in total?

* exp/lighthorizon: Add JSON content type to responses. (#4509)

* exp/lighthorizon: *Correctly* set `Content-Type`, plus JSONify errors (#4513)

* exp/lighthorizon/services: Move service-specific stuff to its own file. (#4502)

* exp/lighthorizon, xdr: Rename `CheckpointIndex` to better reflect its capabilty. (#4510)

* Rename NextActive -> NextActiveBit to be descriptive

* exp/lighthorizon: Add a suite of tools to manage the on-disk ledger cache. (#4522)

* Run 'go mod tidy' after merge

* exp/lighthorizon: add horizon web docker/k8s deployment (#4519)

* It seems like the merge caused some deleted files to stay in:

  The commit b3407fd from
  PR #4418 deleted these files, so we just do the same.

  A quick manual inspection showed us that the deltas
  transferred over, just not the deletions, for some reason.
Idk why these changes ended up in the code, kinda sus...

More deleted files snuck in?

* One more that didn't get removed 🤔

* all: Incorporate generics into Light Horizon code. (#4537)

* bump go version to 18 on lighthorizon docker images, they need it now (#4541)

* exp/lighthorizon/actions: use standard Problem model on API error responses (#4542)

* exp/lighthorizon/build/index-batch: carry over map/reduce updates to latest docker layout on feature branch (#4543)

* exp/lighthorizon: Properly transform transactions into JSON. (#4531)

* exp/lighthorizon: Add a set of tools to aide in index inspection. (#4561)

* exp/lighthorizon/cmd: index batch fix s3 sub paths in reduce (#4552)

* exp/lighthorzon: Add a generic, thread-safe `SafeSet`. (#4572)

* support/storage: Make the on-disk cache thread-safe. (#4575)

* exp/lighthorizon: Incorporate tool subcommands into the webserver. (#4579)

* exp/lighthorizon/index/cmd: Fix index single watch, slow down the retry on not-found ledgers  (#4582)

* exp/lighthorizon: Refactor archive interface and support parallel ledger downloads. (#4548)
- Refactor and simplify Archive abstraction to incorporate MetaArchive
- Actually add & use parallel downloads, preparing checkpoint chunks
- Fix test structures and mocking
- Fix cache to ignore on-disk if lockfile present

* exp/lighthorizon: Minor error-handling and deployment improvements. (#4599)
- actually set the PARALLEL_DOWNLOADS parameter to use #4468
- return a 404 rather than a 500 if a ledger is missing as its more descriptive
- handle `count = 0` in average metric calculations
* exp/lighthorizon/index: Add ability to disable bits in index. (#4601)
* exp/lighthorizon: Add parameters to preload ledger cache. (#4615)
* Add ability to preload cache in parallel after launching webserver
* Default to 1 day of ledgers @ 6s each

---------

Co-authored-by: Bartek Nowotarski <bartek@nowotarski.info>
Co-authored-by: Paul Bellamy <paul@stellar.org>
Co-authored-by: Bartek <bartek@MacBook-Air-Bartek.local>
Co-authored-by: Bartek <bartek@air-bartek.home>
Co-authored-by: tamirms <tamirms@gmail.com>
Co-authored-by: George <Shaptic@users.noreply.github.com>
Co-authored-by: stfung77 <stfung77@gmail.com>
Co-authored-by: Leigh McCulloch <351529+leighmcculloch@users.noreply.github.com>
Co-authored-by: Alfonso Acosta <alfonso@stellar.org>
Co-authored-by: Alfonso Acosta <fons@syntacticsugar.consulting>
Co-authored-by: Graydon Hoare <graydon@pobox.com>
Co-authored-by: Satyam Zode <5508956+satyamz@users.noreply.github.com>
Co-authored-by: Satyam Zode <satyamz@users.noreply.github.com>
Co-authored-by: erika-sdf <92893480+erika-sdf@users.noreply.github.com>
Co-authored-by: iateadonut <orienta.dan@gmail.com>
Co-authored-by: Shawn Reuland <shawn@stellar.org>
Co-authored-by: shawn <sreuland@users.noreply.github.com>
Co-authored-by: Shivendra Mishra <shivendra_mishra@live.com>
Co-authored-by: Jacek Nykis <jacek@stellar.org>
Co-authored-by: jacekn <jacekn@users.noreply.github.com>
Co-authored-by: George Kudrayvtsev <george@stellar.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tamirms tamirms tamirms approved these changes

@2opremio 2opremio Awaiting requested review from 2opremio

@sreuland sreuland Awaiting requested review from sreuland

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Feature Request: Allow captive core to run with sqlite database
6 participants
@paulbellamy @sreuland @2opremio @tamirms @jacekn @ire-and-curses