exp/ingest: ledger backend support

[ire-and-curses]

If you're making a doc PR or something tiny where the below is irrelevant, just delete this
template and use a short description.

PR Structure

• This PR has reasonably narrow scope (if not, break it down into smaller PRs).
• This PR avoids mixing refactoring changes with feature changes (split into two PRs
  otherwise).
• This PR's title starts with name of package that is most changed in the PR, ex.
  services/friendbot

Thoroughness

• This PR adds tests for the most critical parts of the new functionality or fixes.
• I've updated any docs (developer docs, .md
  files, etc... affected by this change). Take a look in the docs folder for a given service,
  like this one.

Release planning

• I've updated the relevant CHANGELOG (here for Horizon) if
  needed with deprecations, added features, breaking changes, and DB schema changes.
• I've decided if this PR requires a new major/minor version according to
  semver, or if it's mainly a patch change. The PR is targeted at the next
  release branch if it's not a patch change.

Summary

Goal and scope

This PR adds support for reading transaction information and headers of individual ledgers from a stellar-core database. This work is part of the ingest project, and is intended to allow a data pipeline to poll for current ledger information for a detailed and current view of the ledger state.

Summary of changes

Following the design described in the ingestion plan, this PR provides

• interface io.LedgerReadCloser -> implemented by io.DBLedgerReadCloser
• interface ledgerbackend.LedgerBackend -> implemented by ledgerbackend.DatabaseBackend
• adapter implementation ingestadapters.LedgerBackendAdapter
• supporting datastructures (especially LedgerTransaction and LedgerCloseMeta)

A simple demo running against a local stellar-core docker container is included (ingest/cmd/main).

Known limitations & issues

I ran into a number of questions during the implementation and would appreciate any feedback.

1. I modified some of the interfaces. Specifically, I added Close methods to allow the DB session to be cleared up. I also added an Init method for the LedgerReadCloser, to allow the internal data to be loaded in advance of the first Read() call. Does this make sense?
2. Should there be an interface that LedgerBackendAdapter should conform to?
3. I did a lot of manual testing of this code but it needs more unit tests to verify the data that is returned is correct. This is hard to do currently as DB calls are internal to the objects. I could try to refactor the code through dependency injection to allow mock DB objects to be provided, but I'm a little worried it will add a lot of complexity. I'm open to suggestions for how best to do that, or if there's another approach to testing that would be better.
4. I'm not sure whether I have deserialised the ledger hash correctly, would appreciate a sanity check there.
5. The returned data is my best guess at what's required for the pipeline. Does it look sufficient, or are there things missing?
6. During my testing I found it helpful to see a timestamp for the latest ledger sequence number. This is not part of the interface, so I just logged it internally. We could consider returning the timestamp to the caller so they could e.g. monitor staleness.
7. I used individual raw string queries because they were simple. These could be combined to reduce the number of DB calls if needed. I'm also not sure if there's a better way to construct the queries using the DB objects provided by support/db.
8. Once final interfaces are merged, the docs should be updated.

What shouldn't be reviewed

Should all be reviewed.

[bartekn]

Looks good. There are a few things to change but in general it goes in the right direction. I answered your questions in code comments. Let me know if I missed anything.

[bartekn]

I'm open to suggestions for how best to do that, or if there's another approach to testing that would be better.

I think we should just mock db.Session:

1. Create a private interface in ledgerbackend package with methods we use (SelectRaw, GetRaw...).
2. Create a mock.
3. Then in tests program a mock to set the results to what you need using Call.Run.

However, I think we need at least a few tests that work against a running database to ensure queries are correct etc. These can be run in CI only.

[ire-and-curses]

This is a great idea. It's not possible at present because we access session.DB.Close() in DatabaseBackend.Close(). Since DB is a struct field, not a method, it can't be accessed through an interface.

One solution would be to add a new method session.Close() to the db package. This could delegate to the underlying *sqlx.DB.Close() for the real thing, and be a no-op for the mock. What do you think?

[bartekn]

Yeah, let's do it.

[ire-and-curses]

@bartekn I've implemented all requested changes except the three unresolved issues above:

1. No SQL error raised for empty rows
2. Missing Close() method for session mock interface
3. Keep or remove redundant index field?

Please take a look and let me know your preferences and I'll finish this up.

[ire-and-curses]

@bartekn remaining changes have been made - PTAL. I can work on adding more detailed unit and CI tests now, but I think I'd prefer to merge this and add them in a new PR, so the code is generally available (it's still in exp), unless you feel strongly the other way.

[bartekn]

Last issues. The one in DBLedgerReadCloser.GetHeader() is important so will approve once it's fixed.