Horizon 0.21.0 CHANGELOG and docs (#1770)

services/horizon/CHANGELOG.md

@@ -8,6 +8,12 @@ bumps.  A breaking change will get clearly notified in this log.
 
 ## v0.21.0
 
+### Database migration notes
+
+This version adds a new index on a table used by experimental ingestion system. If it has not been enabled, migration will be instant. If you migrate from a previous version with experimental ingestion system enabled database migration can take a couple minutes.
+
+### Changes
+
 * `/paths/strict-send` can now accept a `destination_account` parameter. If `destination_account` is provided then the endpoint will return all payment paths which terminate with an asset held by `destination_account`. Note that the endpoint will accept `destination_account` or `destination_assets` but not both. `destination_assets` is a comma separated list of assets encoded as `native` or `code:issuer`.
 * `/paths/strict-receive` can now accept a `source_assets` parameter instead of `source_account` parameter. If `source_assets` is provided the endpoint will return all payment paths originating from an asset in `source_assets`. Note that the endpoint will accept `source_account` or `source_assets` but not both. `source_assets` is a comma separated list of assets encoded as `native` or `code:issuer`.
 * Add experimental support for `/offers`. To enable it, set `--enable-experimental-ingestion` CLI param or `ENABLE_EXPERIMENTAL_INGESTION=true` env variable.
@@ -16,8 +22,11 @@ bumps.  A breaking change will get clearly notified in this log.
 * Improved the speed of state ingestion in experimental ingestion system.
 * Fixed a bug in "Signers for Account" (experimental) transaction meta ingesting code.
 * Fixed performance issue in Effects related endpoints.
+* Fixed DoS vector in Go HTTP/2 implementation.
 * Dropped support for Go 1.10, 1.11.
 
+Check [Beta Testing New Ingestion System](https://github.com/stellar/go/blob/master/services/horizon/internal/expingest/BETA_TESTING.md) if you want to test new ingestion system.
+
 ## v0.20.1
 
 * Add `--ingest-state-reader-temp-set` flag (`INGEST_STATE_READER_TEMP_SET` env variable) which defines the storage type used for temporary objects during state ingestion in the new ingestion system. The possible options are: `memory` (requires ~1.5GB RAM, fast) and `postgres` (stores data in temporary table in Postgres, less RAM but slower).

services/horizon/internal/expingest/BETA_TESTING.md

@@ -12,7 +12,7 @@ The new ingestion system solves issues found in the previous version like: incon
 
 ## Prerequisities
 
-* The init stage (state ingestion) for public network requires around 1.5GB of RAM. The memory is released after the state ingestion. State ingestion is performed only once, restarting the server will not trigger it unless Horizon has been upgraded to a newer version (with updated ingestion pipeline). We are currently working on alternative solutions to make RAM requirements smaller however we believe that it will become smaller and smaller as more buckets are [CAP-20](https://github.com/stellar/stellar-protocol/blob/master/core/cap-0020.md) compatible. The CPU footprint of the new ingestion is really small. We were able to run experimental ingestion on `c5.large` instance on AWS.
+* The init stage (state ingestion) for public network requires around 1.5GB of RAM. The memory is released after the state ingestion. State ingestion is performed only once, restarting the server will not trigger it unless Horizon has been upgraded to a newer version (with updated ingestion pipeline). We are currently working on alternative solutions to make RAM requirements smaller however we believe that it will become smaller and smaller as more buckets are [CAP-20](https://github.com/stellar/stellar-protocol/blob/master/core/cap-0020.md) compatible. The CPU footprint of the new ingestion is really small. We were able to run experimental ingestion on `c5.large` instance on AWS. The init stage takes a few minutes on `c5.large`.
 * The state data requires additional 1.3GB DB disk space (as of version 0.20.1).
 * Flags needed to enable experimental ingestion:
   * `ENABLE_EXPERIMENTAL_INGESTION=true`
@@ -26,14 +26,20 @@ Endpoints that display state information like `/paths` (built using offers) are
 
 ### State ingestion is taking a lot of time
 
-State ingestion speed depends mostly on a database specs. In most cases so far the new system was able to ingest the state in around 30 minutes. We are currently working on making this process as fast as possible.
+Since Horizon 0.21.0 the state ingestion shouldn't take more than a couple minutes on AWS `c5.large` instance.
 
-It's also possible that the progress logs (see below) will not show anything new for a longer period of time (>15 minutes) or print a lot of progress entries every few seconds. This is happening because of the way history archives are designed, the ingestion is still working but it's processing so called `DEADENTRY`'ies: if there is a lot of them in the bucket, there are no _active_ entries to process. We plan to improve the progress logs to display actual percentage progress so it's easier to estimate ETA.
+It's possible that the progress logs (see below) will not show anything new for a longer period of time or print a lot of progress entries every few seconds. This is happening because of the way history archives are designed, the ingestion is still working but it's processing so called `DEADENTRY`'ies: if there is a lot of them in the bucket, there are no _active_ entries to process. We plan to improve the progress logs to display actual percentage progress so it's easier to estimate ETA.
 
 If you see that ingestion is not proceeding for a very long period of time:
 1. Check the RAM usage on the machine. It's possible that system run out of RAM and it using swap memory that is extremely slow.
 2. If above is not the case, file a new issue in this repository.
 
+### CPU usage goes high every few minutes
+
+This is _by design_. Horizon 0.21.0 introduced a state verifier routine that compares state in local storage to history archives every 64 ledgers to ensure data changes are applied correctly. If data corruption is detected Horizon will block access to endpoints serving invalid data.
+
+We recommend to keep this security feature turned on however if it's causing problems (due to CPU usage) this can be disabled by `--ingest-disable-state-verification` CLI param or `INGEST-DISABLE-STATE-VERIFICATION` env variable.
+
 ## Reading the logs
 
 In order to check the progress and the status of experimental ingestion you should check the logs. All logs connected to experimental ingestion are tagged with `service=expingest`.