Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GCP stateless HA support via Firestore and GCS #2821

Merged
merged 1 commit into from Sep 24, 2019

Conversation

@joshdurbin
Copy link
Contributor

joshdurbin commented Jul 2, 2019

This follows information in this issue and
adds support for stateless HA clusters backed by GCP services (Firestore and GCS). Screen recordings / sessions are stored in GCS while event information and backend support is implemented in Firestore.

Firestore must be configured for "Native" mode, which allows for document snapshotting.

Firestore Indexes

Firestore automatically indexes all individual fields and requires custom indexes be created for compound queries. Both the Firestore Backend and Firestore Events "Backend/Handler" check, create, and block until indexes are created. This is done to prevent query errors on cluster startup.

Two composite indexes are required for the events collection:

  1. EventNamespace ascending, then on CreatedAt ascending
  2. SessionID ascending, then on EventIndex ascending

One composite index is required for the cluster-data collection:

  1. key ascending, then on expires ascending

Firestore Doc IDs

Firestore Document IDs must be unique, cannot start with periods, and cannot contain forward
slashes. In order to support more straight forward fetching but work within the requirements
of Firestore, Event Document IDs are the concatenation of the session ID (a UUID) and event type joined with a dash -, ex: 13498a42-69a8-4fa2-b39d-b0c49e346713-user.login. Firestore Backend Document IDs are a SHA1 hash of the document's key.

Both Firestore Event and Firestore Backend's support record purging; events are purged based on the created timestamp of the event as they're not updated, while records stored in the Firestore Backend are purged based on and only if they have expires set.

Realtime updates

Firestore Backend watches collection and document snapshots and reports those to all watchers. Document added and modified events are pushed as put operations to the buffer. Removal events are also pushed. Retrieval events are NOT pushed to the buffer.

@gravitational-jenkins

This comment has been minimized.

Copy link

gravitational-jenkins commented Jul 2, 2019

Can one of the admins verify this patch?

@joshdurbin joshdurbin force-pushed the bigcommerce:gcp_ha_support branch 2 times, most recently from 7a5da14 to 5740611 Jul 2, 2019
@joshdurbin joshdurbin changed the title GCP stateless HA support via Firestore and GCS WIP: GCP stateless HA support via Firestore and GCS Jul 3, 2019
@joshdurbin joshdurbin force-pushed the bigcommerce:gcp_ha_support branch 4 times, most recently from cd2e73f to 654babe Jul 3, 2019
@joshdurbin joshdurbin force-pushed the bigcommerce:gcp_ha_support branch from c3260d9 to a788443 Jul 16, 2019
@joshdurbin joshdurbin changed the title WIP: GCP stateless HA support via Firestore and GCS GCP stateless HA support via Firestore and GCS Jul 16, 2019
@joshdurbin joshdurbin force-pushed the bigcommerce:gcp_ha_support branch from a788443 to 1b59526 Jul 16, 2019
Key: []byte(r.Key),
Value: []byte(r.Value),
Expires: expires,
ID: r.ID,

This comment has been minimized.

Copy link
@klizhentas

klizhentas Jul 16, 2019

Contributor

Research the revision property, can we use some incremental unique doc id from firestore?

return &lease
}

// hashKey hashes keys using SHA1 enabling fast lookups

This comment has been minimized.

Copy link
@klizhentas

klizhentas Jul 16, 2019

Contributor

Let's use reversible encoding here, instead of sha1 - e.g. hex or base64 depending on what works for firebase, alternatively, we can use replacement escaping symbol for \

This comment has been minimized.

Copy link
@joshdurbin

joshdurbin Jul 20, 2019

Author Contributor

Going w/ string replacement so that document IDs are easy to find in the Firestore UI.

func RetryingAsyncFunctionRunner(ctx context.Context, retryConfig utils.LinearConfig, logger *log.Logger, task func() error, taskName string) {
retry, err := utils.NewLinear(retryConfig)
if err != nil {
logger.Errorf("bad retry parameters: %v, returning and not running", err)

This comment has been minimized.

Copy link
@klizhentas

klizhentas Jul 16, 2019

Contributor

General note on style log entries are proper english sentences - Bad retry parameters: %v, returning and not running. and also for errors it's the opposite -they are lowercase with no dot at the end.


// watchCollection watches a firestore collection for changes and pushes those changes, events into the buffer for watchers
func (b *FirestoreBackend) watchCollection() error {
snaps := b.svc.Collection(b.CollectionName).Snapshots(b.clientContext)

This comment has been minimized.

Copy link
@klizhentas

klizhentas Jul 16, 2019

Contributor

Need to test this on 10K nodes (you don't have to, we will do it later after merge)

This comment has been minimized.

Copy link
@joshdurbin

joshdurbin Jul 22, 2019

Author Contributor

I tested a new instance, created an account, and created 5 sessions for a few minutes and watched the prometheus metrics in two cases; one with just that strategy and two with the code below, generating 100_000 junk records.

The prometheus metrics without 100_000 junk records are:

# HELP audit_failed_disk_monitoring Number of times disk monitoring failed.
# TYPE audit_failed_disk_monitoring counter
audit_failed_disk_monitoring 0
# HELP audit_failed_emit_events Number of times emitting audit event failed.
# TYPE audit_failed_emit_events counter
audit_failed_emit_events 0
# HELP audit_percentage_disk_space_used Percentage disk space used.
# TYPE audit_percentage_disk_space_used gauge
audit_percentage_disk_space_used 16
# HELP audit_server_open_files Number of open audit files
# TYPE audit_server_open_files gauge
audit_server_open_files 0
# HELP auth_generate_requests Number of current generate requests
# TYPE auth_generate_requests gauge
auth_generate_requests 0
# HELP auth_generate_requests_throttled_total Number of throttled requests to generate new server keys
# TYPE auth_generate_requests_throttled_total counter
auth_generate_requests_throttled_total 0
# HELP auth_generate_requests_total Number of requests to generate new server keys
# TYPE auth_generate_requests_total counter
auth_generate_requests_total 3
# HELP auth_generate_seconds Latency for generate requests
# TYPE auth_generate_seconds histogram
auth_generate_seconds_bucket{le="0.001"} 3
auth_generate_seconds_bucket{le="0.002"} 3
auth_generate_seconds_bucket{le="0.004"} 3
auth_generate_seconds_bucket{le="0.008"} 3
auth_generate_seconds_bucket{le="0.016"} 3
auth_generate_seconds_bucket{le="0.032"} 3
auth_generate_seconds_bucket{le="0.064"} 3
auth_generate_seconds_bucket{le="0.128"} 3
auth_generate_seconds_bucket{le="0.256"} 3
auth_generate_seconds_bucket{le="0.512"} 3
auth_generate_seconds_bucket{le="1.024"} 3
auth_generate_seconds_bucket{le="2.048"} 3
auth_generate_seconds_bucket{le="4.096"} 3
auth_generate_seconds_bucket{le="8.192"} 3
auth_generate_seconds_bucket{le="16.384"} 3
auth_generate_seconds_bucket{le="32.768"} 3
auth_generate_seconds_bucket{le="+Inf"} 3
auth_generate_seconds_sum 2.5200000000000003e-07
auth_generate_seconds_count 3
# HELP backend_batch_read_requests_total Number of read requests to the backend
# TYPE backend_batch_read_requests_total counter
backend_batch_read_requests_total{component="backend"} 205
backend_batch_read_requests_total{component="cache"} 278
# HELP backend_batch_read_seconds Latency for batch read operations
# TYPE backend_batch_read_seconds histogram
backend_batch_read_seconds_bucket{component="backend",le="0.001"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.002"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.004"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.008"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.016"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.032"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.064"} 117
backend_batch_read_seconds_bucket{component="backend",le="0.128"} 200
backend_batch_read_seconds_bucket{component="backend",le="0.256"} 204
backend_batch_read_seconds_bucket{component="backend",le="0.512"} 205
backend_batch_read_seconds_bucket{component="backend",le="1.024"} 205
backend_batch_read_seconds_bucket{component="backend",le="2.048"} 205
backend_batch_read_seconds_bucket{component="backend",le="4.096"} 205
backend_batch_read_seconds_bucket{component="backend",le="8.192"} 205
backend_batch_read_seconds_bucket{component="backend",le="16.384"} 205
backend_batch_read_seconds_bucket{component="backend",le="32.768"} 205
backend_batch_read_seconds_bucket{component="backend",le="+Inf"} 205
backend_batch_read_seconds_sum{component="backend"} 13.914725686999988
backend_batch_read_seconds_count{component="backend"} 205
backend_batch_read_seconds_bucket{component="cache",le="0.001"} 276
backend_batch_read_seconds_bucket{component="cache",le="0.002"} 276
backend_batch_read_seconds_bucket{component="cache",le="0.004"} 277
backend_batch_read_seconds_bucket{component="cache",le="0.008"} 278
backend_batch_read_seconds_bucket{component="cache",le="0.016"} 278
backend_batch_read_seconds_bucket{component="cache",le="0.032"} 278
backend_batch_read_seconds_bucket{component="cache",le="0.064"} 278
backend_batch_read_seconds_bucket{component="cache",le="0.128"} 278
backend_batch_read_seconds_bucket{component="cache",le="0.256"} 278
backend_batch_read_seconds_bucket{component="cache",le="0.512"} 278
backend_batch_read_seconds_bucket{component="cache",le="1.024"} 278
backend_batch_read_seconds_bucket{component="cache",le="2.048"} 278
backend_batch_read_seconds_bucket{component="cache",le="4.096"} 278
backend_batch_read_seconds_bucket{component="cache",le="8.192"} 278
backend_batch_read_seconds_bucket{component="cache",le="16.384"} 278
backend_batch_read_seconds_bucket{component="cache",le="32.768"} 278
backend_batch_read_seconds_bucket{component="cache",le="+Inf"} 278
backend_batch_read_seconds_sum{component="cache"} 0.058935650000000006
backend_batch_read_seconds_count{component="cache"} 278
# HELP backend_batch_write_requests_total Number of batch write requests to the backend
# TYPE backend_batch_write_requests_total counter
backend_batch_write_requests_total{component="cache"} 70
# HELP backend_batch_write_seconds Latency for backend batch write operations
# TYPE backend_batch_write_seconds histogram
backend_batch_write_seconds_bucket{component="cache",le="0.001"} 68
backend_batch_write_seconds_bucket{component="cache",le="0.002"} 69
backend_batch_write_seconds_bucket{component="cache",le="0.004"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.008"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.016"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.032"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.064"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.128"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.256"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.512"} 70
backend_batch_write_seconds_bucket{component="cache",le="1.024"} 70
backend_batch_write_seconds_bucket{component="cache",le="2.048"} 70
backend_batch_write_seconds_bucket{component="cache",le="4.096"} 70
backend_batch_write_seconds_bucket{component="cache",le="8.192"} 70
backend_batch_write_seconds_bucket{component="cache",le="16.384"} 70
backend_batch_write_seconds_bucket{component="cache",le="32.768"} 70
backend_batch_write_seconds_bucket{component="cache",le="+Inf"} 70
backend_batch_write_seconds_sum{component="cache"} 0.015011324000000003
backend_batch_write_seconds_count{component="cache"} 70
# HELP backend_read_requests_total Number of read requests to the backend
# TYPE backend_read_requests_total counter
backend_read_requests_total{component="backend"} 1548
backend_read_requests_total{component="cache"} 2623
# HELP backend_read_seconds Latency for read operations
# TYPE backend_read_seconds histogram
backend_read_seconds_bucket{component="backend",le="0.001"} 1548
backend_read_seconds_bucket{component="backend",le="0.002"} 1548
backend_read_seconds_bucket{component="backend",le="0.004"} 1548
backend_read_seconds_bucket{component="backend",le="0.008"} 1548
backend_read_seconds_bucket{component="backend",le="0.016"} 1548
backend_read_seconds_bucket{component="backend",le="0.032"} 1548
backend_read_seconds_bucket{component="backend",le="0.064"} 1548
backend_read_seconds_bucket{component="backend",le="0.128"} 1548
backend_read_seconds_bucket{component="backend",le="0.256"} 1548
backend_read_seconds_bucket{component="backend",le="0.512"} 1548
backend_read_seconds_bucket{component="backend",le="1.024"} 1548
backend_read_seconds_bucket{component="backend",le="2.048"} 1548
backend_read_seconds_bucket{component="backend",le="4.096"} 1548
backend_read_seconds_bucket{component="backend",le="8.192"} 1548
backend_read_seconds_bucket{component="backend",le="16.384"} 1548
backend_read_seconds_bucket{component="backend",le="32.768"} 1548
backend_read_seconds_bucket{component="backend",le="+Inf"} 1548
backend_read_seconds_sum{component="backend"} 0.0026599250000000035
backend_read_seconds_count{component="backend"} 1548
backend_read_seconds_bucket{component="cache",le="0.001"} 2623
backend_read_seconds_bucket{component="cache",le="0.002"} 2623
backend_read_seconds_bucket{component="cache",le="0.004"} 2623
backend_read_seconds_bucket{component="cache",le="0.008"} 2623
backend_read_seconds_bucket{component="cache",le="0.016"} 2623
backend_read_seconds_bucket{component="cache",le="0.032"} 2623
backend_read_seconds_bucket{component="cache",le="0.064"} 2623
backend_read_seconds_bucket{component="cache",le="0.128"} 2623
backend_read_seconds_bucket{component="cache",le="0.256"} 2623
backend_read_seconds_bucket{component="cache",le="0.512"} 2623
backend_read_seconds_bucket{component="cache",le="1.024"} 2623
backend_read_seconds_bucket{component="cache",le="2.048"} 2623
backend_read_seconds_bucket{component="cache",le="4.096"} 2623
backend_read_seconds_bucket{component="cache",le="8.192"} 2623
backend_read_seconds_bucket{component="cache",le="16.384"} 2623
backend_read_seconds_bucket{component="cache",le="32.768"} 2623
backend_read_seconds_bucket{component="cache",le="+Inf"} 2623
backend_read_seconds_sum{component="cache"} 0.0059324000000000095
backend_read_seconds_count{component="cache"} 2623
# HELP backend_watcher_queues_total Watcher queue sizes
# TYPE backend_watcher_queues_total gauge
backend_watcher_queues_total{component="auth:cache"} 0
# HELP backend_watchers_total Number of active backend watchers
# TYPE backend_watchers_total gauge
backend_watchers_total{component="backend"} 1
backend_watchers_total{component="cache"} 5
# HELP backend_write_requests_total Number of write requests to the backend
# TYPE backend_write_requests_total counter
backend_write_requests_total{component="backend"} 710
backend_write_requests_total{component="cache"} 111
# HELP backend_write_seconds Latency for backend write operations
# TYPE backend_write_seconds histogram
backend_write_seconds_bucket{component="backend",le="0.001"} 0
backend_write_seconds_bucket{component="backend",le="0.002"} 0
backend_write_seconds_bucket{component="backend",le="0.004"} 0
backend_write_seconds_bucket{component="backend",le="0.008"} 0
backend_write_seconds_bucket{component="backend",le="0.016"} 0
backend_write_seconds_bucket{component="backend",le="0.032"} 0
backend_write_seconds_bucket{component="backend",le="0.064"} 1
backend_write_seconds_bucket{component="backend",le="0.128"} 454
backend_write_seconds_bucket{component="backend",le="0.256"} 675
backend_write_seconds_bucket{component="backend",le="0.512"} 706
backend_write_seconds_bucket{component="backend",le="1.024"} 709
backend_write_seconds_bucket{component="backend",le="2.048"} 709
backend_write_seconds_bucket{component="backend",le="4.096"} 709
backend_write_seconds_bucket{component="backend",le="8.192"} 709
backend_write_seconds_bucket{component="backend",le="16.384"} 709
backend_write_seconds_bucket{component="backend",le="32.768"} 710
backend_write_seconds_bucket{component="backend",le="+Inf"} 710
backend_write_seconds_sum{component="backend"} 114.26281696300003
backend_write_seconds_count{component="backend"} 710
backend_write_seconds_bucket{component="cache",le="0.001"} 75
backend_write_seconds_bucket{component="cache",le="0.002"} 81
backend_write_seconds_bucket{component="cache",le="0.004"} 95
backend_write_seconds_bucket{component="cache",le="0.008"} 111
backend_write_seconds_bucket{component="cache",le="0.016"} 111
backend_write_seconds_bucket{component="cache",le="0.032"} 111
backend_write_seconds_bucket{component="cache",le="0.064"} 111
backend_write_seconds_bucket{component="cache",le="0.128"} 111
backend_write_seconds_bucket{component="cache",le="0.256"} 111
backend_write_seconds_bucket{component="cache",le="0.512"} 111
backend_write_seconds_bucket{component="cache",le="1.024"} 111
backend_write_seconds_bucket{component="cache",le="2.048"} 111
backend_write_seconds_bucket{component="cache",le="4.096"} 111
backend_write_seconds_bucket{component="cache",le="8.192"} 111
backend_write_seconds_bucket{component="cache",le="16.384"} 111
backend_write_seconds_bucket{component="cache",le="32.768"} 111
backend_write_seconds_bucket{component="cache",le="+Inf"} 111
backend_write_seconds_sum{component="cache"} 0.175271155
backend_write_seconds_count{component="cache"} 111
# HELP etcd_backend_batch_read_requests Number of read requests to the database
# TYPE etcd_backend_batch_read_requests counter
etcd_backend_batch_read_requests 0
# HELP etcd_backend_batch_read_seconds Latency for etcd read operations
# TYPE etcd_backend_batch_read_seconds histogram
etcd_backend_batch_read_seconds_bucket{le="0.001"} 0
etcd_backend_batch_read_seconds_bucket{le="0.002"} 0
etcd_backend_batch_read_seconds_bucket{le="0.004"} 0
etcd_backend_batch_read_seconds_bucket{le="0.008"} 0
etcd_backend_batch_read_seconds_bucket{le="0.016"} 0
etcd_backend_batch_read_seconds_bucket{le="0.032"} 0
etcd_backend_batch_read_seconds_bucket{le="0.064"} 0
etcd_backend_batch_read_seconds_bucket{le="0.128"} 0
etcd_backend_batch_read_seconds_bucket{le="0.256"} 0
etcd_backend_batch_read_seconds_bucket{le="0.512"} 0
etcd_backend_batch_read_seconds_bucket{le="1.024"} 0
etcd_backend_batch_read_seconds_bucket{le="2.048"} 0
etcd_backend_batch_read_seconds_bucket{le="4.096"} 0
etcd_backend_batch_read_seconds_bucket{le="8.192"} 0
etcd_backend_batch_read_seconds_bucket{le="16.384"} 0
etcd_backend_batch_read_seconds_bucket{le="32.768"} 0
etcd_backend_batch_read_seconds_bucket{le="+Inf"} 0
etcd_backend_batch_read_seconds_sum 0
etcd_backend_batch_read_seconds_count 0
# HELP etcd_backend_read_requests Number of read requests to the database
# TYPE etcd_backend_read_requests counter
etcd_backend_read_requests 0
# HELP etcd_backend_read_seconds Latency for etcd read operations
# TYPE etcd_backend_read_seconds histogram
etcd_backend_read_seconds_bucket{le="0.001"} 0
etcd_backend_read_seconds_bucket{le="0.002"} 0
etcd_backend_read_seconds_bucket{le="0.004"} 0
etcd_backend_read_seconds_bucket{le="0.008"} 0
etcd_backend_read_seconds_bucket{le="0.016"} 0
etcd_backend_read_seconds_bucket{le="0.032"} 0
etcd_backend_read_seconds_bucket{le="0.064"} 0
etcd_backend_read_seconds_bucket{le="0.128"} 0
etcd_backend_read_seconds_bucket{le="0.256"} 0
etcd_backend_read_seconds_bucket{le="0.512"} 0
etcd_backend_read_seconds_bucket{le="1.024"} 0
etcd_backend_read_seconds_bucket{le="2.048"} 0
etcd_backend_read_seconds_bucket{le="4.096"} 0
etcd_backend_read_seconds_bucket{le="8.192"} 0
etcd_backend_read_seconds_bucket{le="16.384"} 0
etcd_backend_read_seconds_bucket{le="32.768"} 0
etcd_backend_read_seconds_bucket{le="+Inf"} 0
etcd_backend_read_seconds_sum 0
etcd_backend_read_seconds_count 0
# HELP etcd_backend_tx_requests Number of transaction requests to the database
# TYPE etcd_backend_tx_requests counter
etcd_backend_tx_requests 0
# HELP etcd_backend_tx_seconds Latency for etcd transaction operations
# TYPE etcd_backend_tx_seconds histogram
etcd_backend_tx_seconds_bucket{le="0.001"} 0
etcd_backend_tx_seconds_bucket{le="0.002"} 0
etcd_backend_tx_seconds_bucket{le="0.004"} 0
etcd_backend_tx_seconds_bucket{le="0.008"} 0
etcd_backend_tx_seconds_bucket{le="0.016"} 0
etcd_backend_tx_seconds_bucket{le="0.032"} 0
etcd_backend_tx_seconds_bucket{le="0.064"} 0
etcd_backend_tx_seconds_bucket{le="0.128"} 0
etcd_backend_tx_seconds_bucket{le="0.256"} 0
etcd_backend_tx_seconds_bucket{le="0.512"} 0
etcd_backend_tx_seconds_bucket{le="1.024"} 0
etcd_backend_tx_seconds_bucket{le="2.048"} 0
etcd_backend_tx_seconds_bucket{le="4.096"} 0
etcd_backend_tx_seconds_bucket{le="8.192"} 0
etcd_backend_tx_seconds_bucket{le="16.384"} 0
etcd_backend_tx_seconds_bucket{le="32.768"} 0
etcd_backend_tx_seconds_bucket{le="+Inf"} 0
etcd_backend_tx_seconds_sum 0
etcd_backend_tx_seconds_count 0
# HELP etcd_backend_write_requests Number of wrtie requests to the database
# TYPE etcd_backend_write_requests counter
etcd_backend_write_requests 0
# HELP etcd_backend_write_seconds Latency for etcd write operations
# TYPE etcd_backend_write_seconds histogram
etcd_backend_write_seconds_bucket{le="0.001"} 0
etcd_backend_write_seconds_bucket{le="0.002"} 0
etcd_backend_write_seconds_bucket{le="0.004"} 0
etcd_backend_write_seconds_bucket{le="0.008"} 0
etcd_backend_write_seconds_bucket{le="0.016"} 0
etcd_backend_write_seconds_bucket{le="0.032"} 0
etcd_backend_write_seconds_bucket{le="0.064"} 0
etcd_backend_write_seconds_bucket{le="0.128"} 0
etcd_backend_write_seconds_bucket{le="0.256"} 0
etcd_backend_write_seconds_bucket{le="0.512"} 0
etcd_backend_write_seconds_bucket{le="1.024"} 0
etcd_backend_write_seconds_bucket{le="2.048"} 0
etcd_backend_write_seconds_bucket{le="4.096"} 0
etcd_backend_write_seconds_bucket{le="8.192"} 0
etcd_backend_write_seconds_bucket{le="16.384"} 0
etcd_backend_write_seconds_bucket{le="32.768"} 0
etcd_backend_write_seconds_bucket{le="+Inf"} 0
etcd_backend_write_seconds_sum 0
etcd_backend_write_seconds_count 0
# HELP firestore_events_backend_batch_read_requests Number of batch read requests to firestore events
# TYPE firestore_events_backend_batch_read_requests counter
firestore_events_backend_batch_read_requests 7
# HELP firestore_events_backend_batch_read_seconds Latency for firestore events batch read operations
# TYPE firestore_events_backend_batch_read_seconds histogram
firestore_events_backend_batch_read_seconds_bucket{le="0.001"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.002"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.004"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.008"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.016"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.032"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.064"} 3
firestore_events_backend_batch_read_seconds_bucket{le="0.128"} 7
firestore_events_backend_batch_read_seconds_bucket{le="0.256"} 7
firestore_events_backend_batch_read_seconds_bucket{le="0.512"} 7
firestore_events_backend_batch_read_seconds_bucket{le="1.024"} 7
firestore_events_backend_batch_read_seconds_bucket{le="2.048"} 7
firestore_events_backend_batch_read_seconds_bucket{le="4.096"} 7
firestore_events_backend_batch_read_seconds_bucket{le="8.192"} 7
firestore_events_backend_batch_read_seconds_bucket{le="16.384"} 7
firestore_events_backend_batch_read_seconds_bucket{le="32.768"} 7
firestore_events_backend_batch_read_seconds_bucket{le="+Inf"} 7
firestore_events_backend_batch_read_seconds_sum 0.5001422440000001
firestore_events_backend_batch_read_seconds_count 7
# HELP firestore_events_backend_batch_write_requests Number of batch write requests to firestore events
# TYPE firestore_events_backend_batch_write_requests counter
firestore_events_backend_batch_write_requests 14
# HELP firestore_events_backend_batch_write_seconds Latency for firestore events batch write operations
# TYPE firestore_events_backend_batch_write_seconds histogram
firestore_events_backend_batch_write_seconds_bucket{le="0.001"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.002"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.004"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.008"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.016"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.032"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.064"} 2
firestore_events_backend_batch_write_seconds_bucket{le="0.128"} 13
firestore_events_backend_batch_write_seconds_bucket{le="0.256"} 14
firestore_events_backend_batch_write_seconds_bucket{le="0.512"} 14
firestore_events_backend_batch_write_seconds_bucket{le="1.024"} 14
firestore_events_backend_batch_write_seconds_bucket{le="2.048"} 14
firestore_events_backend_batch_write_seconds_bucket{le="4.096"} 14
firestore_events_backend_batch_write_seconds_bucket{le="8.192"} 14
firestore_events_backend_batch_write_seconds_bucket{le="16.384"} 14
firestore_events_backend_batch_write_seconds_bucket{le="32.768"} 14
firestore_events_backend_batch_write_seconds_bucket{le="+Inf"} 14
firestore_events_backend_batch_write_seconds_sum 1.293583958
firestore_events_backend_batch_write_seconds_count 14
# HELP firestore_events_backend_write_requests Number of write requests to firestore events
# TYPE firestore_events_backend_write_requests counter
firestore_events_backend_write_requests 2
# HELP firestore_events_backend_write_seconds Latency for firestore events write operations
# TYPE firestore_events_backend_write_seconds histogram
firestore_events_backend_write_seconds_bucket{le="0.001"} 0
firestore_events_backend_write_seconds_bucket{le="0.002"} 0
firestore_events_backend_write_seconds_bucket{le="0.004"} 0
firestore_events_backend_write_seconds_bucket{le="0.008"} 0
firestore_events_backend_write_seconds_bucket{le="0.016"} 0
firestore_events_backend_write_seconds_bucket{le="0.032"} 0
firestore_events_backend_write_seconds_bucket{le="0.064"} 0
firestore_events_backend_write_seconds_bucket{le="0.128"} 1
firestore_events_backend_write_seconds_bucket{le="0.256"} 1
firestore_events_backend_write_seconds_bucket{le="0.512"} 2
firestore_events_backend_write_seconds_bucket{le="1.024"} 2
firestore_events_backend_write_seconds_bucket{le="2.048"} 2
firestore_events_backend_write_seconds_bucket{le="4.096"} 2
firestore_events_backend_write_seconds_bucket{le="8.192"} 2
firestore_events_backend_write_seconds_bucket{le="16.384"} 2
firestore_events_backend_write_seconds_bucket{le="32.768"} 2
firestore_events_backend_write_seconds_bucket{le="+Inf"} 2
firestore_events_backend_write_seconds_sum 0.36986882
firestore_events_backend_write_seconds_count 2
# HELP gcs_event_storage_downloads Number of downloads from the GCS backend
# TYPE gcs_event_storage_downloads counter
gcs_event_storage_downloads 0
# HELP gcs_event_storage_downloads_seconds Latency for GCS download operations
# TYPE gcs_event_storage_downloads_seconds histogram
gcs_event_storage_downloads_seconds_bucket{le="0.001"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.002"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.004"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.008"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.016"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.032"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.064"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.128"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.256"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.512"} 0
gcs_event_storage_downloads_seconds_bucket{le="1.024"} 0
gcs_event_storage_downloads_seconds_bucket{le="2.048"} 0
gcs_event_storage_downloads_seconds_bucket{le="4.096"} 0
gcs_event_storage_downloads_seconds_bucket{le="8.192"} 0
gcs_event_storage_downloads_seconds_bucket{le="16.384"} 0
gcs_event_storage_downloads_seconds_bucket{le="32.768"} 0
gcs_event_storage_downloads_seconds_bucket{le="+Inf"} 0
gcs_event_storage_downloads_seconds_sum 0
gcs_event_storage_downloads_seconds_count 0
# HELP gcs_event_storage_uploads Number of uploads to the GCS backend
# TYPE gcs_event_storage_uploads counter
gcs_event_storage_uploads 0
# HELP gcs_event_storage_uploads_seconds Latency for GCS upload operations
# TYPE gcs_event_storage_uploads_seconds histogram
gcs_event_storage_uploads_seconds_bucket{le="0.001"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.002"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.004"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.008"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.016"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.032"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.064"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.128"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.256"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.512"} 0
gcs_event_storage_uploads_seconds_bucket{le="1.024"} 0
gcs_event_storage_uploads_seconds_bucket{le="2.048"} 0
gcs_event_storage_uploads_seconds_bucket{le="4.096"} 0
gcs_event_storage_uploads_seconds_bucket{le="8.192"} 0
gcs_event_storage_uploads_seconds_bucket{le="16.384"} 0
gcs_event_storage_uploads_seconds_bucket{le="32.768"} 0
gcs_event_storage_uploads_seconds_bucket{le="+Inf"} 0
gcs_event_storage_uploads_seconds_sum 0
gcs_event_storage_uploads_seconds_count 0
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 1.1855e-05
go_gc_duration_seconds{quantile="0.25"} 2.3168e-05
go_gc_duration_seconds{quantile="0.5"} 2.8492e-05
go_gc_duration_seconds{quantile="0.75"} 4.2054e-05
go_gc_duration_seconds{quantile="1"} 0.000201878
go_gc_duration_seconds_sum 0.002210812
go_gc_duration_seconds_count 54
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 381
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.12.7"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.674728e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 4.4393692e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.652192e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 4.711685e+06
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 6.643587413729206e-05
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.63168e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.674728e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 1.8825216e+07
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 4.3892736e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 171191
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.2717952e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.563806766329105e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 4.882876e+06
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 13888
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 462384
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 557056
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 5.0030032e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.565464e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 4.390912e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 4.390912e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.353164e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 32
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 1569.894
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 1863.958
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 1863.958
http_request_duration_microseconds_sum{handler="prometheus"} 6918.215
http_request_duration_microseconds_count{handler="prometheus"} 4
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="prometheus",quantile="0.5"} 325
http_request_size_bytes{handler="prometheus",quantile="0.9"} 325
http_request_size_bytes{handler="prometheus",quantile="0.99"} 325
http_request_size_bytes_sum{handler="prometheus"} 1300
http_request_size_bytes_count{handler="prometheus"} 4
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 4
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 3759
http_response_size_bytes{handler="prometheus",quantile="0.9"} 3793
http_response_size_bytes{handler="prometheus",quantile="0.99"} 3793
http_response_size_bytes_sum{handler="prometheus"} 15076
http_response_size_bytes_count{handler="prometheus"} 4
# HELP remote_clusters Number inbound connections from remote clusters and clusters stats
# TYPE remote_clusters gauge
remote_clusters{cluster="C02WG09CHTDH"} 0
# HELP rx Number of bytes received.
# TYPE rx counter
rx 4527
# HELP server_interactive_sessions_total Number of active sessions
# TYPE server_interactive_sessions_total gauge
server_interactive_sessions_total 5
# HELP tx Number of bytes transmitted.
# TYPE tx counter
tx 4038

The prometheus metrics for the same activity PLUS the 100_000 junk records are:

# HELP audit_failed_disk_monitoring Number of times disk monitoring failed.
# TYPE audit_failed_disk_monitoring counter
audit_failed_disk_monitoring 0
# HELP audit_failed_emit_events Number of times emitting audit event failed.
# TYPE audit_failed_emit_events counter
audit_failed_emit_events 0
# HELP audit_percentage_disk_space_used Percentage disk space used.
# TYPE audit_percentage_disk_space_used gauge
audit_percentage_disk_space_used 16
# HELP audit_server_open_files Number of open audit files
# TYPE audit_server_open_files gauge
audit_server_open_files -1
# HELP auth_generate_requests Number of current generate requests
# TYPE auth_generate_requests gauge
auth_generate_requests 0
# HELP auth_generate_requests_throttled_total Number of throttled requests to generate new server keys
# TYPE auth_generate_requests_throttled_total counter
auth_generate_requests_throttled_total 0
# HELP auth_generate_requests_total Number of requests to generate new server keys
# TYPE auth_generate_requests_total counter
auth_generate_requests_total 0
# HELP auth_generate_seconds Latency for generate requests
# TYPE auth_generate_seconds histogram
auth_generate_seconds_bucket{le="0.001"} 0
auth_generate_seconds_bucket{le="0.002"} 0
auth_generate_seconds_bucket{le="0.004"} 0
auth_generate_seconds_bucket{le="0.008"} 0
auth_generate_seconds_bucket{le="0.016"} 0
auth_generate_seconds_bucket{le="0.032"} 0
auth_generate_seconds_bucket{le="0.064"} 0
auth_generate_seconds_bucket{le="0.128"} 0
auth_generate_seconds_bucket{le="0.256"} 0
auth_generate_seconds_bucket{le="0.512"} 0
auth_generate_seconds_bucket{le="1.024"} 0
auth_generate_seconds_bucket{le="2.048"} 0
auth_generate_seconds_bucket{le="4.096"} 0
auth_generate_seconds_bucket{le="8.192"} 0
auth_generate_seconds_bucket{le="16.384"} 0
auth_generate_seconds_bucket{le="32.768"} 0
auth_generate_seconds_bucket{le="+Inf"} 0
auth_generate_seconds_sum 0
auth_generate_seconds_count 0
# HELP backend_batch_read_requests_total Number of read requests to the backend
# TYPE backend_batch_read_requests_total counter
backend_batch_read_requests_total{component="backend"} 188
backend_batch_read_requests_total{component="cache"} 304
# HELP backend_batch_read_seconds Latency for batch read operations
# TYPE backend_batch_read_seconds histogram
backend_batch_read_seconds_bucket{component="backend",le="0.001"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.002"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.004"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.008"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.016"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.032"} 0
backend_batch_read_seconds_bucket{component="backend",le="0.064"} 88
backend_batch_read_seconds_bucket{component="backend",le="0.128"} 173
backend_batch_read_seconds_bucket{component="backend",le="0.256"} 183
backend_batch_read_seconds_bucket{component="backend",le="0.512"} 187
backend_batch_read_seconds_bucket{component="backend",le="1.024"} 188
backend_batch_read_seconds_bucket{component="backend",le="2.048"} 188
backend_batch_read_seconds_bucket{component="backend",le="4.096"} 188
backend_batch_read_seconds_bucket{component="backend",le="8.192"} 188
backend_batch_read_seconds_bucket{component="backend",le="16.384"} 188
backend_batch_read_seconds_bucket{component="backend",le="32.768"} 188
backend_batch_read_seconds_bucket{component="backend",le="+Inf"} 188
backend_batch_read_seconds_sum{component="backend"} 15.471601142000003
backend_batch_read_seconds_count{component="backend"} 188
backend_batch_read_seconds_bucket{component="cache",le="0.001"} 304
backend_batch_read_seconds_bucket{component="cache",le="0.002"} 304
backend_batch_read_seconds_bucket{component="cache",le="0.004"} 304
backend_batch_read_seconds_bucket{component="cache",le="0.008"} 304
backend_batch_read_seconds_bucket{component="cache",le="0.016"} 304
backend_batch_read_seconds_bucket{component="cache",le="0.032"} 304
backend_batch_read_seconds_bucket{component="cache",le="0.064"} 304
backend_batch_read_seconds_bucket{component="cache",le="0.128"} 304
backend_batch_read_seconds_bucket{component="cache",le="0.256"} 304
backend_batch_read_seconds_bucket{component="cache",le="0.512"} 304
backend_batch_read_seconds_bucket{component="cache",le="1.024"} 304
backend_batch_read_seconds_bucket{component="cache",le="2.048"} 304
backend_batch_read_seconds_bucket{component="cache",le="4.096"} 304
backend_batch_read_seconds_bucket{component="cache",le="8.192"} 304
backend_batch_read_seconds_bucket{component="cache",le="16.384"} 304
backend_batch_read_seconds_bucket{component="cache",le="32.768"} 304
backend_batch_read_seconds_bucket{component="cache",le="+Inf"} 304
backend_batch_read_seconds_sum{component="cache"} 0.083000583
backend_batch_read_seconds_count{component="cache"} 304
# HELP backend_batch_write_requests_total Number of batch write requests to the backend
# TYPE backend_batch_write_requests_total counter
backend_batch_write_requests_total{component="cache"} 70
# HELP backend_batch_write_seconds Latency for backend batch write operations
# TYPE backend_batch_write_seconds histogram
backend_batch_write_seconds_bucket{component="cache",le="0.001"} 64
backend_batch_write_seconds_bucket{component="cache",le="0.002"} 67
backend_batch_write_seconds_bucket{component="cache",le="0.004"} 69
backend_batch_write_seconds_bucket{component="cache",le="0.008"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.016"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.032"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.064"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.128"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.256"} 70
backend_batch_write_seconds_bucket{component="cache",le="0.512"} 70
backend_batch_write_seconds_bucket{component="cache",le="1.024"} 70
backend_batch_write_seconds_bucket{component="cache",le="2.048"} 70
backend_batch_write_seconds_bucket{component="cache",le="4.096"} 70
backend_batch_write_seconds_bucket{component="cache",le="8.192"} 70
backend_batch_write_seconds_bucket{component="cache",le="16.384"} 70
backend_batch_write_seconds_bucket{component="cache",le="32.768"} 70
backend_batch_write_seconds_bucket{component="cache",le="+Inf"} 70
backend_batch_write_seconds_sum{component="cache"} 0.036286640999999994
backend_batch_write_seconds_count{component="cache"} 70
# HELP backend_read_requests_total Number of read requests to the backend
# TYPE backend_read_requests_total counter
backend_read_requests_total{component="backend"} 1626
backend_read_requests_total{component="cache"} 2893
# HELP backend_read_seconds Latency for read operations
# TYPE backend_read_seconds histogram
backend_read_seconds_bucket{component="backend",le="0.001"} 1626
backend_read_seconds_bucket{component="backend",le="0.002"} 1626
backend_read_seconds_bucket{component="backend",le="0.004"} 1626
backend_read_seconds_bucket{component="backend",le="0.008"} 1626
backend_read_seconds_bucket{component="backend",le="0.016"} 1626
backend_read_seconds_bucket{component="backend",le="0.032"} 1626
backend_read_seconds_bucket{component="backend",le="0.064"} 1626
backend_read_seconds_bucket{component="backend",le="0.128"} 1626
backend_read_seconds_bucket{component="backend",le="0.256"} 1626
backend_read_seconds_bucket{component="backend",le="0.512"} 1626
backend_read_seconds_bucket{component="backend",le="1.024"} 1626
backend_read_seconds_bucket{component="backend",le="2.048"} 1626
backend_read_seconds_bucket{component="backend",le="4.096"} 1626
backend_read_seconds_bucket{component="backend",le="8.192"} 1626
backend_read_seconds_bucket{component="backend",le="16.384"} 1626
backend_read_seconds_bucket{component="backend",le="32.768"} 1626
backend_read_seconds_bucket{component="backend",le="+Inf"} 1626
backend_read_seconds_sum{component="backend"} 0.0034168239999999993
backend_read_seconds_count{component="backend"} 1626
backend_read_seconds_bucket{component="cache",le="0.001"} 2893
backend_read_seconds_bucket{component="cache",le="0.002"} 2893
backend_read_seconds_bucket{component="cache",le="0.004"} 2893
backend_read_seconds_bucket{component="cache",le="0.008"} 2893
backend_read_seconds_bucket{component="cache",le="0.016"} 2893
backend_read_seconds_bucket{component="cache",le="0.032"} 2893
backend_read_seconds_bucket{component="cache",le="0.064"} 2893
backend_read_seconds_bucket{component="cache",le="0.128"} 2893
backend_read_seconds_bucket{component="cache",le="0.256"} 2893
backend_read_seconds_bucket{component="cache",le="0.512"} 2893
backend_read_seconds_bucket{component="cache",le="1.024"} 2893
backend_read_seconds_bucket{component="cache",le="2.048"} 2893
backend_read_seconds_bucket{component="cache",le="4.096"} 2893
backend_read_seconds_bucket{component="cache",le="8.192"} 2893
backend_read_seconds_bucket{component="cache",le="16.384"} 2893
backend_read_seconds_bucket{component="cache",le="32.768"} 2893
backend_read_seconds_bucket{component="cache",le="+Inf"} 2893
backend_read_seconds_sum{component="cache"} 0.00754319799999999
backend_read_seconds_count{component="cache"} 2893
# HELP backend_watcher_queues_total Watcher queue sizes
# TYPE backend_watcher_queues_total gauge
backend_watcher_queues_total{component="auth:cache"} 0
# HELP backend_watchers_total Number of active backend watchers
# TYPE backend_watchers_total gauge
backend_watchers_total{component="backend"} 1
backend_watchers_total{component="cache"} 5
# HELP backend_write_requests_failed_total Number of failed write requests to the backend
# TYPE backend_write_requests_failed_total counter
backend_write_requests_failed_total{component="backend"} 2
# HELP backend_write_requests_total Number of write requests to the backend
# TYPE backend_write_requests_total counter
backend_write_requests_total{component="backend"} 877
backend_write_requests_total{component="cache"} 135
# HELP backend_write_seconds Latency for backend write operations
# TYPE backend_write_seconds histogram
backend_write_seconds_bucket{component="backend",le="0.001"} 0
backend_write_seconds_bucket{component="backend",le="0.002"} 0
backend_write_seconds_bucket{component="backend",le="0.004"} 0
backend_write_seconds_bucket{component="backend",le="0.008"} 0
backend_write_seconds_bucket{component="backend",le="0.016"} 0
backend_write_seconds_bucket{component="backend",le="0.032"} 0
backend_write_seconds_bucket{component="backend",le="0.064"} 1
backend_write_seconds_bucket{component="backend",le="0.128"} 460
backend_write_seconds_bucket{component="backend",le="0.256"} 846
backend_write_seconds_bucket{component="backend",le="0.512"} 867
backend_write_seconds_bucket{component="backend",le="1.024"} 877
backend_write_seconds_bucket{component="backend",le="2.048"} 877
backend_write_seconds_bucket{component="backend",le="4.096"} 877
backend_write_seconds_bucket{component="backend",le="8.192"} 877
backend_write_seconds_bucket{component="backend",le="16.384"} 877
backend_write_seconds_bucket{component="backend",le="32.768"} 877
backend_write_seconds_bucket{component="backend",le="+Inf"} 877
backend_write_seconds_sum{component="backend"} 126.3226184850002
backend_write_seconds_count{component="backend"} 877
backend_write_seconds_bucket{component="cache",le="0.001"} 81
backend_write_seconds_bucket{component="cache",le="0.002"} 98
backend_write_seconds_bucket{component="cache",le="0.004"} 111
backend_write_seconds_bucket{component="cache",le="0.008"} 135
backend_write_seconds_bucket{component="cache",le="0.016"} 135
backend_write_seconds_bucket{component="cache",le="0.032"} 135
backend_write_seconds_bucket{component="cache",le="0.064"} 135
backend_write_seconds_bucket{component="cache",le="0.128"} 135
backend_write_seconds_bucket{component="cache",le="0.256"} 135
backend_write_seconds_bucket{component="cache",le="0.512"} 135
backend_write_seconds_bucket{component="cache",le="1.024"} 135
backend_write_seconds_bucket{component="cache",le="2.048"} 135
backend_write_seconds_bucket{component="cache",le="4.096"} 135
backend_write_seconds_bucket{component="cache",le="8.192"} 135
backend_write_seconds_bucket{component="cache",le="16.384"} 135
backend_write_seconds_bucket{component="cache",le="32.768"} 135
backend_write_seconds_bucket{component="cache",le="+Inf"} 135
backend_write_seconds_sum{component="cache"} 0.25040869699999996
backend_write_seconds_count{component="cache"} 135
# HELP etcd_backend_batch_read_requests Number of read requests to the database
# TYPE etcd_backend_batch_read_requests counter
etcd_backend_batch_read_requests 0
# HELP etcd_backend_batch_read_seconds Latency for etcd read operations
# TYPE etcd_backend_batch_read_seconds histogram
etcd_backend_batch_read_seconds_bucket{le="0.001"} 0
etcd_backend_batch_read_seconds_bucket{le="0.002"} 0
etcd_backend_batch_read_seconds_bucket{le="0.004"} 0
etcd_backend_batch_read_seconds_bucket{le="0.008"} 0
etcd_backend_batch_read_seconds_bucket{le="0.016"} 0
etcd_backend_batch_read_seconds_bucket{le="0.032"} 0
etcd_backend_batch_read_seconds_bucket{le="0.064"} 0
etcd_backend_batch_read_seconds_bucket{le="0.128"} 0
etcd_backend_batch_read_seconds_bucket{le="0.256"} 0
etcd_backend_batch_read_seconds_bucket{le="0.512"} 0
etcd_backend_batch_read_seconds_bucket{le="1.024"} 0
etcd_backend_batch_read_seconds_bucket{le="2.048"} 0
etcd_backend_batch_read_seconds_bucket{le="4.096"} 0
etcd_backend_batch_read_seconds_bucket{le="8.192"} 0
etcd_backend_batch_read_seconds_bucket{le="16.384"} 0
etcd_backend_batch_read_seconds_bucket{le="32.768"} 0
etcd_backend_batch_read_seconds_bucket{le="+Inf"} 0
etcd_backend_batch_read_seconds_sum 0
etcd_backend_batch_read_seconds_count 0
# HELP etcd_backend_read_requests Number of read requests to the database
# TYPE etcd_backend_read_requests counter
etcd_backend_read_requests 0
# HELP etcd_backend_read_seconds Latency for etcd read operations
# TYPE etcd_backend_read_seconds histogram
etcd_backend_read_seconds_bucket{le="0.001"} 0
etcd_backend_read_seconds_bucket{le="0.002"} 0
etcd_backend_read_seconds_bucket{le="0.004"} 0
etcd_backend_read_seconds_bucket{le="0.008"} 0
etcd_backend_read_seconds_bucket{le="0.016"} 0
etcd_backend_read_seconds_bucket{le="0.032"} 0
etcd_backend_read_seconds_bucket{le="0.064"} 0
etcd_backend_read_seconds_bucket{le="0.128"} 0
etcd_backend_read_seconds_bucket{le="0.256"} 0
etcd_backend_read_seconds_bucket{le="0.512"} 0
etcd_backend_read_seconds_bucket{le="1.024"} 0
etcd_backend_read_seconds_bucket{le="2.048"} 0
etcd_backend_read_seconds_bucket{le="4.096"} 0
etcd_backend_read_seconds_bucket{le="8.192"} 0
etcd_backend_read_seconds_bucket{le="16.384"} 0
etcd_backend_read_seconds_bucket{le="32.768"} 0
etcd_backend_read_seconds_bucket{le="+Inf"} 0
etcd_backend_read_seconds_sum 0
etcd_backend_read_seconds_count 0
# HELP etcd_backend_tx_requests Number of transaction requests to the database
# TYPE etcd_backend_tx_requests counter
etcd_backend_tx_requests 0
# HELP etcd_backend_tx_seconds Latency for etcd transaction operations
# TYPE etcd_backend_tx_seconds histogram
etcd_backend_tx_seconds_bucket{le="0.001"} 0
etcd_backend_tx_seconds_bucket{le="0.002"} 0
etcd_backend_tx_seconds_bucket{le="0.004"} 0
etcd_backend_tx_seconds_bucket{le="0.008"} 0
etcd_backend_tx_seconds_bucket{le="0.016"} 0
etcd_backend_tx_seconds_bucket{le="0.032"} 0
etcd_backend_tx_seconds_bucket{le="0.064"} 0
etcd_backend_tx_seconds_bucket{le="0.128"} 0
etcd_backend_tx_seconds_bucket{le="0.256"} 0
etcd_backend_tx_seconds_bucket{le="0.512"} 0
etcd_backend_tx_seconds_bucket{le="1.024"} 0
etcd_backend_tx_seconds_bucket{le="2.048"} 0
etcd_backend_tx_seconds_bucket{le="4.096"} 0
etcd_backend_tx_seconds_bucket{le="8.192"} 0
etcd_backend_tx_seconds_bucket{le="16.384"} 0
etcd_backend_tx_seconds_bucket{le="32.768"} 0
etcd_backend_tx_seconds_bucket{le="+Inf"} 0
etcd_backend_tx_seconds_sum 0
etcd_backend_tx_seconds_count 0
# HELP etcd_backend_write_requests Number of wrtie requests to the database
# TYPE etcd_backend_write_requests counter
etcd_backend_write_requests 0
# HELP etcd_backend_write_seconds Latency for etcd write operations
# TYPE etcd_backend_write_seconds histogram
etcd_backend_write_seconds_bucket{le="0.001"} 0
etcd_backend_write_seconds_bucket{le="0.002"} 0
etcd_backend_write_seconds_bucket{le="0.004"} 0
etcd_backend_write_seconds_bucket{le="0.008"} 0
etcd_backend_write_seconds_bucket{le="0.016"} 0
etcd_backend_write_seconds_bucket{le="0.032"} 0
etcd_backend_write_seconds_bucket{le="0.064"} 0
etcd_backend_write_seconds_bucket{le="0.128"} 0
etcd_backend_write_seconds_bucket{le="0.256"} 0
etcd_backend_write_seconds_bucket{le="0.512"} 0
etcd_backend_write_seconds_bucket{le="1.024"} 0
etcd_backend_write_seconds_bucket{le="2.048"} 0
etcd_backend_write_seconds_bucket{le="4.096"} 0
etcd_backend_write_seconds_bucket{le="8.192"} 0
etcd_backend_write_seconds_bucket{le="16.384"} 0
etcd_backend_write_seconds_bucket{le="32.768"} 0
etcd_backend_write_seconds_bucket{le="+Inf"} 0
etcd_backend_write_seconds_sum 0
etcd_backend_write_seconds_count 0
# HELP firestore_events_backend_batch_read_requests Number of batch read requests to firestore events
# TYPE firestore_events_backend_batch_read_requests counter
firestore_events_backend_batch_read_requests 8
# HELP firestore_events_backend_batch_read_seconds Latency for firestore events batch read operations
# TYPE firestore_events_backend_batch_read_seconds histogram
firestore_events_backend_batch_read_seconds_bucket{le="0.001"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.002"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.004"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.008"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.016"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.032"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.064"} 0
firestore_events_backend_batch_read_seconds_bucket{le="0.128"} 8
firestore_events_backend_batch_read_seconds_bucket{le="0.256"} 8
firestore_events_backend_batch_read_seconds_bucket{le="0.512"} 8
firestore_events_backend_batch_read_seconds_bucket{le="1.024"} 8
firestore_events_backend_batch_read_seconds_bucket{le="2.048"} 8
firestore_events_backend_batch_read_seconds_bucket{le="4.096"} 8
firestore_events_backend_batch_read_seconds_bucket{le="8.192"} 8
firestore_events_backend_batch_read_seconds_bucket{le="16.384"} 8
firestore_events_backend_batch_read_seconds_bucket{le="32.768"} 8
firestore_events_backend_batch_read_seconds_bucket{le="+Inf"} 8
firestore_events_backend_batch_read_seconds_sum 0.694322218
firestore_events_backend_batch_read_seconds_count 8
# HELP firestore_events_backend_batch_write_requests Number of batch write requests to firestore events
# TYPE firestore_events_backend_batch_write_requests counter
firestore_events_backend_batch_write_requests 18
# HELP firestore_events_backend_batch_write_seconds Latency for firestore events batch write operations
# TYPE firestore_events_backend_batch_write_seconds histogram
firestore_events_backend_batch_write_seconds_bucket{le="0.001"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.002"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.004"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.008"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.016"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.032"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.064"} 0
firestore_events_backend_batch_write_seconds_bucket{le="0.128"} 13
firestore_events_backend_batch_write_seconds_bucket{le="0.256"} 18
firestore_events_backend_batch_write_seconds_bucket{le="0.512"} 18
firestore_events_backend_batch_write_seconds_bucket{le="1.024"} 18
firestore_events_backend_batch_write_seconds_bucket{le="2.048"} 18
firestore_events_backend_batch_write_seconds_bucket{le="4.096"} 18
firestore_events_backend_batch_write_seconds_bucket{le="8.192"} 18
firestore_events_backend_batch_write_seconds_bucket{le="16.384"} 18
firestore_events_backend_batch_write_seconds_bucket{le="32.768"} 18
firestore_events_backend_batch_write_seconds_bucket{le="+Inf"} 18
firestore_events_backend_batch_write_seconds_sum 1.896814674
firestore_events_backend_batch_write_seconds_count 18
# HELP firestore_events_backend_write_requests Number of write requests to firestore events
# TYPE firestore_events_backend_write_requests counter
firestore_events_backend_write_requests 6
# HELP firestore_events_backend_write_seconds Latency for firestore events write operations
# TYPE firestore_events_backend_write_seconds histogram
firestore_events_backend_write_seconds_bucket{le="0.001"} 0
firestore_events_backend_write_seconds_bucket{le="0.002"} 0
firestore_events_backend_write_seconds_bucket{le="0.004"} 0
firestore_events_backend_write_seconds_bucket{le="0.008"} 0
firestore_events_backend_write_seconds_bucket{le="0.016"} 0
firestore_events_backend_write_seconds_bucket{le="0.032"} 0
firestore_events_backend_write_seconds_bucket{le="0.064"} 0
firestore_events_backend_write_seconds_bucket{le="0.128"} 3
firestore_events_backend_write_seconds_bucket{le="0.256"} 6
firestore_events_backend_write_seconds_bucket{le="0.512"} 6
firestore_events_backend_write_seconds_bucket{le="1.024"} 6
firestore_events_backend_write_seconds_bucket{le="2.048"} 6
firestore_events_backend_write_seconds_bucket{le="4.096"} 6
firestore_events_backend_write_seconds_bucket{le="8.192"} 6
firestore_events_backend_write_seconds_bucket{le="16.384"} 6
firestore_events_backend_write_seconds_bucket{le="32.768"} 6
firestore_events_backend_write_seconds_bucket{le="+Inf"} 6
firestore_events_backend_write_seconds_sum 0.8536411470000002
firestore_events_backend_write_seconds_count 6
# HELP gcs_event_storage_downloads Number of downloads from the GCS backend
# TYPE gcs_event_storage_downloads counter
gcs_event_storage_downloads 0
# HELP gcs_event_storage_downloads_seconds Latency for GCS download operations
# TYPE gcs_event_storage_downloads_seconds histogram
gcs_event_storage_downloads_seconds_bucket{le="0.001"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.002"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.004"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.008"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.016"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.032"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.064"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.128"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.256"} 0
gcs_event_storage_downloads_seconds_bucket{le="0.512"} 0
gcs_event_storage_downloads_seconds_bucket{le="1.024"} 0
gcs_event_storage_downloads_seconds_bucket{le="2.048"} 0
gcs_event_storage_downloads_seconds_bucket{le="4.096"} 0
gcs_event_storage_downloads_seconds_bucket{le="8.192"} 0
gcs_event_storage_downloads_seconds_bucket{le="16.384"} 0
gcs_event_storage_downloads_seconds_bucket{le="32.768"} 0
gcs_event_storage_downloads_seconds_bucket{le="+Inf"} 0
gcs_event_storage_downloads_seconds_sum 0
gcs_event_storage_downloads_seconds_count 0
# HELP gcs_event_storage_uploads Number of uploads to the GCS backend
# TYPE gcs_event_storage_uploads counter
gcs_event_storage_uploads 0
# HELP gcs_event_storage_uploads_seconds Latency for GCS upload operations
# TYPE gcs_event_storage_uploads_seconds histogram
gcs_event_storage_uploads_seconds_bucket{le="0.001"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.002"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.004"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.008"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.016"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.032"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.064"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.128"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.256"} 0
gcs_event_storage_uploads_seconds_bucket{le="0.512"} 0
gcs_event_storage_uploads_seconds_bucket{le="1.024"} 0
gcs_event_storage_uploads_seconds_bucket{le="2.048"} 0
gcs_event_storage_uploads_seconds_bucket{le="4.096"} 0
gcs_event_storage_uploads_seconds_bucket{le="8.192"} 0
gcs_event_storage_uploads_seconds_bucket{le="16.384"} 0
gcs_event_storage_uploads_seconds_bucket{le="32.768"} 0
gcs_event_storage_uploads_seconds_bucket{le="+Inf"} 0
gcs_event_storage_uploads_seconds_sum 0
gcs_event_storage_uploads_seconds_count 0
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 8.99e-06
go_gc_duration_seconds{quantile="0.25"} 1.4424e-05
go_gc_duration_seconds{quantile="0.5"} 2.2126e-05
go_gc_duration_seconds{quantile="0.75"} 3.2661e-05
go_gc_duration_seconds{quantile="1"} 0.000160642
go_gc_duration_seconds_sum 0.001752287
go_gc_duration_seconds_count 53
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 229
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.12.7"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 5.3468128e+07
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 4.50613008e+08
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 1.63228e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 4.795068e+06
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 5.7552149064962624e-05
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 2.639872e+06
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 5.3468128e+07
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 5.472256e+06
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 5.7245696e+07
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 334068
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 0
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 6.2717952e+07
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.563811319909208e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 5.129136e+06
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 13888
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 16384
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 584352
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 655360
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 5.7333232e+07
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.47888e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 4.390912e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 4.390912e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 7.353164e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 31
# HELP http_request_duration_microseconds The HTTP request latencies in microseconds.
# TYPE http_request_duration_microseconds summary
http_request_duration_microseconds{handler="prometheus",quantile="0.5"} 1607.427
http_request_duration_microseconds{handler="prometheus",quantile="0.9"} 2152.303
http_request_duration_microseconds{handler="prometheus",quantile="0.99"} 2223.088
http_request_duration_microseconds_sum{handler="prometheus"} 27694.036
http_request_duration_microseconds_count{handler="prometheus"} 17
# HELP http_request_size_bytes The HTTP request sizes in bytes.
# TYPE http_request_size_bytes summary
http_request_size_bytes{handler="prometheus",quantile="0.5"} 325
http_request_size_bytes{handler="prometheus",quantile="0.9"} 325
http_request_size_bytes{handler="prometheus",quantile="0.99"} 325
http_request_size_bytes_sum{handler="prometheus"} 5525
http_request_size_bytes_count{handler="prometheus"} 17
# HELP http_requests_total Total number of HTTP requests made.
# TYPE http_requests_total counter
http_requests_total{code="200",handler="prometheus",method="get"} 17
# HELP http_response_size_bytes The HTTP response sizes in bytes.
# TYPE http_response_size_bytes summary
http_response_size_bytes{handler="prometheus",quantile="0.5"} 3800
http_response_size_bytes{handler="prometheus",quantile="0.9"} 3811
http_response_size_bytes{handler="prometheus",quantile="0.99"} 3815
http_response_size_bytes_sum{handler="prometheus"} 64460
http_response_size_bytes_count{handler="prometheus"} 17
# HELP remote_clusters Number inbound connections from remote clusters and clusters stats
# TYPE remote_clusters gauge
remote_clusters{cluster="C02WG09CHTDH"} 0
# HELP rx Number of bytes received.
# TYPE rx counter
rx 83799
# HELP server_interactive_sessions_total Number of active sessions
# TYPE server_interactive_sessions_total gauge
server_interactive_sessions_total 5
# HELP tx Number of bytes transmitted.
# TYPE tx counter
tx 610546

The code for generating records:

// Returns an int >= min, < max
func randomInt(min, max int) int {
	return min + rand.Intn(max-min)
}

// Generate a random string of A-Z chars with len = l
func randomString(len int) string {
	bytes := make([]byte, len)
	for i := 0; i < len; i++ {
		bytes[i] = byte(randomInt(65, 90))
	}
	return string(bytes)
}

func (b *FirestoreBackend) batchInsertTestData() {

	rand.Seed(time.Now().UnixNano())

	batch := b.svc.Batch()

	outerCounter := 0
	innerCounter := 0

	for {
		if outerCounter >= 100000 {
			break
		}

		if innerCounter >= 490 {
			b.Logger.Infof("Writing outer counter: %v, inner counter: %v ...", outerCounter, innerCounter)
			_, _ = batch.Commit(b.clientContext)
			innerCounter = 0
		}

		slices := make([]string, 0)
		slicesNo := randomInt(1, 8)
		sliceCount := 0

		for {

			if sliceCount >= slicesNo {
				break
			}

			sliceCount = sliceCount + 1

			slices = append(slices, randomString(randomInt(5, 12)))
		}

		key := strings.Join(slices, documentNameReplacementCharacter)

		var r record
		r.Key = key
		r.Value = string(randomString(randomInt(30, 200)))
		r.Timestamp = b.clock.Now().UTC().Unix()
		r.ID = b.clock.Now().UTC().UnixNano()

		batch.Create(b.svc.Collection(b.CollectionName).Doc(key), r)

		outerCounter = outerCounter + 1
		innerCounter = innerCounter + 1
	}
}
b := &FirestoreBackend{
svc: firestoreClient,
Entry: l,
BKConfig: *cfg,

This comment has been minimized.

Copy link
@klizhentas

klizhentas Jul 16, 2019

Contributor

use gofmt

This comment has been minimized.

Copy link
@klizhentas

klizhentas Jul 16, 2019

Contributor

and goimports :)


if !doc.Exists() {
return trace.NotFound("key %s does not exist", string(key))
} else {

This comment has been minimized.

Copy link
@klizhentas

klizhentas Jul 16, 2019

Contributor

because here you return, you don't need else clause

This comment has been minimized.

"github.com/gravitational/teleport"
"github.com/gravitational/teleport/lib/session"

"cloud.google.com/go/storage"

This comment has been minimized.

Copy link
@klizhentas

klizhentas Jul 16, 2019

Contributor

(
"std packages"

"teleport packages"

 "vendored stuff"

)

@@ -2415,4 +2441,4 @@ func initSelfSignedHTTPSCert(cfg *Config) (err error) {
return trace.Wrap(err, "error writing key PEM")
}
return nil
}
}

This comment has been minimized.

Copy link
@klizhentas

klizhentas Jul 16, 2019

Contributor

missing newlines at the end of file

@joshdurbin joshdurbin force-pushed the bigcommerce:gcp_ha_support branch 6 times, most recently from 5dcac26 to c8faefe Jul 20, 2019
@klizhentas klizhentas added this to the 4.2 "Alameda" milestone Jul 25, 2019
@joshdurbin joshdurbin force-pushed the bigcommerce:gcp_ha_support branch from 657f7c7 to e2d24e1 Aug 1, 2019
@joshdurbin joshdurbin force-pushed the bigcommerce:gcp_ha_support branch 2 times, most recently from 4959245 to 7b632ba Aug 16, 2019
…store-backed events, and firestore backend storage
@joshdurbin joshdurbin force-pushed the bigcommerce:gcp_ha_support branch from 762d6a0 to d346f2b Sep 5, 2019
@klizhentas klizhentas merged commit d346f2b into gravitational:master Sep 24, 2019
@benarent benarent mentioned this pull request Nov 9, 2019
4 of 8 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.