Permalink
Browse files

Remove LRU cache from Alpha (#2837)

The LRU cache in Alpha has been causing a lot of contention. This PR removes the cache altogether, so posting lists are read directly from Badger as needed. Instead of one global LRU cache, each Txn object has a local cache, which is used both for mutations and for queries. Query functions share this local cache.

Pros: We have seen an astounding 5-10x query latency drops due to this PR! I've also seen 1.7x jump in mutation throughput while running the live loader.

Cons: This PR introduces a query local cache, which can potentially cause issues if a query requests for more data than the RAM available in the server. However, that can be dealt with in future PRs.

The `lru_mb` flag has not been removed yet. We intend to use it later when we introduce an LRU cache in Badger.

Changes:
* Remove LRU cache.
* Make query after mutation before commit work. Make tests work.
* Return list from txn.Store
* Fix the query package test failure. Also Txn should always use a local cache.
* No need to commit in memory
* Bring GetNoStore back
* No need to track keys during rolling up.
* Make LocalCache use byte slice instead of string as argument.
  • Loading branch information...
manishrjain committed Dec 31, 2018
1 parent 787799b commit b9990f4619b64615c2c18bb7627d198b4397109c
@@ -13,7 +13,7 @@ services:
- 6080:6080
environment:
- GOPATH=$GOPATH
command: /gobin/dgraph zero --my=zero1:5080 --replicas 3 --idx 1 --bindall --expose_trace --profile_mode block --block_rate 10 --logtostderr -v=2 --jaeger.collector http://jaeger:14268
command: /gobin/dgraph zero --my=zero1:5080 --replicas 3 --idx 1 --bindall --expose_trace --profile_mode block --block_rate 10 --logtostderr -v=2 # --jaeger.collector http://jaeger:14268
volumes:
- type: bind
source: $GOPATH/bin
@@ -35,7 +35,7 @@ services:
- 9999:9999
security_opt:
- seccomp:unconfined
command: /gobin/dgraph alpha --my=dg1:7180 --lru_mb=1024 --zero=zero1:5080 --expose_trace -o 100 --trace 1.0 --profile_mode block --block_rate 10 --logtostderr -v=3 --jaeger.collector http://jaeger:14268
command: /gobin/dgraph alpha --my=dg1:7180 --lru_mb=1024 --zero=zero1:5080 --expose_trace -o 100 --trace 1.0 --profile_mode block --block_rate 10 --logtostderr -v=3 # --jaeger.collector http://jaeger:14268
#command: /gobin/dlv --listen=:9999 --headless=true --api-version=2 --init $GOPATH/src/github.com/dgraph-io/dgraph/dgraph/dlv.init exec /gobin/dgraph -- alpha --my=dg1:7180 --lru_mb=1024 --zero=zero1:5080 -o 100 --expose_trace --trace 1.0 --profile_mode block --block_rate 10 --logtostderr -v=3 --jaeger.collector http://jaeger:14268

jaeger:
@@ -53,34 +53,31 @@ services:
environment:
- COLLECTOR_ZIPKIN_HTTP_PORT=9411

node-exporter:
image: quay.io/prometheus/node-exporter
container_name: node-exporter
pid: "host"
volumes:
- type: bind
source: /
target: /host
read_only: true

prometheus:
image: prom/prometheus
container_name: prometheus
hostname: prometheus
ports:
- "9090:9090"
volumes:
- type: bind
source: $GOPATH/src/github.com/dgraph-io/dgraph/dgraph/prometheus.yml
target: /etc/prometheus/prometheus.yml
read_only: true

grafana:
image: grafana/grafana
container_name: grafana
hostname: grafana
ports:
- "3000:3000"
# node-exporter:
# image: quay.io/prometheus/node-exporter
# container_name: node-exporter
# pid: "host"
# volumes:
# - type: bind
# source: /
# target: /host
# read_only: true


# prometheus:
# image: prom/prometheus
# container_name: prometheus
# hostname: prometheus
# ports:
# - "9090:9090"
# volumes:
# - type: bind
# source: $GOPATH/src/github.com/dgraph-io/dgraph/dgraph/prometheus.yml
# target: /etc/prometheus/prometheus.yml
# read_only: true

# grafana:
# image: grafana/grafana
# container_name: grafana
# hostname: grafana
# ports:
# - "3000:3000"
Oops, something went wrong.

0 comments on commit b9990f4

Please sign in to comment.