Dgraph bulk loading in a Kubernetes setup (GKE) - bind: address already in use #2534
Description
If you suspect this could be a bug, follow the template.
- What version of Dgraph are you using? v1.0.7
- Have you tried reproducing the issue with latest release? YES
- What is the hardware spec (RAM, OS)? 2 vCPUs, 13GB RAM, 1 node (dev)
- Steps to reproduce the issue (command/config used to run Dgraph).
# Load the YAML
kubectl create -f /my/path/dgraph-single.yaml
# Copy local RDF dump and schema to GKE's disk
kubectl cp /Volumes/Samsung850EVO/4.rdf.gz default/dgraph-0:/dgraph -c zero
kubectl cp /Volumes/Samsung850EVO/dgraph.schema default/dgraph-0:/dgraph -c zero
# Disable SWAP
kubectl exec dgraph-0 -c zero -- sudo swapoff -a
# Stopping the "server" contaienr
kubectl exec dgraph-0 -c zero -- curl localhost:8080/admin/shutdown
# kubectl exec dgraph-0 -c zero -- dgraph bulk -r /dgraph/4.rdf.gz -s /dgraph/dgraph.schema --map_shards 2 --reduce_shards 1 --http localhost:6080 --zero localhost:5080
- Expected behaviour and actual result.
When loading data with the live loader it works and all is well. However, I have 50GB of RDFs to upload so I need to do it with the bulk loader. I created all of it again, deleted /p and /w folders, shutdown the server using the API endpoint, and tried again with the bulk loader.
$ kubectl exec dgraph-0 -c zero -- dgraph bulk -r /dgraph/4.rdf.gz -s /dgraph/dgraph.schema --map_shards 2 --reduce_shards 1
W0815 23:32:28.800960 42270 flags.go:39] map_shards is DEPRECATED and will be removed in a future version. Use map-shards instead.
W0815 23:32:28.801031 42270 flags.go:39] reduce_shards is DEPRECATED and will be removed in a future version. Use reduce-shards instead.
W0815 23:32:28.801066 42270 flags.go:39] map_shards is DEPRECATED and will be removed in a future version. Use map-shards instead.
W0815 23:32:28.801071 42270 flags.go:39] reduce_shards is DEPRECATED and will be removed in a future version. Use reduce-shards instead.
W0815 23:32:28.801095 42270 flags.go:39] map_shards is DEPRECATED and will be removed in a future version. Use map-shards instead.
W0815 23:32:28.801099 42270 flags.go:39] reduce_shards is DEPRECATED and will be removed in a future version. Use reduce-shards instead.
{
"RDFDir": "/dgraph/4.rdf.gz",
"SchemaFile": "/dgraph/dgraph.schema",
"DgraphsDir": "out",
"TmpDir": "tmp",
"NumGoroutines": 2,
"MapBufSize": 67108864,
"ExpandEdges": true,
"SkipMapPhase": false,
"CleanupTmp": true,
"NumShufflers": 1,
"Version": false,
"StoreXids": false,
"ZeroAddr": "localhost:5080",
"HttpAddr": "localhost:8080",
"MapShards": 2,
"ReduceShards": 1
}
The bulk loader needs to open many files at once. This number depends on the size of the data set loaded, the map file output size, and the level of indexing. 100,000 is adequate for most data set sizes. See `man ulimit` for details of how to change the limit.
Current max open files limit: 1048576
2018/08/15 21:32:30 listen tcp 127.0.0.1:8080: bind: address already in use
command terminated with exit code 1
Any idea how to fix? I also tried --http localhost:6080 --zero localhost:5080, --http localhost:8080 --zero localhost:5080, and without these so the defaults would be used. Always results in this error message.