[Distributed Index] Intermittent panic during start up

[sethrosetter]

"Couchbase Sync Gateway/0.0.1(360;b85b1c8)"

Using dev build feature/distributed_index build-number=360

2015/11/12 11:01:16 janitor: awakes, reason: cfg changed, key: planPIndexes
2015/11/12 11:01:16 janitor: pindexes to remove: 0
2015/11/12 11:01:16 janitor: pindexes to add: 0
2015/11/12 11:01:16 janitor: feeds to remove: 0
2015/11/12 11:01:16 janitor: feeds to add: 0
2015/11/12 11:01:16 planner: awakes, reason: cfg changed, key: nodeDefs-wanted
2015-11-12T11:01:16.772-08:00 DCP: DataUpdate for pindex 0xc2080d2600 called with vbucket: 704.  key: _sync:user: seq: 2
2015-11-12T11:01:16.772-08:00 DCP: DataUpdate for pindex 0xc208440e40 called with vbucket: 608.  key: _sync:syncdata seq: 1
2015-11-12T11:01:16.772-08:00 Changes+: Notifying that changeListener is stopping
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x6fad06]

goroutine 70 [running]:
github.com/couchbase/sync_gateway/db.(*kvChangeIndexWriter).DocChanged(0x0, 0xc2086cb610, 0xb, 0xc2084faa4a, 0x64, 0x64, 0x2, 0x2c0)
    /home/couchbase/jenkins/workspace/sync-gateway-dev-build/feature/distributed_index/community/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/db/kv_change_index_writer.go:145 +0x406
github.com/couchbase/sync_gateway/db.(*kvChangeIndex).DocChanged(0xc208170400, 0xc2086cb610, 0xb, 0xc2084faa4a, 0x64, 0x64, 0x2, 0xc2086c02c0)
    /home/couchbase/jenkins/workspace/sync-gateway-dev-build/feature/distributed_index/community/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/db/kv_change_index.go:194 +0x76
github.com/couchbase/sync_gateway/db.ChangeIndex.DocChanged·fm(0xc2086cb610, 0xb, 0xc2084faa4a, 0x64, 0x64, 0x2, 0x2c0)
    /home/couchbase/jenkins/workspace/sync-gateway-dev-build/feature/distributed_index/community/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/db/database.go:140 +0x86
github.com/couchbase/sync_gateway/db.func·010()
    /home/couchbase/jenkins/workspace/sync-gateway-dev-build/feature/distributed_index/community/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/db/change_listener.go:57 +0x53a
created by github.com/couchbase/sync_gateway/db.(*changeListener).Start
    /home/couchbase/jenkins/workspace/sync-gateway-dev-build/feature/distributed_index/community/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/db/change_listener.go:68 +0x42e

goroutine 1 [chan send]:
github.com/couchbase/cbgt.(*Manager).PlannerKick(0xc208051680, 0xceb0b0, 0x11)
    /home/couchbase/jenkins/workspace/sync-gateway-dev-build/feature/distributed_index/community/app-under-test/sync_gateway/src/github.com/couchbase/cbgt/manager_planner.go:43 +0x24d
github.com/couchbase/cbgt.(*Manager).Kick(0xc208051680, 0xceb0b0, 0x11)
    /home/couchbase/jenkins/workspace/sync-gateway-dev-build/feature/distributed_index/community/app-under-test/sync_gateway/src/github.com/couchbase/cbgt/manager.go:413 +0x62
github.com/couchbase/sync_gateway/rest.(*ServerContext).InitCBGT(0xc2080d4320, 0x0, 0x0)
    /home/couchbase/jenkins/workspace/sync-gateway-dev-build/feature/distributed_index/community/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/rest/server_context.go:304 +0x1fa
github.com/couchbase/sync_gateway/rest.RunServer(0xc208072780)
    /home/couchbase/jenkins/workspace/sync-gateway-dev-build/feature/distributed_index/community/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/rest/config.go:584 +0x4b3
github.com/couchbase/sync_gateway/rest.ServerMain()
    /home/couchbase/jenkins/workspace/sync-gateway-dev-build/feature/distributed_index/community/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/rest/config.go:615 +0x34
main.main()
    /home/couchbase/jenkins/workspace/sync-gateway-dev-build/feature/distributed_index/community/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/main.go:33 +0x195

[sethrosetter]

xxx.xxx.xx.251 was the crash.

The other 3 sync_gateways launched as expected.

Full logs - https://s3.amazonaws.com/sync-gateway-logs/2015-11-12-11-04-23-sglogs.zip

[adamcfraser]

What's the test scenario here? Are all sync_gateways intended to be index writers?

[sethrosetter]

Querying the /db/_config provide some clues

This configuration works

{
"server": "http://172.23.105.165:8091",
"pool": "default",
"bucket": "data-bucket",
"name": "db",
"sync": null,
"users": {
"GUEST": {
"name": "",
"admin_channels": [
"*"
],
"all_channels": null,
"disabled": true
}
},
"feed_type": "DCPSHARD",
"feed_params": {
"num_shards": 64
},
"channel_index": {
"server": "http://172.23.105.165:8091",
"bucket": "index-bucket",
"writer": true
}
}

however if they are launched as non writers, they fail intermittantly

{
"server": "http://172.23.105.165:8091",
"pool": "default",
"bucket": "data-bucket",
"name": "db",
"sync": null,
"users": {
"GUEST": {
"name": "",
"admin_channels": [
"*"
],
"all_channels": null,
"disabled": true
}
},
"feed_type": "DCPSHARD",
"feed_params": {
"num_shards": 64
},
"channel_index": {
"server": "http://172.23.105.165:8091",
"bucket": "index-bucket",
}
}

[sethrosetter]

In the failing case, they are all set as non writers. In the successful case, they are all writers.

[adamcfraser]

That's helpful - I think it's the case that nodes are registering themselves as writers even when the writer flag isn't set to true.

[adamcfraser]

Sync Gateway nodes that aren't index writers don't need the cluster config definition or the writer=true in their config.

If cluster config isn't present, Sync Gateway shouldn't be trying to register the CbgtPindexType during NewServerContext.

If the cluster config is present, but no databases have writer=false, we should still handle the case where a db config is loaded dynamically (from a ConfigServer) with writer=true set, but we shouldn't be starting DCP feeds for the databases with writer=false (which seems to be what's happening above).

[tleyden]

Closing as duplicate of #1322