etcdmain: delete member dir falling back to proxy

[gyuho]

This fixes #3827 where member falls to
back to proxy successfully at first, but fails in subsequent tries.
It fails when there are 'member' and 'proxy' directory in the same place,
one of which did not get cleaned up from failure and causes this conflict
error message: 'invalid datadir. Both member and proxy directories exist.'

[gyuho]

Kindly ping @xiang90 @jonboulle

Please review and let me know if you have any other suggestion to resolve #3827. Thanks!

[xiang90]

@gyuho Can we stop etcd from creating member dir if it does not start as an etcd member?

[gyuho]

@xiang90 Ok, I will try that approach! Thanks

[gyuho]

@xiang90 I looked at the code again, and I think the issue matter only when a member falls back to proxy.

Code1. https://github.com/coreos/etcd/blob/master/etcdmain/etcd.go#L198
Code2. https://github.com/coreos/etcd/blob/master/etcdserver/server.go#L184-L205
Code3. https://github.com/coreos/etcd/blob/master/etcdserver/storage.go#L107-L125

Code1's startEtcd calls Code2's etcdserver.NewServer, which then calls upgradeDataDir in Code3. But startEtcd does not get called if it starts as a proxy (https://github.com/coreos/etcd/blob/master/etcdmain/etcd.go#L113-L137), which means we already do not create member dir if it does not start as an etcd member (it does when it starts as a member but falls back to proxy).

Please let me know if you have any feedback.

[gyuho]

I think we either allow member to have both proxy and member directories for this case, or delete the member directory when a member falls back to proxy. BTW, why we do not allow both proxy and member in the first place?

[xiang90]

@gyuho My question is when the will the user have both dir? That should not happen.

[gyuho]

@xiang90 Please correct me if I am wrong.

https://github.com/coreos/etcd/blob/c0353697da1216fd0555928efb53f4134a2b5ab4/etcdmain/etcd.go#L124-L136

shows if startEtcd returns discovery error, it tries to start as a proxy.
I think startEtcd could try to create both member and proxy directory in this case:

1. startEtcd succeeds creating its member directory`
2. This member crashes and tries to rererun startEtcd
3. But this time, discovery fails in the function getPeerURLsMapAndToken returning discovery type error
4. The logic at https://github.com/coreos/etcd/blob/c0353697da1216fd0555928efb53f4134a2b5ab4/etcdmain/etcd.go#L127-L132 detects this and tries to fall back to proxy
5. But before the crash, it was an etcd member with member directory, so now startProxy fails.

[xiang90]

I think startEtcd could try to create both member and proxy directory in this case:

etcd should probably ONLY create member dir after it finishes bootstrapping, not before.

[xiang90]

In the discovery case, etcd should create member dir after it finishes discovery and finds all its peers.

[gyuho]

Thanks. I will investigate more.

Sincerely,
Gyu-Ho Lee
On Dec 8, 2015 17:35, "Xiang Li" notifications@github.com wrote:

In the discovery case, etcd should create member dir after it finishes
discovery and finds all its peers.

—
Reply to this email directly or view it on GitHub
#3949 (comment).

[gyuho]

Closing this in favor of #4087.