Skip to content

Commit ed80d23

Browse files
committed
Add section Embedding etcd in a Go application.
Signed-off-by: Ronald Ngounou <74538524+ronaldngounou@users.noreply.github.com>
1 parent 5527ab4 commit ed80d23

14 files changed

Lines changed: 544 additions & 276 deletions

code-of-conduct.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# etcd Community Code of Conduct
2+
3+
Please refer to [etcd Community Code of Conduct](https://github.com/etcd-io/etcd/blob/main/code-of-conduct.md).
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
---
2+
title: How to Prevent a Common Failure when Upgrading etcd v3.5 to v3.6
3+
author: "[Benjamin Wang](https://github.com/ahrtr), VMware by Broadcom"
4+
date: 2025-03-27
5+
draft: false
6+
---
7+
8+
There is a common issue [19557][] in the etcd v3.5 to v3.6 upgrade that may cause the upgrade
9+
process to fail. You can find detailed information and related discussions in the issue.
10+
11+
## TL; DR
12+
13+
Users are required to first upgrade to etcd v3.5.20 (or a higher patch version) before upgrading
14+
to etcd v3.6.0. Failure to do so may result in an unsuccessful upgrade.
15+
16+
## What's the symptom?
17+
18+
When upgrading a multi-member etcd cluster from a version between v3.5.1 and v3.5.19 to v3.6.0,
19+
the upgrade may fail due to the error "`membership: too many learner member in cluster`".
20+
21+
## What's the root cause?
22+
23+
By default, etcd allows at most one learner member; the bootstrap process will fail if there are
24+
two or more learners. Due to the issue [19557][], some voting members may revert to learners when
25+
upgrading to v3.6.0, which may cause the upgrade to fail.
26+
27+
The root cause is that when promoting a learner, the related change is persistently stored in
28+
v2store only, not in v3store. As a result, the membership data becomes inconsistent between
29+
the v2store and v3store.
30+
31+
In etcd v3.5, the v2store is the source of truth for the membership data, whereas in etcd v3.6,
32+
the v3store is the source of truth. It's exactly the reason why the issue only occurs when
33+
upgrading from v3.5 to v3.6.
34+
35+
## Which versions are impacted?
36+
37+
The issue was introduced in etcd v3.5.1 via [13348][]. All etcd patch versions from v3.5.1 to
38+
v3.5.19 are affected.
39+
40+
## Which versions contain the fix for this issue?
41+
42+
The issue was fixed in etcd v3.5.20 via [19563][]. **Therefore, users are required to first upgrade to
43+
etcd v3.5.20 (or a higher patch version) before upgrading to etcd v3.6.0; otherwise, the
44+
upgrade may fail.**
45+
46+
## What if users do not follow the guide?
47+
48+
What happens if users still upgrade directly from etcd v3.5.1-v3.5.19 to v3.6.0?
49+
50+
If the etcd cluster isn't affected by the issue, no members will revert to learners
51+
during the upgrade. In this case, the upgrade will succeed without any problems.
52+
53+
If the etcd cluster has already been affected by the issue, there are two possible outcomes:
54+
55+
- If two or more members revert to learners during the upgrade, the upgrade will fail. In that
56+
case, users will have to roll back and first upgrade to etcd v3.5.20 (or a higher version)
57+
before attempting to upgrade to v3.6.0 again.
58+
- If only one member reverts to a learner during the upgrade, the upgrade will still succeed.
59+
However, the issue is that a voting member reverts to a learner, which may confuse users and
60+
also affect the cluster's quorum. We addressed this issue in etcd v3.6.0-rc.3 via [19636][]
61+
by automatically promoting the learner to a voting member during bootstrap.
62+
63+
## How did we prevent this from happening again?
64+
65+
etcd has some end-to-end (e2e) upgrade test cases, but did not detect this issue. Instead,
66+
it was discovered in Kubernetes' workflow test. To address this gap, we added a similar e2e
67+
test via [19634][], which was also backported to release-3.6 via [19662][].
68+
69+
[19557]: https://github.com/etcd-io/etcd/issues/19557
70+
[13348]: https://github.com/etcd-io/etcd/pull/13348
71+
[19563]: https://github.com/etcd-io/etcd/pull/19563
72+
[19636]: https://github.com/etcd-io/etcd/pull/19636
73+
[19634]: https://github.com/etcd-io/etcd/pull/19634
74+
[19662]: https://github.com/etcd-io/etcd/pull/19662

content/en/docs/v3.4/_index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
title: v3.4 docs
33
cascade:
44
version: &vers v3.4
5-
git_version_tag: v3.4.35
5+
git_version_tag: v3.4.36
66
is_deprecated: false
77
exclude_search: true
88
linkTitle: *vers

content/en/docs/v3.5/_index.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: v3.5 docs
33
cascade:
44
version: v3.5
55
versName: &name v3.5
6-
git_version_tag: v3.5.17
6+
git_version_tag: v3.5.21
77
exclude_search: false
88
linkTitle: *name
99
simple_list: true
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
---
2+
title: Embedding etcd in a Go Application
3+
weight: 3562
4+
description: Use etcd `embed` go package to run an etcd server within your application
5+
---
6+
7+
The etcd `embed` go package provides a simple way to embed an etcd server directly into your application.
8+
9+
For more details, see the [embed package documentation](https://pkg.go.dev/go.etcd.io/etcd/server/v3/embed).

content/en/docs/v3.5/downgrades/downgrade_3_5.md

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,9 @@ description: Processes, checklists, and notes on downgrading etcd from 3.5 to 3.
55
---
66

77
In the general case, downgrading from etcd 3.5 to 3.4 can be a zero-downtime, rolling downgrade:
8-
- one by one, stop the etcd 3.5 processes and replace them with etcd 3.4 processes
9-
- after starting any 3.4 processes, new features in 3.5 are not longer available to the cluster
8+
9+
- one by one, stop the etcd 3.5 processes and replace them with etcd 3.4 processes
10+
- after starting any 3.4 processes, new features in 3.5 are not longer available to the cluster
1011

1112
Before [starting a downgrade](#downgrade-procedure), read through the rest of this guide to prepare.
1213

@@ -163,7 +164,9 @@ etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 endpoint stat
163164
+-----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
164165
COMMENT
165166
```
167+
166168
If the server to be stopped is the leader, you can avoid some downtime by `move-leader` to another server before stopping this server.
169+
167170
```bash
168171
etcdctl --endpoints=localhost:2379,localhost:22379,localhost:32379 move-leader 91bc3c398fb3c146
169172

@@ -199,8 +202,8 @@ When each etcd process is stopped, expected errors will be logged by other clust
199202
Restart the etcd server with same configuration but with the new etcd binary and `--next-cluster-version-compatible`.
200203

201204
```diff
202-
-etcd-old --name s1 \
203-
+etcd-new --name s1 \
205+
-etcd-3.5/bin --name s1 \
206+
+etcd-3.4/bin --name s1 \
204207
--data-dir /tmp/etcd/s1 \
205208
--listen-client-urls http://localhost:2379 \
206209
--advertise-client-urls http://localhost:2379 \
@@ -214,6 +217,7 @@ Restart the etcd server with same configuration but with the new etcd binary and
214217

215218
The new 3.4 etcd will publish its information to the cluster. At this point, cluster will start to operate as 3.4 protocol, which is the lowest common version.
216219

220+
```bash
217221
> `{"level":"info","ts":"2024-05-13T21:05:43.981445Z","caller":"membership/cluster.go:561","msg":"set initial cluster version","cluster-id":"ef37ad9dc622a7c4","local-member-id":"8211f1d0f64f3269","cluster-version":"3.0"}`
218222

219223
> `{"level":"info","ts":"2024-05-13T21:05:43.982188Z","caller":"api/capability.go:77","msg":"enabled capabilities for version","cluster-version":"3.0"}`
@@ -225,6 +229,7 @@ The new 3.4 etcd will publish its information to the cluster. At this point, clu
225229
> `{"level":"info","ts":"2024-05-13T21:05:44.000672Z","caller":"etcdserver/server.go:2152","msg":"published local member to cluster through raft","local-member-id":"8211f1d0f64f3269","local-member-attributes":"{Name:infra1 ClientURLs:[http://127.0.0.1:2379]}","request-path":"/0/members/8211f1d0f64f3269/attributes","cluster-id":"ef37ad9dc622a7c4","publish-timeout":"7s"}`
226230

227231
> `{"level":"info","ts":"2024-05-13T21:05:46.452631Z","caller":"membership/cluster.go:549","msg":"updated cluster version","cluster-id":"ef37ad9dc622a7c4","local-member-id":"8211f1d0f64f3269","from":"3.5","from":"3.4"}`
232+
```
228233

229234
Verify that each member, and then the entire cluster, becomes healthy with the new 3.4 etcd binary:
230235

@@ -239,7 +244,7 @@ COMMENT
239244

240245
Un-downgraded members will log info like the following
241246

242-
```
247+
```bash
243248
{"level":"info","ts":"2024-05-13T21:05:46.450764Z","caller":"etcdserver/server.go:2633","msg":"updating cluster version using v2 API","from":"3.5","to":"3.4"}
244249
{"level":"info","ts":"2024-05-13T21:05:46.452419Z","caller":"membership/cluster.go:576","msg":"updated cluster version","cluster-id":"ef37ad9dc622a7c4","local-member-id":"91bc3c398fb3c146","from":"3.5","to":"3.4"}
245250
{"level":"info","ts":"2024-05-13T21:05:46.452547Z","caller":"etcdserver/server.go:2652","msg":"cluster version is updated","cluster-version":"3.4"}
@@ -273,4 +278,4 @@ curl http://localhost:32379/version
273278
COMMENT
274279
```
275280

276-
[etcd-contact]: https://groups.google.com/g/etcd-dev
281+
[etcd-contact]: https://groups.google.com/g/etcd-dev

content/en/docs/v3.5/downgrades/downgrading-etcd.md

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,5 @@ description: Documentation list for downgrading etcd clusters and applications
77
This section contains documents specific to downgrading etcd clusters and applications.
88

99
## Downgrading an etcd v3.x cluster
10-
* [Downgrade etcd from 3.5 to 3.4](../downgrade_3_5/)
1110

12-
[migrate-apps]: ../../op-guide/v2-migration/
11+
* [Downgrade etcd from 3.5 to 3.4](../downgrade_3_5/)

content/en/docs/v3.6/dev-guide/api_concurrency_reference_v3.md

Lines changed: 0 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -13,42 +13,32 @@ The lock service exposes client-side locking facilities as a gRPC interface.
1313
| Lock | LockRequest | LockResponse | Lock acquires a distributed shared lock on a given named lock. On success, it will return a unique key that exists so long as the lock is held by the caller. This key can be used in conjunction with transactions to safely ensure updates to etcd only occur while holding lock ownership. The lock is held until Unlock is called on the key or the lease associate with the owner expires. |
1414
| Unlock | UnlockRequest | UnlockResponse | Unlock takes a key returned by Lock and releases the hold on lock. The next Lock caller waiting for the lock will then be woken up and given ownership of the lock. |
1515

16-
17-
1816
##### message `LockRequest` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)
1917

2018
| Field | Description | Type |
2119
| ----- | ----------- | ---- |
2220
| name | name is the identifier for the distributed shared lock to be acquired. | bytes |
2321
| lease | lease is the ID of the lease that will be attached to ownership of the lock. If the lease expires or is revoked and currently holds the lock, the lock is automatically released. Calls to Lock with the same lease will be treated as a single acquisition; locking twice with the same lease is a no-op. | int64 |
2422

25-
26-
2723
##### message `LockResponse` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)
2824

2925
| Field | Description | Type |
3026
| ----- | ----------- | ---- |
3127
| header | | etcdserverpb.ResponseHeader |
3228
| key | key is a key that will exist on etcd for the duration that the Lock caller owns the lock. Users should not modify this key or the lock may exhibit undefined behavior. | bytes |
3329

34-
35-
3630
##### message `UnlockRequest` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)
3731

3832
| Field | Description | Type |
3933
| ----- | ----------- | ---- |
4034
| key | key is the lock ownership key granted by Lock. | bytes |
4135

42-
43-
4436
##### message `UnlockResponse` (server/etcdserver/api/v3lock/v3lockpb/v3lock.proto)
4537

4638
| Field | Description | Type |
4739
| ----- | ----------- | ---- |
4840
| header | | etcdserverpb.ResponseHeader |
4941

50-
51-
5242
##### service `Election` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)
5343

5444
The election service exposes client-side election facilities as a gRPC interface.
@@ -61,8 +51,6 @@ The election service exposes client-side election facilities as a gRPC interface
6151
| Observe | LeaderRequest | LeaderResponse | Observe streams election proclamations in-order as made by the election's elected leaders. |
6252
| Resign | ResignRequest | ResignResponse | Resign releases election leadership so other campaigners may acquire leadership on the election. |
6353

64-
65-
6654
##### message `CampaignRequest` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)
6755

6856
| Field | Description | Type |
@@ -71,17 +59,13 @@ The election service exposes client-side election facilities as a gRPC interface
7159
| lease | lease is the ID of the lease attached to leadership of the election. If the lease expires or is revoked before resigning leadership, then the leadership is transferred to the next campaigner, if any. | int64 |
7260
| value | value is the initial proclaimed value set when the campaigner wins the election. | bytes |
7361

74-
75-
7662
##### message `CampaignResponse` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)
7763

7864
| Field | Description | Type |
7965
| ----- | ----------- | ---- |
8066
| header | | etcdserverpb.ResponseHeader |
8167
| leader | leader describes the resources used for holding leadereship of the election. | LeaderKey |
8268

83-
84-
8569
##### message `LeaderKey` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)
8670

8771
| Field | Description | Type |
@@ -91,58 +75,44 @@ The election service exposes client-side election facilities as a gRPC interface
9175
| rev | rev is the creation revision of the key. It can be used to test for ownership of an election during transactions by testing the key's creation revision matches rev. | int64 |
9276
| lease | lease is the lease ID of the election leader. | int64 |
9377

94-
95-
9678
##### message `LeaderRequest` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)
9779

9880
| Field | Description | Type |
9981
| ----- | ----------- | ---- |
10082
| name | name is the election identifier for the leadership information. | bytes |
10183

102-
103-
10484
##### message `LeaderResponse` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)
10585

10686
| Field | Description | Type |
10787
| ----- | ----------- | ---- |
10888
| header | | etcdserverpb.ResponseHeader |
10989
| kv | kv is the key-value pair representing the latest leader update. | mvccpb.KeyValue |
11090

111-
112-
11391
##### message `ProclaimRequest` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)
11492

11593
| Field | Description | Type |
11694
| ----- | ----------- | ---- |
11795
| leader | leader is the leadership hold on the election. | LeaderKey |
11896
| value | value is an update meant to overwrite the leader's current value. | bytes |
11997

120-
121-
12298
##### message `ProclaimResponse` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)
12399

124100
| Field | Description | Type |
125101
| ----- | ----------- | ---- |
126102
| header | | etcdserverpb.ResponseHeader |
127103

128-
129-
130104
##### message `ResignRequest` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)
131105

132106
| Field | Description | Type |
133107
| ----- | ----------- | ---- |
134108
| leader | leader is the leadership to relinquish by resignation. | LeaderKey |
135109

136-
137-
138110
##### message `ResignResponse` (server/etcdserver/api/v3election/v3electionpb/v3election.proto)
139111

140112
| Field | Description | Type |
141113
| ----- | ----------- | ---- |
142114
| header | | etcdserverpb.ResponseHeader |
143115

144-
145-
146116
##### message `Event` (api/mvccpb/kv.proto)
147117

148118
| Field | Description | Type |
@@ -151,8 +121,6 @@ The election service exposes client-side election facilities as a gRPC interface
151121
| kv | kv holds the KeyValue for the event. A PUT event contains current kv pair. A PUT event with kv.Version=1 indicates the creation of a key. A DELETE/EXPIRE event contains the deleted key with its modification revision set to the revision of deletion. | KeyValue |
152122
| prev_kv | prev_kv holds the key-value pair before the event happens. | KeyValue |
153123

154-
155-
156124
##### message `KeyValue` (api/mvccpb/kv.proto)
157125

158126
| Field | Description | Type |
@@ -164,5 +132,3 @@ The election service exposes client-side election facilities as a gRPC interface
164132
| value | value is the value held by the key, in bytes. | bytes |
165133
| lease | lease is the ID of the lease that attached to key. When the attached lease expires, the key will be deleted. If lease is 0, then no lease is attached to the key. | int64 |
166134

167-
168-

0 commit comments

Comments
 (0)