etcdserver: remove temp files in snap dir when etcdserver starting

client: fix check datascale command for https endpoints

Fixes etcd-io#12875

etcd-io#12732

Update go for 3.5: 1.15.x -> 1.16.3

Since the Dockerfile files are now per arch, this adjusts to detect ARCH
and builds docker release from the Dockerfile.<ARCH> file.

This fixes so that the ENV vars are taken in the same way as for etcd
binary.

Makefile, build.sh: Fix build process

doc: fix branch management link

…ith ?exclude=NOSPACE

…ealth_check

etcdhttp/metrics.go: exclude alarms from health check conditionally with `?exclude=NOSPACE`

Makefile: Use `gcloud auth configure-docker` instead of `gcloud docker ...` for test-images

(no)StoreV2 (Part 3): Applying consistency fix: ClusterVersionSet (and co) might get not applied on v2store

…r-testing

mvcc/backend tests: Refactor: Do not mix testing&prod code.

…d as codes.Canceled, codes.DeadlineExceeded instead of 'codes.Unknown'

…text

Errors: `context cancelled` or `context deadline exceeded` are exposed as codes.Canceled, codes.DeadlineExceeded instead of 'codes.Unknown'

Infrequently the test flaked. Reproducable with:

```
  go test go.etcd.io/etcd/tests/v3/integration --run TestFirstCommitNotification --count=500
```

The moveLeader finishes when configchange is commited by quorum.
It doesn't guarantee that the 'empty' record was committed by the new leader.
From time to time happened that appliedLeaderIndex was returning 9
(without empty entry) and the test flaked. In healthy case the
appliedIndex returned 10.

Fixed by putting kv pair after leader change. The pair is guaranteed
to be stored on index when put finishes (so the empty entry as well).

…stCommitNotification

Deflake: TestFirstCommitNotification

functional: add disk io failure case

etcdserver/mvcc: update tw.trace.Step condition

correct 'backend' (bbolt) context in aspect of membership.

Prior to this change the 'restored' backend used to still contain:
  - old memberid (mvcc deletion used, why the membership is in bolt
bucket, but not mvcc part):
    ```
	mvs := mvcc.NewStore(s.lg, be, lessor, ci, mvcc.StoreConfig{CompactionBatchLimit: math.MaxInt32})
	defer mvs.Close()
	txn := mvs.Write(traceutil.TODO())
	btx := be.BatchTx()
	del := func(k, v []byte) error {
		txn.DeleteRange(k, nil)
		return nil
	}

	// delete stored members from old cluster since using new members
	btx.UnsafeForEach([]byte("members"), del)
    ```
  - didn't get new members added.

…backend.

(no)StoreV2 (Part 2): Prepare to read membership information from backend

For now verifies whete Backend.cindex is consistent with WAL log,
but should get expanded to cover memberships & revisions.

Verification framework is integrated with:
  - integration tests (by default)
  - `ETCD_VERIFY=all etcdctl snapshot restore` command
  - etcd shutdown when running with `ETCD_VERIFY=all` env.

Verification of persisted data

…ency

Prior to this CL, `ETCDCTL_API=2 etcdctl backup --with-v3` was readacting WAL log
(by removal of some entries), but was NOT updating consistent_index in the backend.
Also the WAL editing logic was buggy, as it didn't took in consideration the fact
that when TERM changes, there can be entries with duplicated indexes in
the log. So its NOT sufficient to subtract number of removed entries to
get accurate log indexes.

The PR replaces removing and shifting of WAL entries with replacing them with an no-op entries.
Thanks to this consistent-index references are staying up to date.

The PR also:
  - updates 'verification' logic to check whether consistent_index does not lag befor last snapshot
  - env-gated execution of verification framework in `etcdctl backup`.

Tested with:
```
(./build.sh && cd tests && EXPECT_DEBUG=TRUE 'env' 'go' 'test' '-timeout=300m' 'go.etcd.io/etcd/tests/v3/e2e' -run=TestCtlV2Backup --count=1000 2>&1 | tee TestCtlV2BackupV3.log)
```

In case of failed verification, the server used to keep opened backend
(so the file was locked on OS level).

Implementation of `--experimental-memory-mlock` backed by `mlockall` syscall is replaced by `Mlock` flag (backed by mlock syscall) of bboltDB.

Implementation of `--experimental-memory-mlock` backed by `mlockall` syscall is replaced by `Mlock` flag (backed by mlock syscall) of bboltDB.

If ctc+c is pressed during the run of etcdctl check perf,
this deletes all the previously created keys that perf check created.

clientv3 logs (especially tests) were poluted with unattributed to testing.T log lines:

```
{"level":"warn","ts":"2021-04-29T12:42:11.055+0200","logger":"etcd-client","caller":"v3/retry_interceptor.go:64","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000fafc0/#initially=[unix://localhost:m10]","attempt":0,"error":"rpc error: code = ResourceExhausted desc = etcdserver: mvcc: database space exceeded"}
```

The reasons were 2 fold:
  - Interceptors were copying logger before "WithLogger" could modify it.
  - We were not propagating the loggers in a few testing contexts.

etcdctl/ctlv3: Cleanup keys on signal interrupt.

…up-cindex-fix

20210429 etcdctl v2 backup cindex fix

server: replace mlockall with `Mlock` in `--experimental-memory-mlock`

…gging

Clientv3 (retry interceptor) logs should use the configured logger

…n tests

Finish migration zap.NewExample -> zaptest.NewLogger(t) in integration tests

server: reapply `Mlock` flag after defrag

No-storeV2: Read membership information from the backend (Part5)

Every transaction committed to backend is writing most recent consistent_index.
Makes sure that even automatically trigger commits of batch-transactions
stays "really" consistent a.d. the most recent WAL log index applied.

…n care by hook.

…rite

* learner support snapshot RPC

* CHANGELOG: update for 12890

Signed-off-by: odidev <odidev@puresoftware.com>

the service parameter not used.

fix doc demo bug

server: make applier use ReadTx() in Txn() instead of ConcurrentReadTx()

Migrate PR testing from travis to GitHub actions

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

Signed-off-by: Sam Batschelet <sbatsche@redhat.com>

server: add support for log rotation

(no)StoreV2 (Part 4): Backend hooks:  precommit updates consistency_index

…hod. Fixed etcd-io#12821

server/etcdmain and tests: Fix goroutine leaks

…-nil

embed: Added a check for a nil value of the request in ServeHTTP method

README for self-hosted-linux-arm64-graviton2-tests.yml

Signed-off-by: Gyuho Lee <leegyuho@amazon.com>

github action: add ARM64 test

Signed-off-by: Gyuho Lee <leegyuho@amazon.com>

Trigger unit test workflow on pull_requests

…e them from travis

* Leave Go tip tests and coverage in travis as one is not yet supported

Flags `--experimental-enable-v2v3` and '-enable-v2' will raise a warning in 3.5,
in 3.6 they are schedule for decomissioning, such that v2store can stop be written in 3.7.

Deprecation plan in: etcd-io#12913

Deprecate V2 API: --enable-v2 and --experimental-enable-v2v3 issue a warning

github actions: Make github actions go1.16.3 tests blocking and remove them from travis

The motivation is need to diagnose flakes like:
https://semaphoreci.com/etcd-io/etcd/branches/pull-request-12940/builds/2.

Now - when the logs are streamed through testing.T logger, the verbosity
is not that significant concern.

…ue-for-e2e

Run e2e tests with: EXPECT_DEBUG='true'

…ivided threshold