etcd-dump-logs: add decoder support #9790

wenjiaswe · 2018-05-30T20:19:02Z

etcd-dump-logs: add decoder support

Added "stream-decoder" flag so user could set the decoder and decoder args they want to use to decode each WAL log entries.

Expected behavior of decoder:
The decoder should be able to take hex encoded lines of binary as input (from etcd-dump-logs), process it, output the status and decoded data, preferably in the format of <DECODER_STATUS>|<DECODED_DATA>. (If the output is not in the preferable format, result will be put into decoded data column regardlessly)

Expected behavior of etcd-dump-logs:
When the stream-decoder is specified, etcd-dump-logs will feed the hex encoded lines of WAL LOG binary into the decoder and output the decoder_status and decoded_data for each entry.

Two simple decoders were added for testing and demonstrating purpose:
Both decoders (decoder_correctoutputformat.sh and decoder_wrongoutputformat.sh ) are simply converting 1234567890 to abcdefghij. decoder_correctoutputformat.sh outputs decoder_status and decoded_data in the preferable format as above (<DECODER_STATUS>|<DECODED_DATA>), while decoder_wrongoutputformat.sh does not. The different result from etcd-dump-logs can be found in decoder_correctoutputformat.output and decoder_wrongoutputformat.output.

Combination usage of etcd-dump-logs and Auger:
Other than the two "fake" decoder above, I would like to mention a real usage of this added feature using Auger as decoder. Auger (please refer to Auger link for further details if you are interested) is a very useful tool in kubernetes community to inspect and analyze kubernetes objects stored in etcd. Now with this change, we could use Auger as the decoder to decode kubernetes objects in WAL log and have them output nicely. Here is the example of using etcd-dump-logs and Auger.

$ etcd-dump-logs -stream-decoder "auger decode --output json --batch-process true" datadir
Snapshot:
empty
Start dupmping log entries from snapshot.
WAL metadata:
nodeID=2f7fbfd36802ff7a clusterID=bea638dce5096eef term=2 commitIndex=3358 vote=2f7fbfd36802ff7a
WAL entries:
lastIndex=3358
term index type data decoder_status decoded_data
1 1 conf method=ConfChangeAddNode id=2f7fbfd36802ff7a AUGER_ERROR:error decoding from application/json: Object 'Kind' is missing in '{"id":3422665156528897914,"peerURLs":["https://kubernetes-master:2380"],"name":"etcd-kubernetes-master"}'
2 2 norm AUGER_ERROR:error reading input, does not appear to contain valid JSON or binary data
2 3 norm method=PUT path="/0/members/2f7fbfd36802ff7a/attributes" val="{"name":"etcd-kubernetes-master","clientURLs":["http://127.0.0.1:2379"]}" AUGER_ERROR:error reading input, does not appear to contain valid JSON or binary data
2 4 norm method=PUT path="/0/version" val="3.2.0" AUGER_ERROR:error reading input, does not appear to contain valid JSON or binary data
2 5 norm method=QGET path="" AUGER_ERROR:error reading input, does not appear to contain valid JSON or binary data
2 6 norm method=QGET path="" AUGER_ERROR:error reading input, does not appear to contain valid JSON or binary data
2 7 norm header:<ID:18409135401817889912 > txn:<compare:<target:MOD key:"/registry/ranges/serviceips" mod_revision:0 > success:<request_put:<key:"/registry/ranges/serviceips" value:"k8s\000\n\025\n\002v1\022\017RangeAllocation\022#\n\022\n\000\022\000\032\000"\000*\0002\0008\000B\000z\000\022\01310.0.0.0/16\032\000\032\000"\000" > > failure:<request_range:<key:"/registry/ranges/serviceips" > > > AUGER_OK {"kind":"RangeAllocation","apiVersion":"v1","metadata":{"creationTimestamp":null},"range":"10.0.0.0/16","data":""}
2 8 norm header:<ID:18409135401817889914 > txn:<compare:<target:MOD key:"/registry/ranges/servicenodeports" mod_revision:0 > success:<request_put:<key:"/registry/ranges/servicenodeports" value:"k8s\000\n\025\n\002v1\022\017RangeAllocation\022#\n\022\n\000\022\000\032\000"\000*\0002\0008\000B\000z\000\022\01330000-32767\032\000\032\000"\000" > > failure:<request_range:<key:"/registry/ranges/servicenodeports" > > > AUGER_OK {"kind":"RangeAllocation","apiVersion":"v1","metadata":{"creationTimestamp":null},"range":"30000-32767","data":""}

README.md is added.
As etcd-dump-logs is getting more powerful and complicated, I think it's easier to have a README.md to explain the usage.

@gyuho @xiang90 @jpbetz

gyuho

Some minor formatting issue. Can we also rebase from current master branch? We've bumped up the base test image to use Go 1.10.2.

Will try this new patch as well.

gyuho · 2018-05-30T20:46:06Z

tools/etcd-dump-logs/README.md

+Lists all the interested entries from WAL log.
+
+```
+


remove this unneeded blank line at 39?

gyuho · 2018-05-30T20:46:38Z

tools/etcd-dump-logs/README.md

+Entry types () count is : 4
+```
+
+[decoder_correctoutputformat.sh]: https://github.com/wenjiaswe/etcd/blob/decoder-support/tools/etcd-dump-logs/testdecoder/decoder_correctoutputformat.sh


[decoder_correctoutputformat.sh]: ./testdecoder/decoder_correctoutputformat.sh?

gyuho · 2018-05-30T20:47:22Z

tools/etcd-dump-logs/etcd-dump-log_test.go

@@ -45,6 +45,9 @@ func TestEtcdDumpLogEntryType(t *testing.T) {
 		t.Skipf("%q does not exist", dumpLogsBinary)
 	}

+	decoder_correctoutputformat := path.Join(binDir + "/testdecoder/decoder_correctoutputformat.sh")


filepath.Join(binDir, "/testdecoder/decoder_correctoutputformat.sh")?

gyuho · 2018-05-30T20:47:39Z

tools/etcd-dump-logs/etcd-dump-log_test.go

@@ -45,6 +45,9 @@ func TestEtcdDumpLogEntryType(t *testing.T) {
 		t.Skipf("%q does not exist", dumpLogsBinary)
 	}

+	decoder_correctoutputformat := path.Join(binDir + "/testdecoder/decoder_correctoutputformat.sh")
+	decoder_wrongoutputformat := path.Join(binDir + "/testdecoder/decoder_wrongoutputformat.sh")


filepath.Join(binDir, "/testdecoder/decoder_wrongoutputformat.sh")?

gyuho · 2018-05-30T20:48:41Z

tools/etcd-dump-logs/main.go

@@ -31,6 +35,8 @@ import (
 	"github.com/coreos/etcd/wal/walpb"

 	"go.uber.org/zap"
+	"bytes"


Can you move "bytes" and "os" at the same level as "encoding/hex"? Just minor gofmt issue.

gyuho · 2018-05-30T20:49:42Z

tools/etcd-dump-logs/README.md

+etcd-dump-logs inspects etcd db files.
+
+```
+


Remove unnecessary blank line at 6?

wenjiaswe

@gyuho Thank you very much for your review, I have addressed all your comments and rebased from the latest master branch, I have reran the unit tests and they all passed:

$ PASSES=unit PKG=./tools/etcd-dump-logs TESTCASE="\bTestEtcdDumpLogEntryType\b" ./test
Running with TEST_CPUS: 1,2,4
Starting 'unit' pass at Wed May 30 15:44:58 PDT 2018
Running unit tests...

gyuho · 2018-05-31T19:19:32Z

tools/etcd-dump-logs/main.go

+		if err != nil {
+			log.Panic(err)
+		}
+		os.Stderr.WriteString("decoder stderr: " + stderr.String())


if stderr.String() != "" { os.Stderr.WriteString("decoder stderr: " + stderr.String()) }

?

Thanks! Fixed!

gyuho

I manually tried it and works great. Thanks a lot!
Just minor nits on decoder stderr: output. And lgtm.

gyuho · 2018-05-31T20:47:59Z

tools/etcd-dump-logs/README.md

+   9	         6	norm	ID:10 lease_grant:<TTL:1 ID:1 > 	decoder output format is not right, print output anyway	jhjadbjdjhjaajja
+  12	         7	norm	ID:13 auth_enable:<> 	decoder output format is not right, print output anyway	jhjdcbcejj
+  27	         8	norm	???	decoder output format is not right, print output anyway	cf
+decoder stderr:


Remove this?

gyuho · 2018-05-31T20:48:05Z

tools/etcd-dump-logs/README.md

+   9	         6	norm	ID:10 lease_grant:<TTL:1 ID:1 > 	ERROR	jhjadbjdjhjaajja
+  12	         7	norm	ID:13 auth_enable:<> 	ERROR	jhjdcbcejj
+  27	         8	norm	???	ERROR	cf
+decoder stderr:


Remove this?

gyuho · 2018-05-31T21:30:21Z

Manually addressed README.md typos. Merging. Thanks.

wenjiaswe · 2018-05-31T22:26:20Z

Thanks, @gyuho !

gyuho suggested changes May 30, 2018

View reviewed changes

wenjiaswe force-pushed the decoder-support branch 5 times, most recently from c0a49a3 to 3ef3cff Compare May 30, 2018 22:40

wenjiaswe commented May 30, 2018

View reviewed changes

wenjiaswe force-pushed the decoder-support branch from 3ef3cff to 4dfad83 Compare May 30, 2018 23:39

gyuho reviewed May 31, 2018

View reviewed changes

gyuho approved these changes May 31, 2018

View reviewed changes

wenjiaswe force-pushed the decoder-support branch from 4dfad83 to 5871905 Compare May 31, 2018 20:46

gyuho reviewed May 31, 2018

View reviewed changes

etcd-dump-logs: add decoder support

29121a8

gyuho force-pushed the decoder-support branch from 5871905 to 29121a8 Compare May 31, 2018 21:30

gyuho merged commit cc50a70 into etcd-io:master May 31, 2018

wenjiaswe deleted the decoder-support branch June 19, 2018 22:32

jingyih mentioned this pull request Sep 29, 2018

Enable etcd-dump-logs tool to decode kubernetes data objects in WAL log #9631

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

etcd-dump-logs: add decoder support #9790

etcd-dump-logs: add decoder support #9790

wenjiaswe commented May 30, 2018 •

edited

Loading

gyuho left a comment

gyuho May 30, 2018

gyuho May 30, 2018

gyuho May 30, 2018

gyuho May 30, 2018

gyuho May 30, 2018

gyuho May 30, 2018

wenjiaswe left a comment

gyuho May 31, 2018

wenjiaswe May 31, 2018

gyuho left a comment

gyuho May 31, 2018

gyuho May 31, 2018

gyuho commented May 31, 2018

wenjiaswe commented May 31, 2018

etcd-dump-logs: add decoder support #9790

etcd-dump-logs: add decoder support #9790

Conversation

wenjiaswe commented May 30, 2018 • edited Loading

gyuho left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wenjiaswe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gyuho left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gyuho commented May 31, 2018

wenjiaswe commented May 31, 2018

wenjiaswe commented May 30, 2018 •

edited

Loading