docs: adding f.a.q.

Signed-off-by: André Martins <andre@cilium.io>
cilium · Feb 3, 2017 · 011f048 · 011f048
1 parent 99fa6a0
commit 011f048
Show file tree

Hide file tree

Showing 2 changed files with 170 additions and 0 deletions.
diff --git a/README.md b/README.md
@@ -32,6 +32,7 @@ for containers.
  * 5-min Quickstart: [Using the prebuilt docker images](examples/docker-compose/README.md)
  * For Developers: [Setting up a vagrant environment](doc/vagrant.md)
  * Manual installation: [Detailed installation instructions](doc/installation.md)
+ * F.A.Q.: [F.A.Q.](doc/faq.md)
 
 ## Demo Tutorials
 

diff --git a/doc/faq.md b/doc/faq.md
@@ -0,0 +1,169 @@
+# F.A.Q.
+
+This document contains some answers that will might help you debugging or getting to know
+how cilium tooling works.
+
+- [How do I use `cilium monitor`?](#how-do-i-use-cilium-monitor)
+- [The output of `cilium monitor` is too verbose I can't debug anything!](#the-output-of-cilium-monitor-is-too-verbose-i-cant-debug-anything)
+- [How can I debug my policy tree?](#how-can-I-debug-my-policy-tree)
+- [I'm sure the policy I've created is correct but the endpoints can't reach each other, why?](#im-sure-the-policy-ive-created-is-correct-but-the-endpoints-cant-reach-each-other-why)
+- [How can I see the data stored in the KV Store by cilium?](#how-can-i-see-the-data-stored-in-the-kv-store-by-cilium)
+
+### How do I use `cilium monitor`?
+
+The `monitor` is a powerful tool for debugging networking traffic generated by cilium.
+Although the output might look complex at the first sight, it will be easy to understand
+after reading this answer.
+
+`cilium monitor` can only be run with root privileges. It outputs all traffic coming in
+and going out from endpoints, presenting an extensive detail to help you out figure it out
+what's cilium doing to each packet.
+
+This is an output of `cilium monitor` (*except the line numbers on the left side*)
+```
+01: 00000000  ba 16 f9 d4 27 28 1e 7c  15 d3 2f 6f 08 00 45 00  |....'(.|../o..E.|
+02: 00000010  00 34 4e 1e 40 00 40 06  b2 ea 0a 01 2a fc 0a 01  |.4N.@.@.....*...|
+03: 00000020  fa bd a9 38 18 eb 45 f7  44 53 d5 bd 40 00 80 11  |...8..E.DS..@...|
+04: 00000030  00 dd 39 e2 00 00 01 01  08 0a 34 f7 10 d1 53 63  |..9.......4...Sc|
+05: 00000040  db 24 00 00                                       |.$..|
+06: 
+07: CPU 05: MARK 0x59e92ce5 FROM 13949 DEBUG: CT lookup: sport=52672 dport=6443 nexthdr=6 flags=0
+08: CPU 05: MARK 0x59e92ce5 FROM 13949 DEBUG: CT lookup address: b21a8c0
+09: CPU 05: MARK 0x59e92ce5 FROM 13949 DEBUG: CT entry found lifetime=360, revnat=4
+10: CPU 05: MARK 0x59e92ce5 FROM 13949 DEBUG: CT verdict: Reply
+11: CPU 05: MARK 0x59e92ce5 FROM 13949 DEBUG: Reverse NAT lookup, index=4
+12: CPU 05: MARK 0x59e92ce5 FROM 13949 DEBUG: Performing reverse NAT, address=100200a port=443
+13: CPU 05: MARK 0x59e92ce5 FROM 13949 DEBUG: 664 bytes Delivery to ifindex 1420
+14: 00000000  be 3a 70 81 a8 e5 a2 c8  c9 1c 4c 6d 08 00 45 00  |.:p.......Lm..E.|
+15: 00000010  02 8a bc bc 40 00 40 06  a6 2b 0a 20 00 01 0a 01  |....@.@..+. ....|
+16: 00000020  c1 64 01 bb cd c0 06 76  75 b2 44 80 28 41 80 18  |.d.....vu.D.(A..|
+17: 00000030  01 2c d8 02 00 00 01 01  08 0a 7b 5f a6 bb 99 00  |.,........{_....|
+18: 00000040  d0 7c 17 03 03 02 51 00  00 00 00 00 00 47 e2 f4  |.|....Q......G..|
+19: 00000050  63 04 81 ae 1f 49 fd 8d  90 53 6f 92 13 23 06 7a  |c....I...So..#.z|
+20: 00000060  d7 df 9d 73 7d 23 fe f8  d5 2f 1e f4 12 c7 a1 dd  |...s}#.../......|
+21: 00000070  f7 54 7b d2 e7 47 a0 8c  1b 5a 41 88              |.T{..G...ZA.|
+```
+
+Lines 1 to 5 show the first bytes of a packet before being processed by cilium. The packet
+is truncated to the first 64 Bytes and the values seen here are the same ones if you see
+with other tool such as tcpdump or Wireshark.
+
+Lines 7 to 13 contains a descriptive message of the actions cilium is making to that
+particular packet or decisions based on the packet value. For example, line 07 says: the
+packet was processed on CPU 05, cilium mark the packet with the value `0x59e92ce5`, to help
+reading the monitor messages, it's FROM endpoint `13949` and got a DEBUG message saying
+`CT lookup: sport=52672 dport=6443 nexthdr=6 flags=0` from the bpf program.
+
+Lines 14 to 21 shows the first 128 Bytes of the packets processed by the bpf cilium
+program that is going to be delivered to the interface with index number `1420` (it says
+at end of line 13).
+
+The monitor also offers an option `-d` that is used to dissect the packets. For example,
+the previous lines 1 to 6 and lines 14 to 21 will be replaced with something similar to this:
+
+```
+Ethernet        {Contents=[..14..] Payload=[..54..] SrcMAC=ba:16:f9:d4:27:28 DstMAC=1e:7c:15:d3:2f:6f EthernetType=IPv4 Length=0}
+IPv4    {Contents=[..20..] Payload=[..34..] Version=4 IHL=5 TOS=0 Length=57 Id=51206 Flags=DF FragOffset=0 TTL=64 Protocol=TCP Checksum=14589 SrcIP=10.1.250.189 DstIP=10.1.42.252 Options=[] Padding=[]}
+TCP     {Contents=[..32..] Payload=[43, 79] SrcPort=6379 DstPort=45570 Seq=2424400994 Ack=1620506753 DataOffset=8 FIN=false SYN=false RST=false PSH=true ACK=true URG=false ECE=false CWR=false NS=false Window=219 Checksum=14823 Urgent=0 Options=[TCPOption(NOP:), TCPOption(NOP:), TCPOption(Timestamps:1538605104/4052254881 0x5bb54030f18880a1)] Padding=[]}
+  Packet has been truncated
+```
+
+### The output of `cilium monitor` is too verbose I can't debug anything!
+
+1) List your endpoints with `cilium endpoint list`.
+
+2) For all endpoints that you **don't** want to debug, disable the debug logging messages.
+For example, for endpoint 173: `cilium endpoint config 173 Debug=false`
+
+3) Make sure debug is enabled for the endpoints you want to debug.
+For example, for endpoint 29381:
+```
+$ cilium endpoint config 29381
+Conntrack                Enabled
+ConntrackAccounting      Enabled
+Debug                    Enabled
+DropNotification         Enabled
+LearnTraffic             Disabled
+NAT46                    Enabled
+Policy                   Enabled
+```
+
+4) Run `cilium monitor`, it should be less verbose, containing only the debug messages for
+the endpoints that you want to debug.
+
+### How can I debug my policy tree?
+
+Simple run `cilium policy allowed --source value --destination value` where value can be a
+label or an label ID (**not endpoint ID**). If the value is a label, you can specify
+multiple sources and/or multiple destinations.
+
+Cilium outputs the tracing and the verdict for the given values for the policy tree on the
+given node. For example: `cilium policy allowed -s 260 -d 261` outputs the following:
+
+```
+Resolving policy: From: [k8s:io.cilium.k8s.k8s-app.guestbook=web k8s:io.kubernetes.pod.namespace=default] => To: [k8s:io.cilium.k8s.k8s-app.guestbook=redis k8s:io.kubernetes.pod.namespace=default]
+Root's [io.cilium] rules verdict: [undecided]
+Searching in [io.cilium]'s children that have the coverage for: [[k8s:io.cilium.k8s.k8s-app.guestbook=redis k8s:io.kubernetes.pod.namespace=default]]
+  Coverage found in [io.cilium.k8s], processing rules...
+    Rule has no coverage: [Coverage: [k8s:k8s-app=kubernetes-dashboard] Allowing: [{always-accept reserved:host}]]
+    Rule has no coverage: [Coverage: [k8s:k8s-app=kube-dns] Allowing: [{always-accept k8s:io.kubernetes.pod.namespace=kube-system} {always-accept k8s:io.kubernetes.pod.namespace=default} {always-accept reserved:host}]]
+  No conclusion in [io.cilium.k8s] rules, current verdict: [undecided]
+    Coverage found in [io.cilium.k8s.k8s-app], processing rules...
+      Found coverage rule: [Coverage: [k8s:guestbook=redis] Allowing: [{always-accept k8s:guestbook=redis} {always-accept k8s:guestbook=web}]]
+        No matching labels in allow rule: [{label: k8s:guestbook=redis, action: always-accept}]
+        Found label matching [k8s:io.cilium.k8s.k8s-app.guestbook=web] in rule: [{label: k8s:guestbook=web, action: always-accept}]
+Root's [io.cilium] children verdict: [always-accept]
+Final verdict: [ACCEPT]
+```
+
+The output describes what's the verdict of a policy node and their respective rules. For
+each policy node's children in the tree, it's added a space to help readability. The final
+verdict shows the policy result, and therefore the policy being enforced on the current
+node for the destination specified.
+
+### I'm sure the policy I've created is correct but the endpoints can't reach each other, why?
+
+There are numerous reasons for that to happen we can give you a basic walkthrough:
+
+1) Is the destination machine reachable by the source machine? [Yes]
+- If no, check the network connection between the machines.
+
+2) Is there a firewall between them or a rule blocking the particular traffic? [No]
+- If yes, add a rule to allow the particular traffic on the firewall.
+
+3) If you are using cilium in VxLAN or GENEVE mode (`-t vxlan` or `-t geneve`) make sure
+you are not blocking ports 4789, 8472 and 6081 tcp/udp on your firewall.
+
+4) `cilium policy allowed -s sourceID -d destinationID` outputs `Final tree decision: accept`? [Yes]
+- If no, check which labels the **source** has, with `cilium endpoint labels sourceID`, and
+add a rule for them as consumers in the destination's policy tree.
+
+5) `cilium monitor` outputs the traffic going out on the source node? [Yes]
+- If it's too difficult to read the monitor output, check the F.A.Q. on how to use
+`cilium monitor`. If the traffic is not going out, recompile the bpf program of the source
+endpoint `cilium endpoint recompile endpointID`, and check again.
+
+6) `cilium monitor` outputs the traffic going in on the destination node? [Yes]
+- If it's too difficult to read the monitor output, check the F.A.Q. on how to use
+`cilium monitor`. If the traffic is not going in, recompile the bpf program of the
+destination endpoint `cilium endpoint recompile endpointID`, and check again.
+
+If you still can't figure it out what's going on, reach us on [Slack](https://cilium.herokuapp.com).
+
+### How can I see the data stored in the KV Store by cilium?
+
+In **etcdv3**:
+```
+ETCDCTL_API=3 etcdctl get --from-key=true "cilium-net"
+```
+or, with curl which is can be too verbose and the data is base64 encoded:
+
+```
+curl 'http://IP:2379/v3alpha/kv/range' -X POST -d '{ "key" : "AA==", "range_end" : "AA==" }'
+```
+
+In **consul v0.6.4**:
+
+```
+curl "http://IP:8500/v1/kv/cilium-net?recurse"
+```