Add masquerade binding mechanism. #1746

SchSeba · 2018-11-22T21:58:50Z

Changes:

added new interface type (proxy)
added coreos/iptables package into glide
added unit tests
added functinal tests
updated some packages

What this PR does / why we need it:
Add a new binding mechanism call proxy.

This implementation insert iptable rules on the virt-launcher pod.
MASQUERADE all the outgoing traffic from the vm with the ip of the pod.
NAT (SNAT and DNAT) all the ports configured on the vm yaml.
NAT all the localhost traffic from the configured ports to the vm.

Release note:

proxy binding mechanism

This change is

SchSeba · 2018-11-23T18:45:55Z

/cc @rmohr

Can you please take a look.

SchSeba · 2018-11-27T13:00:38Z

/cc @phoracek @booxter

Can you please take a look on this PR ?

booxter

Some testing coverage expansion would be great.

booxter · 2018-11-27T22:43:40Z

pkg/virt-api/webhooks/validating-webhook/validating-webhook.go

@@ -868,6 +868,12 @@ func ValidateVirtualMachineInstanceSpec(field *k8sfield.Path, spec *v1.VirtualMa
 					Message: fmt.Sprintf("Slirp interface only implemented with pod network"),
 					Field:   field.Child("domain", "devices", "interfaces").Index(idx).Child("name").String(),
 				})
+			} else if iface.Proxy != nil && networkData.Pod == nil {


please add a unit test for this scenario

booxter · 2018-11-27T22:45:39Z

pkg/virt-controller/services/template.go

@@ -798,6 +798,15 @@ func getCniInterfaceList(vmi *v1.VirtualMachineInstance) (ifaceListString string
 	return
 }

+func haveProxyInterface(vmi *v1.VirtualMachineInstance) bool {


where is it used?

booxter · 2018-11-27T22:46:53Z

pkg/virt-launcher/virtwrap/api/converter.go

@@ -935,7 +935,7 @@ func Convert_v1_VirtualMachine_To_api_Domain(vmi *v1.VirtualMachineInstance, dom
 				domainIface.Address = addr
 			}

-			if iface.Bridge != nil {
+			if iface.Bridge != nil || iface.Proxy != nil {


is it possible to get a unit test for converting a vmi with a proxy interface?

booxter · 2018-11-27T22:50:38Z

pkg/virt-launcher/virtwrap/network/common.go

+
+	return iptablesObject.Append(table, chain, rulespec...)
+}
+func (h *NetworkUtilsHandler) GetAvailableAddrsFromCIDR(s string) ([]string, error) {


It's not exactly clear to me why these functions should be part of the handler and not just regular functions.

The function is here because I use it also as mock for the podinterface unit tests

Perhaps it makes sense in context of what we already have here.

booxter · 2018-11-27T22:51:52Z

pkg/virt-launcher/virtwrap/network/common.go

+		ips = append(ips, fmt.Sprintf("%s/%d", ip.String(), subnet))
+	}
+
+	if len(ips) < 4 {


Could you please explain in a comment why 4 is a special number here, and why you think this check belongs to this method (and not to its consumer who may probably be a better fit to tell whether the number is enough for their needs)?

I change the function name and also add a comment.

booxter · 2018-11-27T23:15:29Z

pkg/virt-launcher/virtwrap/network/podinterface.go

+}
+
+func (p *ProxyPodInterface) getVMNetwork() (string, string, error) {
+	ips, err := Handler.GetAvailableAddrsFromCIDR(p.vmNetworkCIDR)


how huge do we expect the CIDRs to be? Is it a problem that, while you need just two addresses, you are going to generate all of them inside the handler method? (Which, in theory could be thousands of throw away entries).

I change the function to GetHostAndGwAddressesFromCIDR just return the gw and the vm ip address.

The function stop after find 4 ips

booxter · 2018-11-27T23:17:59Z

pkg/virt-launcher/virtwrap/network/podinterface.go

+
+	err = Handler.LinkSetMaster(b.podNicLink, bridge)
+	if err != nil {
+		log.Log.Reason(err).Errorf("faild to connect interface %s to bridge %s", b.podInterfaceName, bridge.Name)


booxter · 2018-11-27T23:22:39Z

pkg/virt-launcher/virtwrap/network/podinterface.go

+	}
+
+	if len(ips) < 2 {
+		return "", "", fmt.Errorf("need 2 ip address in the CIDR but find only %d", len(ips))


"2 ip addresses", "found"

I removed this. the function now return gw and vm addresses or error.

booxter · 2018-11-27T23:24:41Z

tests/vmi_networking_test.go

+			_, err = virtClient.VirtualMachineInstance(tests.NamespaceTestDefault).Create(clientVMI)
+			Expect(err).ToNot(HaveOccurred())
+
+			By("waiting for the virtual machines to by ready")


by ready -> become ready?

booxter · 2018-11-27T23:26:45Z

tests/vmi_networking_test.go

+				&expect.BSnd{S: "echo $?\n"},
+				&expect.BExp{R: "0"},
+			}, 30)
+			Expect(err).ToNot(HaveOccurred())


could you please also add a negative check that ports not registered with the container are NOT accessible?

fabiand · 2018-11-28T09:39:59Z

api/openapi-spec/swagger.json

@@ -4538,6 +4538,9 @@
       "$ref": "#/definitions/v1.Port"
      }
     },
+     "proxy": {


Please consider to call this masquerade or nat.

To me at least, proxy is rather a process which does the relaying.
And I'd also like to keep the process bound proxying free, because I hope that we'll have some process bound proxying in future, as it's the only K8s compatible way to do things.

Why would the App Operator care if the proxy is implemented by a process, or as suggested here, by the kernel? I would assume that if we find an efficient way to do what you hope, we can just flip the implementation.

Why is kernel-based less k8s compatible?

(I'm not going for a holy war about the name proxy, just willing to understand your position)

Please reach out to @SchSeba to understand the differences between a kernel and process based proxy approach for kubernetes.

A user would usually not care, but if sees a spec, then I would like to leave the best hints to give a user transparency. That#s why I favor to have a more kernel-ish name for this implementation and proxy for a more process-ish implementation.

SchSeba · 2018-11-28T14:30:19Z

Rebase

booxter

Thanks. There is a comment from @fabiand about naming of the feature that I think we should address (answer or adopt in code) before merging.

SchSeba · 2018-12-02T20:03:05Z

There is an error for the masquerade test

{"component":"virt-launcher","kind":"","level":"error","msg":"Defining the VirtualMachineInstance failed.","name":"testvmiswlkx788gh4wjpwz78wlj55nmq8n8zft9rz66snkwq4wgnmf","namespace":"kubevirt-test-default","pos":"libvirt_helper.go:101","reason":"virError(Code=8, Domain=44, Message='invalid argument: could not find capabilities for arch=x86_64 domaintype=kvm ')","timestamp":"2018-12-02T16:40:29.751213Z","uid":"f0262580-f650-11e8-b453-525500d15501"}

for reference
https://jenkins.ovirt.org/job/kubevirt_kubevirt_standard-check-pr/2726//artifact/check-patch.k8s-multus-1.11.1-release.el7.x86_64/mock_logs/script/stdout_stderr.log

I will try to run the tests again

SchSeba · 2018-12-02T20:03:13Z

ci test please

SchSeba · 2018-12-02T20:03:44Z

@fabiand Change the interface as requested can you please take a look

fabiand

wfm - thanks!

fabiand

wfm

SchSeba · 2018-12-03T13:33:25Z

Multiple error on the tests

â€¢ Failure [109.562 seconds]
Networking
/root/go/src/kubevirt.io/kubevirt/tests/vmi_networking_test.go:48
  VirtualMachineInstance with masquerade binding mechanism
  /root/go/src/kubevirt.io/kubevirt/tests/vmi_networking_test.go:616
    should allow regular network connection [It]
    /root/go/src/kubevirt.io/kubevirt/tests/vmi_networking_test.go:620

    Expected error:
        <expect.TimeoutError>: 30000000000
        expect: timer expired after 30 seconds
    not to have occurred

    /root/go/src/kubevirt.io/kubevirt/tests/vmi_networking_test.go:671
------------------------------
STEP: creating two virtual machines
STEP: waiting for the virtual machines to become ready
{"component":"tests","kind":"VirtualMachineInstance","level":"info","msg":"Created virtual machine pod virt-launcher-testvmi7tqnq7hzsqmpv45n4jdll8bfkvt5fcvxsq4679bsd9","name":"testvmi7tqnq7hzsqmpv45n4jdll8bfkvt5fcvxsq467g4gdsf6vn48","namespace":"kubevirt-test-default","pos":"utils.go:270","timestamp":"2018-12-02T21:06:03.306216Z","uid":"13adb077-f676-11e8-8b73-525500d15501"}
{"component":"tests","kind":"VirtualMachineInstance","level":"info","msg":"VirtualMachineInstance defined.","name":"testvmi7tqnq7hzsqmpv45n4jdll8bfkvt5fcvxsq467g4gdsf6vn48","namespace":"kubevirt-test-default","pos":"utils.go:270","timestamp":"2018-12-02T21:06:18.189213Z","uid":"13adb077-f676-11e8-8b73-525500d15501"}
{"component":"tests","kind":"VirtualMachineInstance","level":"info","msg":"VirtualMachineInstance started.","name":"testvmi7tqnq7hzsqmpv45n4jdll8bfkvt5fcvxsq467g4gdsf6vn48","namespace":"kubevirt-test-default","pos":"utils.go:270","timestamp":"2018-12-02T21:06:18.207509Z","uid":"13adb077-f676-11e8-8b73-525500d15501"}
{"component":"tests","kind":"VirtualMachineInstance","level":"info","msg":"Created virtual machine pod virt-launcher-testvmi2mlr9ggcv8mrsdz2bt4n2bm5pv4nclwqp9pwcfw9ps","name":"testvmi2mlr9ggcv8mrsdz2bt4n2bm5pv4nclwqp9pwc6cpk8pztr6z","namespace":"kubevirt-test-default","pos":"utils.go:270","timestamp":"2018-12-02T21:07:06.265913Z","uid":"13b2a684-f676-11e8-8b73-525500d15501"}
{"component":"tests","kind":"VirtualMachineInstance","level":"info","msg":"VirtualMachineInstance defined.","name":"testvmi2mlr9ggcv8mrsdz2bt4n2bm5pv4nclwqp9pwc6cpk8pztr6z","namespace":"kubevirt-test-default","pos":"utils.go:270","timestamp":"2018-12-02T21:07:06.266135Z","uid":"13b2a684-f676-11e8-8b73-525500d15501"}
{"component":"tests","kind":"VirtualMachineInstance","level":"info","msg":"VirtualMachineInstance started.","name":"testvmi2mlr9ggcv8mrsdz2bt4n2bm5pv4nclwqp9pwc6cpk8pztr6z","namespace":"kubevirt-test-default","pos":"utils.go:270","timestamp":"2018-12-02T21:07:06.266509Z","uid":"13b2a684-f676-11e8-8b73-525500d15501"}
STEP: expose service for the server virtual machine
STEP: check ping to google
STEP: start a tcp server
STEP: Connect from the client
{"component":"tests","kind":"VirtualMachineInstance","level":"info","msg":"[{1 \r\n\r\n$  [$ ]} {3 echo test | nc -w 1 server-tcp 8080 1\u003e /dev/null\r\nnc: bad address 'server-tcp'\r\n$  [$ ]} {5 echo $?\r\n1\r\n$  []}]","name":"testvmi2mlr9ggcv8mrsdz2bt4n2bm5pv4nclwqp9pwc6cpk8pztr6z","namespace":"kubevirt-test-default","pos":"utils.go:1712","timestamp":"2018-12-02T21:07:48.452142Z","uid":""}
â€¢ [SLOW TEST:40.909 seconds]

Run functional tests with focus

/hold

booxter · 2018-12-04T15:16:50Z

@SchSeba you need to rebase it. Also, is hold label valid still?

booxter · 2018-12-04T23:12:37Z

tests/vmi_networking_test.go

@@ -594,6 +594,70 @@ var _ = Describe("Networking", func() {
 		})
 	})

+	FContext("VirtualMachineInstance with masquerade binding mechanism", func() {


Wait, you disabled all tests but this one.

Yes I know just want to be sure this test pass on the CI before I run it with all the other tests.

SchSeba · 2018-12-05T07:19:29Z

@booxter I will wait for the CI to finish and then I will remove the hold

SchSeba · 2018-12-05T09:59:01Z

The logs look good now.
on the network provider there is one error related to a know issue we have of the mac starting with "fe"

Now I think this PR is ready to be merge.
/hold cancel

fabiand · 2018-12-05T10:38:30Z

@mfranczy do you mind doing a sanity check?

If @booxter can then give it's blessing the we should be good I guess.

As always: Please keep an eye on the tests.

mfranczy

In general LGTM but I added a few comments.

mfranczy · 2018-12-05T11:30:41Z

pkg/virt-launcher/virtwrap/network/podinterface.go

+
+	err = Handler.LinkSetUp(bridgeNic)
+	if err != nil {
+		log.Log.Reason(err).Errorf("failed to bring link up for interface: %s", bridgeNic)


Wrong type, bridgeNic is not a string.
FilteredLogger.Errorf format %s has arg bridgeNic of wrong type *kubevirt.io/kubevirt/vendor/github.com/vishvananda/netlink.Dummy

should be changed to bridgeNic.Name

mfranczy · 2018-12-05T11:36:32Z

tests/vmi_networking_test.go

+			pingVirtualMachine(serverVMI, "8.8.8.8", "\\$ ")
+			pingVirtualMachine(clientVMI, "google.com", "\\$ ")
+
+			By("start a tcp server")


Please keep the same style everywhere, as example By("start a tcp server") should be changed to By("starting a tcp server")

mfranczy · 2018-12-05T11:42:19Z

pkg/virt-launcher/virtwrap/api/converter_test.go

+			vmi.Spec.Domain.Devices.Interfaces = []v1.Interface{iface1}
+
+			domain := vmiToDomain(vmi, c)
+			Expect(domain).ToNot(Equal(nil))


Might be changed to Expect(domain).ToNot(BeNil())

mfranczy · 2018-12-05T11:47:29Z

tests/vmi_networking_test.go

+			}, 30)
+			Expect(err).ToNot(HaveOccurred())
+
+			By("Reject the connection from the client to un registered port")


should be unregistered

mfranczy · 2018-12-05T12:04:54Z

pkg/virt-launcher/virtwrap/network/podinterface.go

+	p.podNicLink = link
+
+	// Get interface MTU
+	p.vif.Mtu = uint16(p.podNicLink.Attrs().MTU)


For me this conversion looks dangerous, the range of uint16 is 0 to 65535 and MTU is int type. I would add additional checks.

I don't think we need to add any other check here. I get the podNicLink from the system. I tried to create an interface with a bad MTU ( < 0 and > 65535) and it fails on the system level.

Okay, but what about jumbograms? Do we support it or not?
According to RFC doc

A "jumbogram" is an IPv6 packet containing a payload longer than
65,535 octets. This document describes the IPv6 Jumbo Payload
option, which provides the means of specifying such large payload
lengths. It also describes the changes needed to TCP and UDP to make
use of jumbograms.

Jumbograms are relevant only to IPv6 nodes that may be attached to
links with a link MTU greater than 65,575 octets, and need not be
implemented or understood by IPv6 nodes that do not support
attachment to links with such large MTUs.

So if some crazy guy will need to use it then this conversion will fail.
Or maybe there is some check somewhere else which prevents using such a high MTU?

Ah forget about jumbogram comment, virtio does not support it. But I still think there should be some additional check or VIF.Mtu field should be the same type. It's just my opinion after all, it's up to you.

I think it will be hard to chnage the variable to int because of the dhcp option

mtuArray := make([]byte, 2) binary.BigEndian.PutUint16(mtuArray, mtu) dhcpOptions := dhcp.Options{ dhcp.OptionSubnetMask: []byte(clientMask), dhcp.OptionRouter: []byte(routerIP), dhcp.OptionDomainNameServer: bytes.Join(dnsIPs, nil), dhcp.OptionInterfaceMTU: mtuArray, }

I can add if statement to check this < 0 and > 65535

I think it makes sense to guard the code and not rely on virtio limitations. A if statement is the way to go.

That being said... Isn't it the same bug that is perhaps present in other places of code? We may want to fix them too. I would be fine if e.g. this part is moved into a separate PR, as long as we are assured this happens.

I see this bug in at least one other place, for BridgePodInterface. (Not a surprise considering that a lot of this code is a copy-pasted and then adjusted code for bridge implementation.) @SchSeba please tell us what you prefer: fixing the issue here or sending a follow up PR. I am fine with both approaches. (I am not fine with leaving it not handled for long though.)

@booxter I just add this to the code for the bridge and the masquerade bindings.
Also added a unit test for that please see my last commit

mfranczy · 2018-12-05T12:07:27Z

pkg/virt-launcher/virtwrap/network/podinterface.go

+	Handler.StartDHCP(p.vif, p.gatewayAddr, p.bridgeInterfaceName, p.iface.DHCPOptions)
+}
+
+func (p *MasqueradePodInterface) decorateConfig() error {


What's the point fo returning an error here when you always return nil?

This function is part of an interface. if I change it I will need to check all the other bindings

mfranczy · 2018-12-05T12:14:17Z

tests/vmi_networking_test.go

+		It("should allow regular network connection", func() {
+			By("creating two virtual machines")
+			ports := []v1.Port{{Name: "http", Port: 8080}}
+			serverVMI = tests.NewRandomVMIWithMasqueradeInterfaceEphemeralDiskAndUserdata(tests.ContainerDiskFor(tests.ContainerDiskCirros), "#!/bin/bash\necho 'hello'\n", ports)


that's quite a long func name NewRandomVMIWithMasqueradeInterfaceEphemeralDiskAndUserdata :), may I suggest to rename it and move it to local Context scope? You're using this function only here so I have doubts if we need it in utils. My opinion is, if someone will need to use it somewhere else than he/she should moved it then.

Changes: * added new interface type (proxy) * added coreos/iptables package into glide * added unit tests * added functinal tests * updated some packages

SchSeba

@mfranczy Thanks for the review I updated the last commit with the requested changes can you please make another review?

SchSeba · 2018-12-05T13:12:39Z

pkg/virt-launcher/virtwrap/network/podinterface.go

+	p.podNicLink = link
+
+	// Get interface MTU
+	p.vif.Mtu = uint16(p.podNicLink.Attrs().MTU)


I don't think we need to add any other check here. I get the podNicLink from the system. I tried to create an interface with a bad MTU ( < 0 and > 65535) and it fails on the system level.

SchSeba · 2018-12-05T13:16:18Z

pkg/virt-launcher/virtwrap/network/podinterface.go

+
+	err = Handler.LinkSetUp(bridgeNic)
+	if err != nil {
+		log.Log.Reason(err).Errorf("failed to bring link up for interface: %s", bridgeNic)


SchSeba · 2018-12-05T13:17:42Z

pkg/virt-launcher/virtwrap/network/podinterface.go

+	Handler.StartDHCP(p.vif, p.gatewayAddr, p.bridgeInterfaceName, p.iface.DHCPOptions)
+}
+
+func (p *MasqueradePodInterface) decorateConfig() error {


This function is part of an interface. if I change it I will need to check all the other bindings

SchSeba · 2018-12-05T13:19:43Z

tests/vmi_networking_test.go

+		It("should allow regular network connection", func() {
+			By("creating two virtual machines")
+			ports := []v1.Port{{Name: "http", Port: 8080}}
+			serverVMI = tests.NewRandomVMIWithMasqueradeInterfaceEphemeralDiskAndUserdata(tests.ContainerDiskFor(tests.ContainerDiskCirros), "#!/bin/bash\necho 'hello'\n", ports)


SchSeba · 2018-12-05T13:20:35Z

tests/vmi_networking_test.go

+			}, 30)
+			Expect(err).ToNot(HaveOccurred())
+
+			By("Reject the connection from the client to un registered port")


SchSeba · 2018-12-05T13:20:40Z

tests/vmi_networking_test.go

+			pingVirtualMachine(serverVMI, "8.8.8.8", "\\$ ")
+			pingVirtualMachine(clientVMI, "google.com", "\\$ ")
+
+			By("start a tcp server")


mfranczy

LGTM

booxter · 2018-12-06T14:12:42Z

pkg/virt-launcher/virtwrap/network/podinterface.go

@@ -193,6 +193,10 @@ func (b *BridgePodInterface) discoverPodNetworkInterface() error {
 		b.vif.MAC = mac
 	}

+	if b.podNicLink.Attrs().MTU < 0 || b.podNicLink.Attrs().MTU > 65535 {
+		return fmt.Errorf("MTU value out of range ")


this trailing space... :)

Also, you could include the failed value too.

booxter · 2018-12-06T14:45:31Z

@SchSeba please add user guide docs

kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API size/XXL labels Nov 22, 2018

kubevirt-bot requested a review from rmohr November 23, 2018 18:45

kubevirt-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 27, 2018

kubevirt-bot requested review from booxter and phoracek November 27, 2018 13:00

booxter suggested changes Nov 27, 2018

View reviewed changes

fabiand requested changes Nov 28, 2018

View reviewed changes

SchSeba force-pushed the proxy-binding branch from 0180c54 to 39162e7 Compare November 28, 2018 14:29

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Nov 28, 2018

booxter approved these changes Nov 28, 2018

View reviewed changes

SchSeba added the sig/network label Nov 30, 2018

kubevirt-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 1, 2018

SchSeba force-pushed the proxy-binding branch from 39162e7 to 3561cec Compare December 2, 2018 15:34

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 2, 2018

SchSeba changed the title ~~Add proxy binding mechanism.~~ Add masquerade binding mechanism. Dec 2, 2018

fabiand reviewed Dec 3, 2018

View reviewed changes

fabiand approved these changes Dec 3, 2018

View reviewed changes

SchSeba force-pushed the proxy-binding branch from 3561cec to 5ef0385 Compare December 3, 2018 13:30

kubevirt-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 3, 2018

SchSeba force-pushed the proxy-binding branch 2 times, most recently from 2241a27 to cb4c06e Compare December 3, 2018 18:57

SchSeba force-pushed the proxy-binding branch from cb4c06e to 993d76a Compare December 3, 2018 20:18

kubevirt-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 4, 2018

SchSeba force-pushed the proxy-binding branch from 993d76a to 373d77f Compare December 4, 2018 12:12

kubevirt-bot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 4, 2018

booxter approved these changes Dec 4, 2018

View reviewed changes

SchSeba force-pushed the proxy-binding branch from 4b205e8 to c914d54 Compare December 4, 2018 17:04

booxter approved these changes Dec 4, 2018

View reviewed changes

booxter suggested changes Dec 4, 2018

View reviewed changes

SchSeba force-pushed the proxy-binding branch from c914d54 to d9acc0f Compare December 5, 2018 07:18

kubevirt-bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 5, 2018

mfranczy reviewed Dec 5, 2018

View reviewed changes

SchSeba added 3 commits December 5, 2018 16:52

Added proxy binding mechanism.

19096ba

Changes: * added new interface type (proxy) * added coreos/iptables package into glide * added unit tests * added functinal tests * updated some packages

renamed proxy interface to masquerade interface

2dd4f83

removed the service creation

b514edf

SchSeba force-pushed the proxy-binding branch from d9acc0f to b514edf Compare December 5, 2018 14:56

SchSeba commented Dec 5, 2018

View reviewed changes

mfranczy approved these changes Dec 5, 2018

View reviewed changes

booxter approved these changes Dec 5, 2018

View reviewed changes

Added statement to validate MTU size before parsing it to uint16.

b9a00d5

booxter reviewed Dec 6, 2018

View reviewed changes

booxter approved these changes Dec 6, 2018

View reviewed changes

booxter merged commit c222089 into kubevirt:master Dec 6, 2018

Add masquerade binding mechanism. #1746

Add masquerade binding mechanism. #1746

Conversation

SchSeba commented Nov 22, 2018 • edited Loading

SchSeba commented Nov 23, 2018

SchSeba commented Nov 27, 2018

booxter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fabiand Nov 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SchSeba commented Nov 28, 2018

booxter left a comment

Choose a reason for hiding this comment

SchSeba commented Dec 2, 2018

SchSeba commented Dec 2, 2018

SchSeba commented Dec 2, 2018

fabiand left a comment

Choose a reason for hiding this comment

fabiand left a comment

Choose a reason for hiding this comment

SchSeba commented Dec 3, 2018

booxter commented Dec 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SchSeba commented Dec 5, 2018

SchSeba commented Dec 5, 2018

fabiand commented Dec 5, 2018

mfranczy left a comment

Choose a reason for hiding this comment

mfranczy Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfranczy Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mfranczy Dec 5, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SchSeba left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SchSeba commented Nov 22, 2018 •

edited

Loading

fabiand Nov 28, 2018 •

edited

Loading

mfranczy Dec 5, 2018 •

edited

Loading

mfranczy Dec 5, 2018 •

edited

Loading

mfranczy Dec 5, 2018 •

edited

Loading

booxter commented Dec 6, 2018 •

edited

Loading