[KIALI-2520] Kiali Test Mesh Operator and Automatic Injection of Sidecar Support #41

gbaufake · 2019-02-13T15:25:06Z

Hello,

Created Kiali Test Mesh Operator (https://github.com/operator-framework/operator-sdk) which will work on Kubernetes and Openshift.

In the beginning, I added the complex-mesh and I intend to add other meshes (scale-mesh, bookinfo, redhat-demo and other meshes might be helpful)

Best Regards,
Guilherme Baufaker Rêgo

mwringe · 2019-02-13T16:35:53Z

You shouldn't have the processed yam file that have the sidecar already injected. This can change with each Istio version and we don't want to update everything each time.

gbaufake · 2019-02-13T16:40:04Z

@mwringe
You are totally right.
I know that is not the best option, I tried to make it flexible to install both istio and maistra proxy, since maistra doesn't have an istioctl command to embedded with own proxy.

Can we assume to use it with automatic injection side-car as option?

mwringe · 2019-02-13T18:25:02Z

Can we assume to use it with automatic injection side-car as option?

Can this not just be an option that someone passes? So that if they have automatic side car it does nothing, if they don't then we run kube-inject on it.

- Automatic/Manual Injection of Istio Sidecar (Configured by CR) - Installation of upstream istioctl version

jmazzitelli · 2019-02-27T19:44:38Z

I have 3.11 and Istio 1.1.snapshot 6. Now I tried this PR.

make operator-deploy-openshift

resulted in

$ make operator-deploy-openshift
About to deploy the Kiali Tesh Mesh Operator to OpenShift
oc process -f operator/kiali-test-mesh-operator/deploy/openshift/operator.yaml | oc create -f -
customresourcedefinition.apiextensions.k8s.io/installations.kiali-test-mesh.kiali.io created
clusterrolebinding.rbac.authorization.k8s.io/kiali-test-mesh-operator created
Error from server (NotFound): namespaces "kiali-test-mesh-operator" not found
Error from server (NotFound): namespaces "kiali-test-mesh-operator" not found
make: *** [operator-deploy-openshift] Error 1

So I created that project via oc new-project kiali-test-mesh-operator and ran it again:

$ make operator-deploy-openshift
About to deploy the Kiali Tesh Mesh Operator to OpenShift
oc process -f operator/kiali-test-mesh-operator/deploy/openshift/operator.yaml | oc create -f -
serviceaccount/kiali-test-mesh-operator created
deployment.apps/kiali-test-mesh-operator created
Error from server (AlreadyExists): customresourcedefinitions.apiextensions.k8s.io "installations.kiali-test-mesh.kiali.io" already exists
Error from server (AlreadyExists): clusterrolebindings.rbac.authorization.k8s.io "kiali-test-mesh-operator" already exists
make: *** [operator-deploy-openshift] Error 1

At this point I do see the operator pod running:

$ oc get pods -n kiali-test-mesh-operator
NAME                                        READY     STATUS    RESTARTS   AGE
kiali-test-mesh-operator-69667b769b-cbr95   1/1       Running   0          2m

I think you should at least make it fault tolerant such that a person doesn't get a failure due to the "Already Exist" errors.

jmazzitelli · 2019-02-27T19:45:58Z

The pod logs show this (it seems to hang here):


  | time="2019-02-27T19:41:58Z" level=info msg="Go Version: go1.10.3"
-- | --
  | time="2019-02-27T19:41:58Z" level=info msg="Go OS/Arch: linux/amd64"
  | time="2019-02-27T19:41:58Z" level=info msg="Version of operator-sdk: v0.4.0"
  | time="2019-02-27T19:41:58Z" level=info msg="Watching kiali-test-mesh-operator namespace."
  | {"level":"info","ts":1551296518.753746,"logger":"leader","msg":"Trying to become the leader."}
  | {"level":"info","ts":1551296518.801706,"logger":"leader","msg":"No pre-existing lock was found."}
  | {"level":"info","ts":1551296518.8055024,"logger":"leader","msg":"Became the leader."}
  | {"level":"info","ts":1551296518.8060036,"logger":"proxy","msg":"Starting to serve","Address":"127.0.0.1:8888"}
  | {"level":"info","ts":1551296518.806502,"logger":"ansible-controller","msg":"Watching resource","Options.Group":"kiali-test-mesh.kiali.io","Options.Version":"v1","Options.Kind":"Installation"}
  | {"level":"info","ts":1551296518.806678,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"installation-controller","source":"kind source: kiali-test-mesh.kiali.io/v1, Kind=Installation"}
  | {"level":"info","ts":1551296518.9069376,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"installation-controller"}
  | {"level":"info","ts":1551296519.0072517,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"installation-controller","worker count":1}

gbaufake · 2019-02-27T19:57:14Z

@jmazzitelli if you have the existing resource running, it will not create the cr because the command is falling... I will include the ignore-errors on the following command.

Thanks for spotting this.

…docs

gbaufake · 2019-03-07T13:55:06Z

ping @jmazzitelli @mwringe can we merge this?

jmazzitelli · 2019-03-11T16:13:50Z

I ran this make deploy-bookinfo-automatic-sidecar And I saw bookinfo namespace get created and all pods deployed. Then I wanted to deploy kiali so I want to kiali's Makefile and ran the make target openshift-deploy which undeploys all things with app=kiali in the istio-system namespace. I then saw my bookinfo demo get deleted for some reason - I do not know why. My "oc project" defaults to "kiali-test-mesh-operator", so I'm not sure why/how my bookinfo got deleted.

I then tried to re-install bookinfo but I couldn't because the CR still exists. How do I undeploy/delete the CR so I can re-try it again?

$ make deploy-bookinfo-automatic-sidecar
Deploy Bookinfo with Automatic Injection of the sidecar on Openshift
oc create -f operator/kiali-test-mesh-operator/deploy/cr/automatic-sidecar/bookinfo-cr.yaml
Error from server (AlreadyExists): error when creating "operator/kiali-test-mesh-operator/deploy/cr/automatic-sidecar/bookinfo-cr.yaml": bookinfos.bookinfo.kiali.io "bookinfo-installation" already exists
make: *** [deploy-bookinfo-automatic-sidecar] Error 1

gbaufake · 2019-03-11T16:18:54Z

@jmazzitelli you can remove everything with operator-remove-openshift and restart the process. I will include the delete targets of the resources.

gbaufake · 2019-03-11T16:25:34Z

@jmazzitelli make remove-bookinfo-automatic-sidecar should do the work now.

Makefile

jmazzitelli · 2019-03-11T18:56:17Z

This shows that I start clean (no operator, no bookinfo), I create operator, add bookinfo, but yet, no bookinfo pods show up?

[jmazzite@jmazzite kiali-test-mesh (gbaufake-ansible-to-operator)]$ oc get project kiali-test-mesh-operator
No resources found.
Error from server (NotFound): namespaces "kiali-test-mesh-operator" not found
[jmazzite@jmazzite kiali-test-mesh (gbaufake-ansible-to-operator)]$ oc get project bookinfo
No resources found.
Error from server (NotFound): namespaces "bookinfo" not found
[jmazzite@jmazzite kiali-test-mesh (gbaufake-ansible-to-operator)]$ oc project
Using project "default" on server "https://192.168.1.19:8443".
[jmazzite@jmazzite kiali-test-mesh (gbaufake-ansible-to-operator)]$ make operator-deploy-openshift
Remove Kiali Test Mesh Operator on Openshift
oc delete --ignore-not-found=true -f operator/kiali-test-mesh-operator/deploy/bookinfo-crd.yaml
oc delete --ignore-not-found=true -f operator/kiali-test-mesh-operator/deploy/complex_mesh-crd.yaml
oc delete --ignore-not-found=true -f operator/kiali-test-mesh-operator/deploy/service_account.yaml
oc delete --ignore-not-found=true -f operator/kiali-test-mesh-operator/deploy/role_binding.yaml
oc delete --ignore-not-found=true -f operator/kiali-test-mesh-operator/deploy/operator.yaml
oc delete namespace kiali-test-mesh-operator --ignore-not-found=true
Deploy Kiali Tesh Mesh Operator on Openshift
oc new-project kiali-test-mesh-operator
Now using project "kiali-test-mesh-operator" on server "https://192.168.1.19:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app centos/ruby-25-centos7~https://github.com/sclorg/ruby-ex.git

to build a new example application in Ruby.
oc create -f operator/kiali-test-mesh-operator/deploy/bookinfo-crd.yaml
customresourcedefinition.apiextensions.k8s.io/bookinfos.bookinfo.kiali.io created
oc create -f operator/kiali-test-mesh-operator/deploy/complex_mesh-crd.yaml
customresourcedefinition.apiextensions.k8s.io/complexmeshes.complexmesh.kiali.io created
oc create -f operator/kiali-test-mesh-operator/deploy/service_account.yaml
serviceaccount/kiali-test-mesh-operator created
oc create -f operator/kiali-test-mesh-operator/deploy/role_binding.yaml
clusterrolebinding.rbac.authorization.k8s.io/kiali-test-mesh-operator created
oc create -f operator/kiali-test-mesh-operator/deploy/operator.yaml
deployment.apps/kiali-test-mesh-operator created
[jmazzite@jmazzite kiali-test-mesh (gbaufake-ansible-to-operator)]$ oc get project bookinfo
No resources found.
Error from server (NotFound): namespaces "bookinfo" not found
[jmazzite@jmazzite kiali-test-mesh (gbaufake-ansible-to-operator)]$ oc project
Using project "kiali-test-mesh-operator" on server "https://192.168.1.19:8443".
[jmazzite@jmazzite kiali-test-mesh (gbaufake-ansible-to-operator)]$ make deploy-bookinfo-manual-sidecar
Deploy Bookinfo with Manual Injection of the sidecar on Openshift
oc create -f operator/kiali-test-mesh-operator/deploy/cr/manual-sidecar/bookinfo-cr.yaml
bookinfo.bookinfo.kiali.io/bookinfo-installation created
[jmazzite@jmazzite kiali-test-mesh (gbaufake-ansible-to-operator)]$ oc get project bookinfo
No resources found.
Error from server (NotFound): namespaces "bookinfo" not found
[jmazzite@jmazzite kiali-test-mesh (gbaufake-ansible-to-operator)]$ oc get project bookinfo
NAME       DISPLAY NAME   STATUS
bookinfo                  Active
[jmazzite@jmazzite kiali-test-mesh (gbaufake-ansible-to-operator)]$ oc get pods -n bookinfo
No resources found.

I never see any bookinfo pods show up. I looked in the operator pod logs and I see an error:

{"level":"error","ts":1552338143.0922954,"logger":"logging_event_handler","msg":"","name":"bookinfo-installation","namespace":"kiali-test-mesh-operator","gvk":"bookinfo.kiali.io/v1, Kind=Bookinfo","event_type":"runner_on_failed","job":"7075431637056415105","EventData.Task":"Deploy details-v1 with manual injection","EventData.TaskArgs":"_ansible_version=2.7.6, definition=2019-03-11T21:02:22.342051Z\twarn\tmodel\tFailed to decode proto: \"unknown field \\\"dnsRefreshRate\\\" in v1alpha1.MeshConfig\". Trying decode with AllowUnknownFields=true\napiVersion: extensions/v1beta1\nkind: Deployment\nmetadata:\n  creationTimestamp: null\n  name: details-v1\n  namespace: bookinfo\nspec:\n  replicas: 1\n  strategy: {}\n  template:\n    metadata:\n      annotations:\n        sidecar.istio.io/status: '{\"version\":\"30110fc7563006252f04bd233bac8449e3bc3e142835fbf01861cbb2cf4f1b5c\",\"initContainers\":[\"istio-init\"],\"containers\":[\"istio-proxy\"],\"volumes\":[\"istio-envoy\",\"istio-certs\"],\"imagePullSecrets\":null}'\n      creationTimestamp: null\n      labels:\n        app: details\n        version: v1\n    spec:\n      containers:\n      - image: istio/examples-bookinfo-details-v1:1.10.0\n        imagePullPolicy: IfNotPresent\n        name: details-container\n        ports:\n        - containerPort: 9080\n        resources: {}\n      - args:\n        - proxy\n        - sidecar\n        - --domain\n        - $(POD_NAMESPACE).svc.cluster.local\n        - --configPath\n        - /etc/istio/proxy\n        - --binaryPath\n        - /usr/local/bin/envoy\n        - --serviceCluster\n        - details.$(POD_NAMESPACE)\n        - --drainDuration\n        - 45s\n        - --parentShutdownDuration\n        - 1m0s\n        - --discoveryAddress\n        - istio-pilot.istio-system:15010\n        - --zipkinAddress\n        - zipkin.istio-system:9411\n        - --connectTimeout\n        - 10s\n        - --proxyAdminPort\n        - \"15000\"\n        - --concurrency\n        - \"2\"\n        - --controlPlaneAuthPolicy\n        - NONE\n        - --statusPort\n        - \"15020\"\n        - --applicationPorts\n        - \"9080\"\n        env:\n        - name: POD_NAME\n          valueFrom:\n            fieldRef:\n              fieldPath: metadata.name\n        - name: POD_NAMESPACE\n          valueFrom:\n            fieldRef:\n              fieldPath: metadata.namespace\n        - name: INSTANCE_IP\n          valueFrom:\n            fieldRef:\n              fieldPath: status.podIP\n        - name: ISTIO_META_POD_NAME\n          valueFrom:\n            fieldRef:\n              fieldPath: metadata.name\n        - name: ISTIO_META_CONFIG_NAMESPACE\n          valueFrom:\n            fieldRef:\n              fieldPath: metadata.namespace\n        - name: ISTIO_META_INTERCEPTION_MODE\n          value: REDIRECT\n        - name: ISTIO_METAJSON_LABELS\n          value: |\n            {\"app\":\"details\",\"version\":\"v1\"}\n        image: docker.io/istio/proxyv2:1.1.0-rc.3\n        imagePullPolicy: IfNotPresent\n        name: istio-proxy\n        ports:\n        - containerPort: 15090\n          name: http-envoy-prom\n          protocol: TCP\n        readinessProbe:\n          failureThreshold: 30\n          httpGet:\n            path: /healthz/ready\n            port: 15020\n          initialDelaySeconds: 1\n          periodSeconds: 2\n        resources:\n          limits:\n            cpu: \"2\"\n            memory: 128Mi\n          requests:\n            cpu: 100m\n            memory: 128Mi\n        securityContext:\n          readOnlyRootFilesystem: true\n          runAsUser: 1337\n        volumeMounts:\n        - mountPath: /etc/istio/proxy\n          name: istio-envoy\n        - mountPath: /etc/certs/\n          name: istio-certs\n          readOnly: true\n      initContainers:\n      - args:\n        - -p\n        - \"15001\"\n        - -u\n        - \"1337\"\n        - -m\n        - REDIRECT\n        - -i\n        - '*'\n        - -x\n        - \"\"\n        - -b\n        - \"9080\"\n        - -d\n        - \"15020\"\n        image: docker.io/istio/proxy_init:1.1.0-rc.3\n        imagePullPolicy: IfNotPresent\n        name: istio-init\n        resources:\n          limits:\n            cpu: 100m\n            memory: 50Mi\n          requests:\n            cpu: 10m\n            memory: 10Mi\n        securityContext:\n          capabilities:\n            add:\n            - NET_ADMIN\n      volumes:\n      - emptyDir:\n          medium: Memory\n        name: istio-envoy\n      - name: istio-certs\n        secret:\n          optional: true\n          secretName: istio.default\nstatus: {}\n, _ansible_selinux_special_fs=['fuse', 'nfs', 'vboxsf', 'ramfs', '9p'], _ansible_no_log=False, _ansible_module_name=k8s, _ansible_debug=False, _ansible_verbosity=2, _ansible_keep_remote_files=False, _ansible_syslog_facility=LOG_USER, _ansible_socket=None, state=present, _ansible_diff=False, _ansible_remote_tmp=~/.ansible/tmp, _ansible_shell_executable=/bin/sh, _ansible_check_mode=False, _ansible_tmpdir=/root/.ansible/tmp/ansible-tmp-1552338142.42-211040099608084/","EventData.FailedTaskPath":"/opt/ansible/roles/common_tasks/deploy_istio_item.yml:20","error":"[playbook task failed]","stacktrace":"github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/travis/gopath/src/github.com/operator-framework/operator-sdk/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/operator-framework/operator-sdk/pkg/ansible/events.loggingEventHandler.Handle\n\t/home/travis/gopath/src/github.com/operator-framework/operator-sdk/pkg/ansible/events/log_events.go:84"}

gbaufake · 2019-03-11T21:16:39Z

it seems there is a problem on istio init docker.
Do you have the logs on this pod?

jmazzitelli · 2019-03-11T23:44:20Z

there is a problem on istio init docker.
Do you have the logs on this pod?

I don't have a pod called "istio init" - this is what I have:

$ oc get pods -n kiali-test-mesh-operator
NAME                                        READY     STATUS    RESTARTS   AGE
kiali-test-mesh-operator-6b499dcc7c-nlckp   1/1       Running   0          4h

$ oc get pods -n istio-system
NAME                                           READY     STATUS      RESTARTS   AGE
istio-citadel-cdd9f99d7-h8zr4                  1/1       Running     0          6h
istio-cleanup-secrets-1.1.0-rc.3-rcb58         0/1       Completed   0          6h
istio-galley-99d64f5bc-hkrxf                   1/1       Running     0          6h
istio-ingressgateway-7cb9d9947-lwbkl           1/1       Running     0          6h
istio-pilot-67d7bdcbc8-84hd6                   2/2       Running     0          6h
istio-policy-646b79bd88-mf8xd                  2/2       Running     3          6h
istio-security-post-install-1.1.0-rc.3-kblc9   0/1       Completed   0          6h
istio-sidecar-injector-7bf6cbc9bb-hwgvk        1/1       Running     0          6h
istio-telemetry-fc87b5b6-564td                 2/2       Running     3          6h
kiali-b9fb6fd-sj5fw                            1/1       Running     0          4h
prometheus-89bc5668c-9jhxl                     1/1       Running     0          6h

jmazzitelli · 2019-03-12T20:28:52Z

I run make operator-deploy-openshift and immediately run make deploy-bookinfo-manual-sidecar -- what I see the bookinfo namespace get created, and then I see the pods come in, but immediately I see the pods all go into "Terminating" state and then disappear

jmazzitelli · 2019-03-12T22:11:58Z

@gbaufake FYI: i was running istio 1.1.0-rc3 (they just released RC4, I might move up to that now)

gbaufake · 2019-03-13T13:53:15Z

@jmazzitelli if you want to use the manual injection of the sidecar. it needs to use the same version or compatible one with the istio that you are running that is the issue.

I created a new image with 1.1.rc4

jmazzitelli

Running with Istio 1.1.0-rc4. I was able to execute these two commands and see things deploy/run without any problems:

make operator-deploy-openshift
make deploy-bookinfo-manual-sidecar

Kiali Test Mesh Operator

7b62049

gbaufake requested review from mwringe and mattmahoneyrh February 13, 2019 15:25

gbaufake mentioned this pull request Feb 13, 2019

Error "The resourceVersion for the provided watch is too old" jaegertracing/jaeger-operator#103

Closed

Adding Read me on Kiali Test Mesh Operator

8117c09

gbaufake force-pushed the ansible-to-operator branch from db562e0 to 8117c09 Compare February 13, 2019 16:07

gbaufake mentioned this pull request Feb 13, 2019

Ansible Operator on an Infinite Loop operator-framework/operator-sdk#1101

Closed

Adding Reconciling to Zero

c6fab4e

gbaufake added the do not merge label Feb 17, 2019

gbaufake force-pushed the ansible-to-operator branch 3 times, most recently from 8eaccd8 to 23e77e4 Compare February 18, 2019 12:42

Guilherme Baufaker Rêgo added 8 commits February 18, 2019 20:18

Inclusion of some capabilities

f8c5d57

- Automatic/Manual Injection of Istio Sidecar (Configured by CR) - Installation of upstream istioctl version

Modification on Checkers to match OCP4

0f47add

Moving Templates to Complex Mesh Folder

3900a42

Modifications on the Makefile

24bb480

Including OC on Dockerfile

d22d2fd

Modifications on CRD

7cd6d7a

Changes on Checker to make it simpler

e3d2003

Including Bookinfo with Automatic Injection/Manual Injection

27667a0

gbaufake force-pushed the ansible-to-operator branch from 23e77e4 to 27667a0 Compare February 21, 2019 00:22

Fixing the Typos

472648d

gbaufake removed the do not merge label Feb 22, 2019

Guilherme Baufaker Rêgo added 4 commits February 22, 2019 16:21

Including Manual Case Injection for Bookinfo

b5948f5

Remove Namespace from K8s creation

17a81b8

Making CR as minimum resource

bf94f0d

Including Readme

dc64cb1

Guilherme Baufaker Rêgo added 4 commits March 6, 2019 17:18

Separate Resources

a64a2b9

More work on Separate Resources

0a55e75

Separating Resources

b839437

Moving CR, Updating Readmes, Remove travis.yml and moving scale mesh …

198520a

…docs

gbaufake changed the title ~~Kiali Test Mesh Operator~~ [KIALI-1924] Kiali Test Mesh Operator Mar 7, 2019

gbaufake changed the title ~~[KIALI-1924] Kiali Test Mesh Operator~~ [KIALI-2520] Kiali Test Mesh Operator and Automatic Injection of Sidecar Support Mar 7, 2019

Including the Deletion of CR on Makefile and README.adoc

3d6ed73

jmazzitelli requested changes Mar 11, 2019

View reviewed changes

Makefile Outdated Show resolved Hide resolved

Makefile Show resolved Hide resolved

Adding Namespace

adac6a0

gbaufake requested a review from jmazzitelli March 12, 2019 19:52

gbaufake mentioned this pull request Mar 12, 2019

Ansible Operator Playbook is running twice operator-framework/operator-sdk#1201

Closed

Guilherme Baufaker Rêgo added 2 commits March 13, 2019 14:41

Changing Default Manual CR to IstioCR

b02efb4

Updating to 0.5.0

1bf6df0

Guilherme Baufaker Rêgo added 2 commits March 13, 2019 16:27

Removing Checkers

72ad3f1

Refactor to use only main.yml

d0c06d5

jmazzitelli approved these changes Mar 13, 2019

View reviewed changes

gbaufake merged commit d86e154 into kiali:master Mar 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[KIALI-2520] Kiali Test Mesh Operator and Automatic Injection of Sidecar Support #41

[KIALI-2520] Kiali Test Mesh Operator and Automatic Injection of Sidecar Support #41

gbaufake commented Feb 13, 2019

mwringe commented Feb 13, 2019

gbaufake commented Feb 13, 2019

mwringe commented Feb 13, 2019

jmazzitelli commented Feb 27, 2019

jmazzitelli commented Feb 27, 2019

gbaufake commented Feb 27, 2019

gbaufake commented Mar 7, 2019

jmazzitelli commented Mar 11, 2019

gbaufake commented Mar 11, 2019

gbaufake commented Mar 11, 2019 •

edited

jmazzitelli commented Mar 11, 2019 •

edited

gbaufake commented Mar 11, 2019 •

edited

jmazzitelli commented Mar 11, 2019

jmazzitelli commented Mar 12, 2019

jmazzitelli commented Mar 12, 2019

gbaufake commented Mar 13, 2019

jmazzitelli left a comment

[KIALI-2520] Kiali Test Mesh Operator and Automatic Injection of Sidecar Support #41

[KIALI-2520] Kiali Test Mesh Operator and Automatic Injection of Sidecar Support #41

Conversation

gbaufake commented Feb 13, 2019

mwringe commented Feb 13, 2019

gbaufake commented Feb 13, 2019

mwringe commented Feb 13, 2019

jmazzitelli commented Feb 27, 2019

jmazzitelli commented Feb 27, 2019

gbaufake commented Feb 27, 2019

gbaufake commented Mar 7, 2019

jmazzitelli commented Mar 11, 2019

gbaufake commented Mar 11, 2019

gbaufake commented Mar 11, 2019 • edited

jmazzitelli commented Mar 11, 2019 • edited

gbaufake commented Mar 11, 2019 • edited

jmazzitelli commented Mar 11, 2019

jmazzitelli commented Mar 12, 2019

jmazzitelli commented Mar 12, 2019

gbaufake commented Mar 13, 2019

jmazzitelli left a comment

Choose a reason for hiding this comment

gbaufake commented Mar 11, 2019 •

edited

jmazzitelli commented Mar 11, 2019 •

edited

gbaufake commented Mar 11, 2019 •

edited