Add flowType field for Flow Exporter #2000

dreamtalen · 2021-03-26T00:14:36Z

Fixes #1925
In this PR, we implemented the logic of flowType value assignment.
We distinguished Pod-To-Pod flows and Pod-To-External flows using the podCIDRs of all nodes in the k8s cluster.

srikartati

Thanks @dreamtalen for working on this.

Retreiving podCIDRs from nodes to maintain Pod Subnets is only possible solution. However, not sure if this is completely foolproof. Please see the comment.

Discussion with @antoninbas offline: One more option is to use nodeRouteController, which is part of AntreaController that takes the responsibility of install routes for antrea-gw on every node(route client). It has podCIDR info for all nodes.
Ok with nodeSpec solution, if it serves the purpose. Otherwise, nodeRouteController should be explored.

srikartati · 2021-03-26T05:38:03Z

pkg/agent/flowexporter/exporter/exporter.go

+		if i == subnet {
+			return true


why not use i.Contains(subnet) to skip the subnet? i is not a great variable name in this context IMO.

Thanks, net.IPNet.Contain function can only reports whether the network includes ip. I added some logic to judge whether the network includes the new network.

srikartati · 2021-03-26T05:45:49Z

pkg/agent/flowexporter/exporter/exporter.go

+
+func fetchPodSubnets(k8sClient kubernetes.Interface) ([]*net.IPNet, error) {
+	podSubnets := []*net.IPNet{}
+	nodeList, err := k8sClient.CoreV1().Nodes().List(context.TODO(), metav1.ListOptions{})


As this is done as part of the initialization of Flow Exporter, can we be sure that all the nodes with Antrea agents are up?

We discussed offline and current solution is if source/destination ip is not known we will do the query and update the podCIDRs first, then update the flow type.

please integrate with the NodeRouteController, it already has this information

Updated the PR integrating with the NodeRouteController.

codecov-io · 2021-03-30T02:21:14Z

Codecov Report

Merging #2000 (c52b20d) into main (e027c7b) will increase coverage by 1.16%.
The diff coverage is 64.00%.

@@            Coverage Diff             @@
##             main    #2000      +/-   ##
==========================================
+ Coverage   60.87%   62.03%   +1.16%     
==========================================
  Files         268      269       +1     
  Lines       20236    20486     +250     
==========================================
+ Hits        12319    12709     +390     
+ Misses       6633     6460     -173     
- Partials     1284     1317      +33

Flag	Coverage Δ
e2e-tests	`24.45% <6.00%> (?)`
kind-e2e-tests	`51.74% <56.00%> (+0.23%)`	⬆️
unit-tests	`41.42% <40.00%> (-0.12%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pkg/util/ip/ip.go	`80.30% <50.00%> (ø)`
...gent/controller/noderoute/node_route_controller.go	`46.39% <53.57%> (+0.55%)`	⬆️
pkg/agent/flowexporter/exporter/exporter.go	`71.70% <80.00%> (+2.44%)`	⬆️
pkg/agent/nodeportlocal/k8s/npl_controller.go	`52.43% <0.00%> (-7.73%)`	⬇️
pkg/apiserver/certificate/certificate.go	`69.86% <0.00%> (-6.85%)`	⬇️
pkg/agent/route/route_linux.go	`40.18% <0.00%> (-2.51%)`	⬇️
pkg/antctl/raw/traceflow/command.go	`25.73% <0.00%> (-0.61%)`	⬇️
pkg/ovs/openflow/ofctrl_packetin.go	`35.08% <0.00%> (ø)`
pkg/agent/cniserver/server.go	`68.91% <0.00%> (+0.32%)`	⬆️
pkg/agent/openflow/client.go	`60.37% <0.00%> (+0.53%)`	⬆️
... and 22 more

srikartati · 2021-03-31T00:28:21Z

pkg/agent/flowexporter/exporter/exporter.go

+		if exp.nodeRouteController != nil {
+			exp.podSubnets = exp.nodeRouteController.GetPodSubnetsFromAllNodes()
+			klog.V(4).Infof("Updated Pod subnets: %v", exp.podSubnets)
+		}


Do we have to do this every time we create the flow record?

srikartati · 2021-03-31T00:35:09Z

pkg/agent/flowexporter/exporter/exporter.go

@@ -551,3 +553,34 @@ func (exp *flowExporter) sendDataSet() (int, error) {
 	klog.V(4).Infof("Data set sent successfully. Bytes sent: %d", sentBytes)
 	return sentBytes, nil
 }
+
+func (exp *flowExporter) findFlowType(record flowexporter.FlowRecord) uint8 {
+	if record.Conn.Mark == openflow.ServiceCTMark || exp.ipInPodSubnets(record.Conn.TupleOrig.SourceAddress) && exp.ipInPodSubnets(record.Conn.TupleOrig.DestinationAddress) {


I think this has to be modified.
Do you mean the following to represent Pod-To-Pod flows and Pod-To-Service flows?
exp.ipInPodSubnets(record.Conn.TupleOrig.SourceAddress) && (record.Conn.Mark == openflow.ServiceCTMark || exp.ipInPodSubnets(record.Conn.TupleOrig.DestinationAddress))

srikartati · 2021-03-31T00:45:31Z

pkg/agent/flowexporter/exporter/exporter.go

+			klog.V(4).Infof("Updated Pod subnets: %v", exp.podSubnets)
+		}
+		if !exp.ipInPodSubnets(record.Conn.TupleOrig.SourceAddress) {
+			return ipfixregistry.FromExternal


We do not support External-To-Pod flows (nodePort service) for now. They will be ignored in the connection flow dump. Prefer removing this to avoid confusion.

srikartati · 2021-03-31T00:47:36Z

pkg/agent/flowexporter/exporter/exporter.go

+	for _, subnet := range exp.podSubnets {
+		if subnet.Contains(ip) {
+			return true
+		}
+	}


Currently, we have fixed mask(same size) for all podCIDRs. Based on that we could optimize further.
Ok to keep this for now and consider the optimization later.

jianjuns · 2021-03-31T21:58:58Z

pkg/agent/flowexporter/utils.go

@@ -27,3 +31,16 @@ func NewConnectionKey(conn *Connection) ConnectionKey {
 		strconv.FormatUint(uint64(conn.TupleOrig.Protocol), 10),
 	}
 }
+
+func IsConnectionDying(conn *Connection) bool {
+	// "TIME_WAIT" state indicates local endpoint has closed the connection


Better to add '.' at the end of the sentence.

Thanks Jianjun, I fing some of your comments are focusing on @zyiou 's change in PR #1904. Yiou, could you please take a look?

jianjuns · 2021-03-31T22:01:52Z

pkg/agent/flowexporter/connections/conntrack_ovs.go

@@ -125,7 +125,7 @@ func (ct *connTrackOvsCtl) ovsAppctlDumpConnections(zoneFilter uint16) ([]*flowe

 // flowStringToAntreaConnection parses the flow string and converts to Antrea connection.
 // Example of flow string:
-// "tcp,orig=(src=10.10.1.2,dst=10.10.1.3,sport=45170,dport=2379,packets=80743,bytes=5416239),reply=(src=10.10.1.3,dst=10.10.1.2,sport=2379,dport=45170,packets=63361,bytes=4811261),start=2020-07-24T05:07:01.591,id=462801621,mark=32,zone=65520,status=SEEN_REPLY|ASSURED|CONFIRMED|SRC_NAT_DONE|DST_NAT_DONE,timeout=86397"
+// "tcp,orig=(src=127.0.0.1,dst=127.0.0.1,sport=45218,dport=2379,packets=320108,bytes=24615344),reply=(src=127.0.0.1,dst=127.0.0.1,sport=2379,dport=45218,packets=239595,bytes=24347883),start=2020-07-24T05:07:03.998,id=3750535678,status=SEEN_REPLY|ASSURED|CONFIRMED|SRC_NAT_DONE|DST_NAT_DONE,timeout=86399,protoinfo=(state_orig=ESTABLISHED,state_reply=ESTABLISHED,wscale_orig=7,wscale_reply=7,flags_orig=WINDOW_SCALE|SACK_PERM|MAXACK_SET,flags_reply=WINDOW_SCALE|SACK_PERM|MAXACK_SET)"


I do not mean we must do it in this PR, but in general I feel such OVS level parsing should be put into pkg/ovs/

Noted. We are planning to decouple conntrack polling and exporting process where we can move ovs specific parsing or handling to pkg/ovs and decouple conntrack part. #1278

jianjuns · 2021-03-31T22:06:09Z

pkg/agent/flowexporter/connections/conntrack_linux.go

@@ -155,6 +155,10 @@ func NetlinkFlowToAntreaConnection(conn *conntrack.Flow) *flowexporter.Connectio
 		SourcePodName:           "",


This file probably worth a separate pkg under pkg/agent/util/conntrack. Again not saying we must do in this PR.

jianjuns · 2021-03-31T22:08:35Z

pkg/agent/flowexporter/exporter/exporter.go

+	} else {
+		// Update Pod subnets to distinguish Pod-To-External flows.
+		if exp.nodeRouteController != nil {
+			exp.podSubnets = exp.nodeRouteController.GetPodSubnetsFromAllNodes()


Do we call it for every flow? Then we should optimize here; probably by maintaining a single map in NodeRouteController and look up there, or even require a podCIDR parameter to add flow type.

We call it if a flow record is not a Pod-To-Pod or Pod-To-Service record. Since new nodes may be added into the cluster anytime, current solution is that if we failed to judge current record is a Pod-To-Pod or Pod-To-Service record, we will fetch the latest PodCIDRs from NodeRouteController, then distinguish if it is a Pod-To-External flow.

But when this happens - failed to judge current record is a Pod-To-Pod or Pod-To-Service record?

To be brief, if the destination IP of a record is not in all PodSubnets we currently known && it's not a Pod-To-Service record.

Does it mean every flow to external will end up here?

Then we should optimize in my mind.

Agreed. @dreamtalen Please optimize this as discussed offline.

Thanks, updated PR with following optimization:
Maintained a PodCIDRIPsMap in NodeRouteController to avoid looking up PodSubsets in flow exporter when send record.
Took advantage of the fixed mask size among PodCIDRs, using Map to accelerate the judgement of IP in PodSubnets.

srikartati · 2021-04-02T04:43:59Z

pkg/agent/controller/noderoute/node_route_controller.go

@@ -97,7 +103,9 @@ func NewNodeRouteController(
 		nodeLister:       nodeInformer.Lister(),
 		nodeListerSynced: nodeInformer.Informer().HasSynced,
 		queue:            workqueue.NewNamedRateLimitingQueue(workqueue.NewItemExponentialFailureRateLimiter(minRetryDelay, maxRetryDelay), "noderoute"),
-		installedNodes:   cache.NewIndexer(nodeRouteInfoKeyFunc, cache.Indexers{nodeRouteInfoPodCIDRIndexName: nodeRouteInfoPodCIDRIndexFunc})}
+		installedNodes:   cache.NewIndexer(nodeRouteInfoKeyFunc, cache.Indexers{nodeRouteInfoPodCIDRIndexName: nodeRouteInfoPodCIDRIndexFunc}),
+		PodCIDRIPsMap:    make(map[string]int),


installedNodes (cache.indexer) already have this mapping functionality of searching using podCIDR through nodeRouteInfoPodCIDRIndexFunc. We do not need a separate map. Please take a look.

Thanks, I updated code using nodeRouteInfoPodCIDRIndexFunc.

srikartati

Overall logic is LGTM. I just had a couple of comments.

Wondering if you got a chance to test this out manually to make sure Flow Aggregator exports Pod-To-External flows.
Not sure if adding an e2e test case for Pod-To-External flow is trivial. If its not trivial we probably can take it up in a different PR. At least we should test manually.

srikartati · 2021-04-02T20:45:59Z

pkg/agent/controller/noderoute/node_route_controller.go

+		}
+	}
+	ipCIDRStr := ipCIDR.String()
+	nodesHaveSamePodCIDR, _ := c.installedNodes.ByIndex(nodeRouteInfoPodCIDRIndexName, ipCIDRStr)


nodeInCluster is better name. I feel the context for nodesHaveSamePodCIDR is different.

Thanks, addressed.

srikartati · 2021-04-02T20:50:23Z

pkg/agent/controller/noderoute/node_route_controller.go

+	PodIPv4CIDRMaskSize = 24
+	IPv4BitLen          = 32
+	PodIPv6CIDRMaskSize = 64
+	IPv6BitLen          = 128


These could be gathered runtime from local podCIDRs. Hardcoded values may work fine for now.. just want to confirm with @antoninbas

It's fine to assume fixed size CIDRs across the cluster for now (because this is how Node IPAM works), but it is not fine to assume that the size is 24 (for IPv4) or 64 (for IPv6). The cluster admin can easily change this by providing a different value with --node-cidr-mask-size for kube-controller-manager.

Thanks, Antonin.

Thanks, I add the logic of gathering PodIPv4/v6CIDRMaskSize from local podCIDRs, if it doesn't exist, using 24 (for IPv4) or 64 (for IPv6) as default value instead.

srikartati · 2021-04-02T20:51:26Z

pkg/agent/controller/noderoute/node_route_controller.go

@@ -615,3 +621,30 @@ func GetNodeAddr(node *corev1.Node) (net.IP, error) {
 	}
 	return ipAddr, nil
 }
+
+func (c *Controller) IPInPodSubnets(ip net.IP) bool {


Could we add a simple unit test for this method?

Sure, added a unit test for IPInPodSubnets function.
For Pod-To-External flows, I have tested manually by watching the records on the ipfix-collector side.

srikartati

Thanks for adding the unit test and sharing the result of the manual test. Please add a TODO in e2e tests to add a test that checks the functionality of Pod-To-External flow.

LGTM except for the corner case situation of network policy only mode.

srikartati · 2021-04-06T20:34:48Z

pkg/agent/flowexporter/exporter/exporter.go

+		klog.Warningf("Can't find flowType without nodeRouteController")
+		return 0
+	}
+	if exp.nodeRouteController.IPInPodSubnets(record.Conn.TupleOrig.SourceAddress) {


This will not work for network-policy only mode traffic as the installed node cache indexer is not maintained at all.
https://github.com/vmware-tanzu/antrea/blob/main/pkg/agent/controller/noderoute/node_route_controller.go#L278

I think we need to check the network subnet of local nodeAddr in nodeConfig to determine if the IP address is inside cluster or not. @jianjuns Could you please confirm if this check is correct or not for network policy only mode?

Thanks Srikar for pointing out it, added a TODO right now and wait for Jianjun's opinion.

Thanks for adding these lines to address this.

My initial assessment on node network based identification is not correct and was mixed up with something else. There is no straightforward way to detect PodIPs at Antrea agent when operating in network policy only mode.
Therefore, we are supporting Pod-To-External flows for all traffic modes except for the special mode of network policy only mode. Pod-To-Pod flows and Pod-To-Service flows are supported in network policy only mode. There is no regression here.

antoninbas · 2021-04-07T18:51:45Z

pkg/agent/controller/noderoute/node_route_controller.go

+		} else {
+			podIPv4CIDRMaskSize = defaultPodIPv4CIDRMaskSize
+		}


can you provide some clarity (and maybe add a comment) as to why this is needed? shouldn't we just return false at this point?

same for the v6 case below

I agree. I thought it might be ok since it is the default setting in Kube-control-manager.
However, thinking more about it it is better to not resolve the flow type and print a warning. This situation also arises for network policy only mode.
https://github.com/vmware-tanzu/antrea/blob/main/pkg/agent/config/node_config.go#L113

Any thoughts @dreamtalen ?

I agree too, addressed.

antoninbas · 2021-04-07T18:53:01Z

test/e2e/flowaggregator_test.go

+	if !strings.Contains(record, fmt.Sprintf("%s: %d", "flowType", flowType)) {
+		t.Errorf("Record does not have correct flowType")
+	}


nit: there is an assert.Contains assertion

Thanks, addressed.

srikartati

LGTM except for one nit.

srikartati · 2021-04-07T23:47:01Z

pkg/agent/controller/noderoute/node_route_controller.go

@@ -52,6 +52,9 @@ const (
 	ovsExternalIDNodeName = "node-name"

 	nodeRouteInfoPodCIDRIndexName = "podCIDR"
+
+	IPv4BitLen = 32


there are constants for these lengths in net package: https://golang.org/pkg/net/

Thanks, I only found bytes length constants, so I changed the code to:

IPv4BitLen = net.IPv4len * 8 IPv6BitLen = net.IPv6len * 8

Thanks for checking.
Realized that this is available here: https://github.com/vmware-tanzu/antrea/blob/main/pkg/util/ip/ip.go#L26

Thanks, I capitalized these constant and updated the code.

srikartati · 2021-04-07T23:56:25Z

pkg/agent/flowexporter/exporter/exporter.go

+		klog.Warningf("Can't find flowType without nodeRouteController")
+		return 0
+	}
+	if exp.nodeRouteController.IPInPodSubnets(record.Conn.TupleOrig.SourceAddress) {


Thanks for adding these lines to address this.

My initial assessment on node network based identification is not correct and was mixed up with something else. There is no straightforward way to detect PodIPs at Antrea agent when operating in network policy only mode.
Therefore, we are supporting Pod-To-External flows for all traffic modes except for the special mode of network policy only mode. Pod-To-Pod flows and Pod-To-Service flows are supported in network policy only mode. There is no regression here.

In this PR, we implemented the logic of flowType value assignment. We distinguished Pod-To-Pod flows and Pod-To-External flows using the podCIDRs of all nodes in the k8s cluster.

srikartati

LGTM
Please edit the description by adding Fixes #XXX to link the issue properly.

dreamtalen · 2021-04-08T18:05:12Z

LGTM
Please edit the description by adding Fixes #XXX to link the issue properly.

Thanks, done.

antoninbas · 2021-04-08T18:06:31Z

pkg/agent/controller/noderoute/node_route_controller.go

+	var curNodeCIDRStr string
+	if ip.To4() != nil {
+		var podIPv4CIDRMaskSize int
+		if c.nodeConfig.PodIPv4CIDR != nil {


we tend to use early returns when possible to reduce indentation levels:

if c.nodeConfig.PodIPv4CIDR == nil { return false } curNodeCIDRStr = c.nodeConfig.PodIPv4CIDR.String() podIPv4CIDRMaskSize, _ := c.nodeConfig.PodIPv4CIDR.Mask.Size()

same below

this can be addressed in a future PR

antoninbas · 2021-04-08T18:09:07Z

pkg/agent/flowexporter/exporter/exporter.go

+func (exp *flowExporter) findFlowType(record flowexporter.FlowRecord) uint8 {
+	// TODO: support Pod-To-External flows in network policy only mode.
+	if exp.isNetworkPolicyOnly {
+		if record.Conn.SourcePodName == "" || record.Conn.DestinationPodName == "" {
+			return ipfixregistry.InterNode
+		}
+		return ipfixregistry.IntraNode
+	}


as discussed with @srikartati offline, I feel like flow type resolution should belong in the Flow Aggregator
The Flow Aggregator could watch all Pods if needed

Thanks @antoninbas. Yes, we have to explore this to support Pod-To-External flows in Nwtqork Policy only mode in the future.

antoninbas · 2021-04-08T18:10:54Z

pkg/agent/flowexporter/exporter/exporter.go

+	// TODO: support Pod-To-External flows in network policy only mode.
+	if exp.isNetworkPolicyOnly {
+		if record.Conn.SourcePodName == "" || record.Conn.DestinationPodName == "" {
+			return ipfixregistry.InterNode


you probably should rename these to FlowTypeInterNode, etc. in go-ipfix so there is more context in the name

As these were used in the context of adding flow types (the function is GetFlowType here), I thought a simple name might be sufficient. If it's confusing, we could change it.
@dreamtalen Could you please create an issue in go-ipfix?

Sure, thanks.

antoninbas · 2021-04-08T18:11:47Z

test/e2e/flowaggregator_test.go

@@ -336,6 +341,11 @@ func checkBandwidthFromRecord(t *testing.T, record, bandwidth string) {
 	}
 }

+// TODO: Add a test that checks the functionality of Pod-To-External flow.


this TODO seems out-of-place on this function, maybe it should have been part of the commit message

srikartati · 2021-04-08T18:36:48Z

/test-all
/test-ipv6-only-e2e
/test-ipv6-e2e

srikartati · 2021-04-08T20:55:47Z

/test-ipv6-only-e2e
/test-ipv6-e2e

srikartati · 2021-04-08T21:45:32Z

I see flow aggregator tests passing both on single stack v6 and dual stack clusters.. there was traceflow test failure in the previous run.
http://10.176.27.169:8080/view/ipv6-only/job/antrea-ipv6-only-e2e-for-pull-request/272/console
http://10.176.27.169:8080/view/dual-stack/job/antrea-ipv6-ds-e2e-for-pull-request/298/console

Merging this

vmwclabot added the cla-not-required label Mar 26, 2021

srikartati reviewed Mar 26, 2021

View reviewed changes

dreamtalen force-pushed the local-add-flowtype branch 2 times, most recently from f94943c to 45c400e Compare March 29, 2021 23:42

dreamtalen force-pushed the local-add-flowtype branch from 45c400e to 8400dbc Compare March 30, 2021 08:12

antoninbas added this to the Antrea v1.0 release milestone Mar 30, 2021

srikartati reviewed Mar 31, 2021

View reviewed changes

jianjuns reviewed Mar 31, 2021

View reviewed changes

dreamtalen force-pushed the local-add-flowtype branch 2 times, most recently from 069b8ed to ea9bdca Compare April 2, 2021 00:36

srikartati reviewed Apr 2, 2021

View reviewed changes

dreamtalen force-pushed the local-add-flowtype branch from ea9bdca to 645423b Compare April 2, 2021 19:00

srikartati reviewed Apr 2, 2021

View reviewed changes

dreamtalen force-pushed the local-add-flowtype branch from 645423b to e9d9b39 Compare April 6, 2021 18:45

srikartati reviewed Apr 6, 2021

View reviewed changes

srikartati requested review from jianjuns and antoninbas April 7, 2021 00:42

antoninbas reviewed Apr 7, 2021

View reviewed changes

dreamtalen force-pushed the local-add-flowtype branch 2 times, most recently from 3ecd2b0 to 6d359c2 Compare April 7, 2021 22:38

srikartati reviewed Apr 7, 2021

View reviewed changes

srikartati requested a review from antoninbas April 8, 2021 05:29

dreamtalen force-pushed the local-add-flowtype branch from 6d359c2 to 55b6c7e Compare April 8, 2021 17:21

Add flowType field for Flow Exporter

c52b20d

In this PR, we implemented the logic of flowType value assignment. We distinguished Pod-To-Pod flows and Pod-To-External flows using the podCIDRs of all nodes in the k8s cluster.

dreamtalen force-pushed the local-add-flowtype branch from 55b6c7e to c52b20d Compare April 8, 2021 17:49

srikartati approved these changes Apr 8, 2021

View reviewed changes

antoninbas approved these changes Apr 8, 2021

View reviewed changes

dreamtalen mentioned this pull request Apr 8, 2021

Rename the flowType field in registry. vmware/go-ipfix#161

Closed

srikartati merged commit 7eeeb49 into antrea-io:main Apr 8, 2021

zyiou added area/flow-visibility Issues or PRs related to flow visibility support in Antrea area/flow-visibility/exporter Issues or PRs related to the Flow Exporter functions in the Agent labels Jun 9, 2021

		@@ -155,6 +155,10 @@ func NetlinkFlowToAntreaConnection(conn conntrack.Flow) flowexporter.Connectio
		SourcePodName: "",

Add flowType field for Flow Exporter #2000

Add flowType field for Flow Exporter #2000

Conversation

dreamtalen commented Mar 26, 2021 • edited Loading

srikartati left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dreamtalen Mar 26, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-io commented Mar 30, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dreamtalen Mar 31, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zyiou Mar 31, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dreamtalen Mar 31, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikartati Apr 2, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikartati left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dreamtalen Apr 6, 2021 • edited Loading

Choose a reason for hiding this comment

srikartati left a comment

Choose a reason for hiding this comment

srikartati Apr 6, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikartati Apr 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikartati left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikartati Apr 7, 2021 • edited Loading

Choose a reason for hiding this comment

srikartati left a comment

Choose a reason for hiding this comment

dreamtalen commented Apr 8, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikartati Apr 8, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srikartati Apr 8, 2021 • edited Loading

Choose a reason for hiding this comment

dreamtalen commented Mar 26, 2021 •

edited

Loading

srikartati left a comment •

edited

Loading

dreamtalen Mar 26, 2021 •

edited

Loading

codecov-io commented Mar 30, 2021 •

edited

Loading

dreamtalen Mar 31, 2021 •

edited

Loading

zyiou Mar 31, 2021 •

edited

Loading

dreamtalen Mar 31, 2021 •

edited

Loading

srikartati Apr 2, 2021 •

edited

Loading

dreamtalen Apr 6, 2021 •

edited

Loading

srikartati Apr 6, 2021 •

edited

Loading

srikartati Apr 7, 2021 •

edited

Loading

srikartati Apr 7, 2021 •

edited

Loading

srikartati Apr 8, 2021 •

edited

Loading

srikartati Apr 8, 2021 •

edited

Loading

srikartati commented Apr 8, 2021 •

edited

Loading