chore: improving e2e testing latency #1760

samhith-kakarla · 2024-06-12T19:09:38Z

creating the pr for testing of ci do not merge right now

codecov · 2024-06-12T19:26:30Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 57.23%. Comparing base (c70540b) to head (d60988c).

❗ Current head d60988c differs from pull request most recent head a8588a8

Please upload reports for the commit a8588a8 to get more accurate results.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1760      +/-   ##
==========================================
- Coverage   60.94%   57.23%   -3.71%     
==========================================
  Files         232      218      -14     
  Lines       20014    17581    -2433     
==========================================
- Hits        12197    10063    -2134     
+ Misses       6959     6664     -295     
+ Partials      858      854       -4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

whynowy · 2024-06-12T21:46:26Z

@samhith-kakarla - you might need to look into the original PR, we want to revert some changes over there:

We do not want to start kafka, nats every time when we run an E2E test, we only want to do that when a Kafka or nats e2e test is running. This means, we still want to use go:generate to do the installation.
With this, it comes a problem for the test api, that it initializes the connection (e.g. Kafka client) when the test api service starts, which requires the kafka/nats services running. To solve the problem, move the client initialization to lazy load mode with a lock.

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

whynowy · 2024-06-18T20:21:03Z

Makefile

@@ -171,9 +149,6 @@ cleanup-e2e:
 # To run just one of the e2e tests by name (i.e. 'make TestCreateSimplePipeline'):
 Test%:
 	$(MAKE) cleanup-e2e
-	$(MAKE) deploy-nats
-	$(MAKE) deploy-kafka
-	$(MAKE) deploy-redis
 	$(MAKE) image e2eapi-image
 	kubectl -n numaflow-system delete po -lapp.kubernetes.io/component=controller-manager,app.kubernetes.io/part-of=numaflow
 	kubectl -n numaflow-system delete po e2e-api-pod  --ignore-not-found=true


Missing go generate?

whynowy · 2024-06-18T20:23:37Z

test/jetstream-e2e/jetstream_test.go

@@ -25,6 +25,10 @@ import (
 	"github.com/stretchr/testify/suite"
 )

+//go:generate kubectl -n numaflow-system delete statefulset nats --ignore-not-found=true
+//go:generate kubectl apply -k ../../config/apps/nats -n numaflow-system
+//go:generate kubectl apply -f testdata/nats-auth-fake-token.yaml -n numaflow-system


Do we need this line?

yes, this is how the resources are getting deployed when go generate is called on the file

Why wasn't this needed before?

because i think the commit where this jetstream test was added(#1723) came after the e2e re-factor pr (#1725) so at this point all the resources were getting deployed in the make file whenever any test was run. But now since we are no longer deploying all the resources in the make file, need this line here.

I don't get it, shouldn't this line be covered in kubectl apply -k ../../config/apps/nats -n numaflow-system?

are you talking about this line //go:generate kubectl apply -f testdata/nats-auth-fake-token.yaml -n numaflow-system if so i have removed it: i have only kept : //go:generatekubectl -n numaflow-system delete statefulset nats --ignore-not-found=true //go:generate kubectl apply -k ../../config/apps/nats -n numaflow-system

whynowy · 2024-06-18T20:24:27Z

test/jetstream-e2e/testdata/nats-auth-fake-token.yaml

@@ -0,0 +1,9 @@
+
+apiVersion: v1


Where is this file used?

it is getting used in the following line: /go:generate kubectl apply -f testdata/nats-auth-fake-token.yaml -n numaflow-system

whynowy · 2024-06-18T20:24:43Z

test/nats-e2e/nats_test.go

@@ -27,6 +27,10 @@ import (
 	. "github.com/numaproj/numaflow/test/fixtures"
 )

+//go:generate kubectl -n numaflow-system delete statefulset nats --ignore-not-found=true
+//go:generate kubectl apply -k ../../config/apps/nats -n numaflow-system
+//go:generate kubectl apply -f testdata/nats-auth-fake-token.yaml -n numaflow-system


Do we need this line?

yes, this is how the resources are getting deployed when go generate is called on the file

whynowy · 2024-06-18T20:25:23Z

test/nats-e2e/testdata/nats-auth-fake-token.yaml

@@ -0,0 +1,9 @@
+


Is this file in-use?

yes it is used here : //go:generate kubectl apply -f testdata/nats-auth-fake-token.yaml -n numaflow-system

whynowy · 2024-06-18T20:28:16Z

test/e2e-api/kafka.go

 func (kh *KafkaController) CreateTopicHandler(w http.ResponseWriter, r *http.Request) {
+
+	m.Lock()


Abstract this as a function with lock, returning a client.

whynowy · 2024-06-18T20:29:58Z

test/e2e-api/http.go

@@ -42,7 +42,20 @@ func NewHttpController() *HttpController {
 	}
 }

+func NewHttpController() *HttpController {


Do we need to make change to this file? Initialization with an HTTP client should be okay, right?

you are right, we do not need to change this file. i will reverse the changes

whynowy · 2024-06-18T20:30:59Z

test/e2e-api/nats.go

 	}
 }

 func (n *NatsController) PumpSubject(w http.ResponseWriter, r *http.Request) {
+	m.Lock()


Same here, abstract this as a function with lock, returning a client.

whynowy · 2024-06-18T20:32:21Z

test/e2e-api/redis.go

 func (h *RedisController) GetMsgCountContains(w http.ResponseWriter, r *http.Request) {
+	m.Lock()


Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

whynowy · 2024-06-20T16:52:47Z

Makefile

 	$(MAKE) image e2eapi-image
 	kubectl -n numaflow-system delete po -lapp.kubernetes.io/component=controller-manager,app.kubernetes.io/part-of=numaflow
 	kubectl -n numaflow-system delete po e2e-api-pod  --ignore-not-found=true
+	go generate $(shell find ./test/kafka-e2e -name '*.go')


Why don't use go generate $(shell find ./test/$* -name '*.go') as above?

i think because that portion of the makefile is To run just one of the e2e tests by name (i.e. 'make TestCreateSimplePipeline'): so i dont think go generate $(shell find ./test/$* -name '*.go') works as this requires test file name as input not a test name.

We can try to do a selective go generate here as well by trying to extract the file name, maybe this can help
shell grep $(*) -R ./test | head -1 | awk -F\/ '{print $$3}'

whynowy · 2024-06-20T16:53:54Z

test/e2e-api/nats.go

-func NewNatsController(url string, token string) *NatsController {
-	opts := []natslib.Option{natslib.Token(token)}
-	nc, err := natslib.Connect(url, opts...)
+func initNewNats(n *NatsController) *NatsController {


Why does it returns a * NatsController?

whynowy · 2024-06-20T16:54:18Z

test/e2e-api/kafka.go

-func NewKafkaController() *KafkaController {
-	// initialize Kafka handlers
-	var brokers = []string{bootstrapServers}
+func initNewKafka(n *KafkaController) *KafkaController {


Why it returns *KafkaController?

i felt it was just cleaner to do that. that way when i call the function i can just re-assign the controller like this kh = initNewKafka(kh). otherwise i would need to return producer, consumer and adminClient individually and re-assign those fields when the init function is called right ?

whynowy · 2024-06-20T16:54:34Z

test/e2e-api/redis.go

-func NewRedisController() *RedisController {
-	// When we use this API to validate e2e test result, we always assume a redis UDSink is used
-	// to persist data to a redis instance listening on port 6379.
+func initNewReddis(n *RedisController) *RedisController {


Same question as above.

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

kohlisid · 2024-06-20T20:46:45Z

test/e2e-api/kafka.go

 	config.Producer.Partitioner = sarama.NewManualPartitioner

-	producer, err := sarama.NewSyncProducer(brokers, config)
+	adminClient, err := sarama.NewClusterAdmin(n.brokers, config)


We can directly assign to n.adminClient and return the same

kohlisid · 2024-06-20T20:54:10Z

test/e2e-api/kafka.go

+	config.Producer.Return.Successes = true
+	config.Producer.Partitioner = sarama.NewManualPartitioner
+
+	consumer, err := sarama.NewConsumer(n.brokers, config)


Can directly assign, same as above.

kohlisid · 2024-06-20T20:54:34Z

test/e2e-api/kafka.go

+	config.Producer.Return.Successes = true
+	config.Producer.Partitioner = sarama.NewManualPartitioner
+
+	producer, err := sarama.NewSyncProducer(n.brokers, config)


Can directly assign, same as above.

kohlisid · 2024-06-20T20:54:52Z

test/e2e-api/kafka.go

+
+}
+
+// func initNewKafka(n *KafkaController) *KafkaController {


Please remove this

kohlisid · 2024-06-20T20:55:16Z

test/e2e-api/kafka.go

+// }
+
+func NewKafkaController() *KafkaController {
+	// initialize Kafka handlers


Please clean up any extra commented lines

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

kohlisid · 2024-06-21T17:50:51Z

test/e2e-api/kafka.go

-		adminClient: adminClient,
-		producer:    producer,
-		consumer:    consumer,
+		adminClient: nil,


Please add a comment here indicating why these are set to nil initially.

kohlisid · 2024-06-21T17:50:56Z

test/e2e-api/kafka.go

-func NewKafkaController() *KafkaController {
-	// initialize Kafka handlers
-	var brokers = []string{bootstrapServers}
+// getter method for lazy loading. create and return the admin client only when required


getKafkaClient is used for getting a Kafka admin client for the given controller config. It is implemented with a lazy loading mechanism 1) A new client created only for the first request 2) Returning the current client if it exists

kohlisid · 2024-06-21T17:53:45Z

@samhith-kakarla Please add the latency improvement stats that you collected here

kohlisid · 2024-06-21T17:56:06Z

test/e2e-api/kafka.go

+	n.mLock.Lock()
+	defer n.mLock.Unlock()
+	if n.consumer != nil {
+		log.Println("consumer already existed")


We can remove this log

kohlisid · 2024-06-21T17:56:19Z

test/e2e-api/kafka.go

+	n.mLock.Lock()
+	defer n.mLock.Unlock()
+	if n.producer != nil {
+		log.Println("producer already existed")


We can remove this log

kohlisid · 2024-06-21T17:58:05Z

test/e2e-api/nats.go

@@ -149,5 +172,6 @@ func (n *NatsController) PumpJetstream(w http.ResponseWriter, r *http.Request) {
 }

 func (n *NatsController) Close() {
+	// no lazy clozing required becuase they check if nc is not null before they close


Typos
// Nil check is not required here because the client.Close() function does a nil check internally.

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

kohlisid · 2024-06-21T18:53:24Z

@samhith-kakarla There are permission issues with the doc you added, can you directly add the raw data here. Might be easier

samhith-kakarla · 2024-06-21T18:57:32Z

ci runtimes before and after improvemnts:

kohlisid · 2024-06-21T19:20:55Z

@samhith-kakarla As discussed, Could you add the verification result for the local testing for single test function with the updated go generate to block extra resources for those cases as well.

samhith-kakarla · 2024-06-21T21:12:51Z

Local Testing results for single test function.

here is the result of the grep command run on the test name:

Here is the results of running make TestKafkaSourceSink:

it finds the right test file, deploys the correct resources and passes.

whynowy · 2024-06-24T00:18:06Z

test/e2e-api/kafka.go

+*/
+
+func (n *KafkaController) getKafkaClient() sarama.ClusterAdmin {
+	n.mLock.Lock()


These 2 lines should be moved to line 54.

whynowy · 2024-06-24T00:23:31Z

test/e2e-api/kafka.go

+  2) Returning the current client if it exists
+*/
+
+func (n *KafkaController) getKafkaClient() sarama.ClusterAdmin {


Since this is just for testing, we don't need to separate them into multiple different get functions, instead, do something like func (n *KafkaController) getProducerAndConsumer() (sarama.ClusterAdmin, sarama.SyncProducer, sarama.Consumer).

This is to avoid too much redundant code.

whynowy · 2024-06-24T00:25:01Z

test/e2e-api/kafka.go

+func (n *KafkaController) getKafkaClient() sarama.ClusterAdmin {
+	n.mLock.Lock()
+	defer n.mLock.Unlock()
+	if n.adminClient != nil {


This check needs to happen again after the lock is acquired.

whynowy · 2024-06-24T00:26:47Z

test/e2e-api/nats.go

-	nc, err := natslib.Connect(url, opts...)
+// getter method for lazy loading. creates and returns nats client only when required
+func (n *NatsController) getNatsClient() *natslib.Conn {
+	n.mLock.Lock()


Lock should happen after client nil check.

whynowy · 2024-06-24T00:27:34Z

test/e2e-api/nats.go

+func (n *NatsController) getNatsClient() *natslib.Conn {
+	n.mLock.Lock()
+	defer n.mLock.Unlock()
+	if n.client != nil {


Similarly, nil check should happen twice, before and after acquiring the lock.

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com> Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

lazy-loading

4ef5619

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

samhith-kakarla force-pushed the testing-fix branch from d2ddc8c to 4ef5619 Compare June 17, 2024 21:41

samhith-kakarla added 4 commits June 17, 2024 15:19

fixing jetstream

ffe53ac

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

jetstream-fixed

dc7d488

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

http

6e49913

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

cleanup

d49e2b2

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

whynowy reviewed Jun 18, 2024

View reviewed changes

fixes

9a24d75

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

whynowy reviewed Jun 20, 2024

View reviewed changes

lazy loading fixes

d60988c

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

kohlisid reviewed Jun 20, 2024

View reviewed changes

test/e2e-api/kafka.go Outdated

}

// func initNewKafka(n *KafkaController) *KafkaController {

Copy link

Contributor

kohlisid Jun 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove this

kohlisid reviewed Jun 20, 2024

View reviewed changes

go generate fixes

8b760b0

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

kohlisid reviewed Jun 21, 2024

View reviewed changes

commenting

808b78d

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

samhith-kakarla force-pushed the testing-fix branch from 6f07f90 to 808b78d Compare June 21, 2024 18:25

kohlisid requested a review from whynowy June 21, 2024 18:33

samhith-kakarla marked this pull request as ready for review June 21, 2024 22:18

samhith-kakarla requested a review from vigith as a code owner June 21, 2024 22:18

whynowy reviewed Jun 24, 2024

View reviewed changes

kafka combined

402ec82

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

samhith-kakarla force-pushed the testing-fix branch from 9a36fe6 to 402ec82 Compare June 24, 2024 18:28

whynowy approved these changes Jun 24, 2024

View reviewed changes

comments update

9aaa037

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

samhith-kakarla force-pushed the testing-fix branch from a42dc4b to 9aaa037 Compare June 24, 2024 21:41

Merge branch 'main' into testing-fix

a8588a8

whynowy merged commit efcd722 into numaproj:main Jun 24, 2024
25 checks passed

samhith-kakarla added a commit to samhith-kakarla/numaflow that referenced this pull request Jul 5, 2024

chore: improving e2e testing latency (numaproj#1760)

75347d5

Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com> Signed-off-by: Samhith Kakarla <samhith_kakarla@intuit.com>

whynowy mentioned this pull request Jul 12, 2024

Refactor the test api and e2e test script to avoid starting unnecessary services #1726

Closed

		func (kh KafkaController) CreateTopicHandler(w http.ResponseWriter, r http.Request) {

		m.Lock()

		func (h RedisController) GetMsgCountContains(w http.ResponseWriter, r http.Request) {
		m.Lock()


		}

		// func initNewKafka(n KafkaController) KafkaController {

chore: improving e2e testing latency #1760

chore: improving e2e testing latency #1760

Conversation

samhith-kakarla commented Jun 12, 2024

codecov bot commented Jun 12, 2024 • edited Loading

Codecov Report

whynowy commented Jun 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samhith-kakarla Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samhith-kakarla Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samhith-kakarla Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

kohlisid Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

samhith-kakarla Jun 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kohlisid commented Jun 21, 2024

kohlisid Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kohlisid Jun 21, 2024 • edited Loading

Choose a reason for hiding this comment

kohlisid commented Jun 21, 2024

samhith-kakarla commented Jun 21, 2024

kohlisid commented Jun 21, 2024

samhith-kakarla commented Jun 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 12, 2024 •

edited

Loading

samhith-kakarla Jun 20, 2024 •

edited

Loading

samhith-kakarla Jun 20, 2024 •

edited

Loading

samhith-kakarla Jun 20, 2024 •

edited

Loading

kohlisid Jun 20, 2024 •

edited

Loading

samhith-kakarla Jun 20, 2024 •

edited

Loading

kohlisid Jun 21, 2024 •

edited

Loading

kohlisid Jun 21, 2024 •

edited

Loading